Nov 01 03:03:51.279266 ip-10-0-141-122 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 01 03:03:51.279280 ip-10-0-141-122 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 01 03:03:51.279289 ip-10-0-141-122 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 01 03:03:51.279614 ip-10-0-141-122 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 01 03:04:01.316598 ip-10-0-141-122 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Nov 01 03:04:01.316615 ip-10-0-141-122 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 5c802453d661409fb1397fbd23e8d7cf --
Nov 01 03:05:53.769856 ip-10-0-141-122 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 03:05:54.185379 ip-10-0-141-122 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 03:05:54.185379 ip-10-0-141-122 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 03:05:54.185379 ip-10-0-141-122 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 03:05:54.185379 ip-10-0-141-122 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 03:05:54.191244 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191166    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:05:54.191244 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191243    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:05:54.191244 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191248    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:05:54.191336 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191251    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:05:54.191336 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191255    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:05:54.191336 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191260    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:05:54.191336 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191265    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:05:54.191336 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191268    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:05:54.191336 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191271    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:05:54.191336 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191274    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:05:54.191336 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191276    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:05:54.191336 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191279    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:05:54.191336 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191282    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:05:54.191336 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191284    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:05:54.191336 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191287    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:05:54.191336 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191290    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:05:54.191336 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191292    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:05:54.191336 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191295    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:05:54.191336 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191298    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:05:54.191336 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191300    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:05:54.191336 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191303    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:05:54.191336 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191305    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:05:54.191803 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191318    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:05:54.191803 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191321    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:05:54.191803 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191324    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:05:54.191803 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191327    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:05:54.191803 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191330    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:05:54.191803 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191332    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:05:54.191803 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191335    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:05:54.191803 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191337    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:05:54.191803 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191340    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:05:54.191803 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191343    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:05:54.191803 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191346    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:05:54.191803 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191349    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:05:54.191803 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191352    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:05:54.191803 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191354    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:05:54.191803 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191357    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:05:54.191803 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191359    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:05:54.191803 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191362    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:05:54.191803 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191365    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:05:54.191803 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191368    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:05:54.191803 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191370    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:05:54.192255 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191373    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:05:54.192255 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191375    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:05:54.192255 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191378    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:05:54.192255 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191381    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:05:54.192255 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191383    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:05:54.192255 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191386    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:05:54.192255 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191389    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:05:54.192255 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191392    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:05:54.192255 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191395    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:05:54.192255 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191397    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:05:54.192255 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191401    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:05:54.192255 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191403    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:05:54.192255 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191407    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:05:54.192255 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191410    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:05:54.192255 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191413    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:05:54.192255 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191415    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:05:54.192255 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191418    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:05:54.192255 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191421    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:05:54.192255 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191423    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:05:54.192255 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191426    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:05:54.192735 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191429    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:05:54.192735 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191431    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:05:54.192735 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191434    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:05:54.192735 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191437    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:05:54.192735 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191440    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:05:54.192735 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191442    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:05:54.192735 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191445    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:05:54.192735 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191447    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:05:54.192735 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191450    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:05:54.192735 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191453    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:05:54.192735 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191455    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:05:54.192735 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191457    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:05:54.192735 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191460    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:05:54.192735 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191463    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:05:54.192735 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191465    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:05:54.192735 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191469    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:05:54.192735 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191471    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:05:54.192735 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191474    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:05:54.192735 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191476    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:05:54.193213 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191479    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:05:54.193213 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191483    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:05:54.193213 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191487    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:05:54.193213 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191492    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:05:54.193213 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191494    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:05:54.193213 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191497    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:05:54.193213 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191500    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:05:54.193213 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191503    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:05:54.193213 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191505    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:05:54.193213 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191508    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:05:54.193213 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191510    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:05:54.193213 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191513    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:05:54.193213 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191516    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:05:54.193213 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191519    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:05:54.193213 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191521    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:05:54.193213 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191524    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:05:54.193213 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191526    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:05:54.193213 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191529    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:05:54.193213 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191531    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:05:54.193213 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191534    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:05:54.193702 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191537    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:05:54.193702 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191539    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:05:54.193702 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191542    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:05:54.193702 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191545    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:05:54.193702 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.191547    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:05:54.193702 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193105    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:05:54.193702 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193113    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:05:54.193702 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193117    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:05:54.193702 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193120    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:05:54.193702 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193123    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:05:54.193702 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193127    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:05:54.193702 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193132    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:05:54.193702 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193135    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:05:54.193702 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193139    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:05:54.193702 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193142    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:05:54.193702 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193146    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:05:54.193702 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193150    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:05:54.193702 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193153    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:05:54.194174 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193155    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:05:54.194174 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193158    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:05:54.194174 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193161    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:05:54.194174 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193163    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:05:54.194174 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193166    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:05:54.194174 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193168    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:05:54.194174 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193171    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:05:54.194174 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193173    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:05:54.194174 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193176    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:05:54.194174 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193179    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:05:54.194174 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193181    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:05:54.194174 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193184    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:05:54.194174 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193186    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:05:54.194174 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193189    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:05:54.194174 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193191    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:05:54.194174 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193194    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:05:54.194174 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193197    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:05:54.194174 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193199    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:05:54.194174 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193202    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:05:54.194174 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193204    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:05:54.194686 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193207    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:05:54.194686 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193210    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:05:54.194686 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193213    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:05:54.194686 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193216    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:05:54.194686 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193219    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:05:54.194686 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193221    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:05:54.194686 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193224    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:05:54.194686 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193227    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:05:54.194686 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193230    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:05:54.194686 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193233    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:05:54.194686 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193235    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:05:54.194686 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193238    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:05:54.194686 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193240    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:05:54.194686 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193243    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:05:54.194686 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193246    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:05:54.194686 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193248    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:05:54.194686 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193250    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:05:54.194686 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193253    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:05:54.194686 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193255    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:05:54.194686 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193258    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:05:54.195163 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193260    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:05:54.195163 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193263    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:05:54.195163 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193265    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:05:54.195163 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193268    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:05:54.195163 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193270    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:05:54.195163 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193273    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:05:54.195163 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193275    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:05:54.195163 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193277    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:05:54.195163 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193281    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:05:54.195163 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193285    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:05:54.195163 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193289    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:05:54.195163 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193292    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:05:54.195163 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193295    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:05:54.195163 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193297    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:05:54.195163 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193300    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:05:54.195163 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193303    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:05:54.195163 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193305    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:05:54.195163 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193308    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:05:54.195163 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193310    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:05:54.195663 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193313    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:05:54.195663 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193315    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:05:54.195663 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193318    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:05:54.195663 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193321    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:05:54.195663 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193324    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:05:54.195663 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193327    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:05:54.195663 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193329    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:05:54.195663 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193331    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:05:54.195663 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193334    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:05:54.195663 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193336    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:05:54.195663 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193339    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:05:54.195663 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193341    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:05:54.195663 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193344    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:05:54.195663 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193346    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:05:54.195663 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193349    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:05:54.195663 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193351    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:05:54.195663 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193354    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:05:54.195663 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193357    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:05:54.195663 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193359    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:05:54.196124 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193362    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:05:54.196124 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193365    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:05:54.196124 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193368    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:05:54.196124 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193370    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:05:54.196124 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193374    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:05:54.196124 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193376    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:05:54.196124 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193379    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:05:54.196124 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193381    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:05:54.196124 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193384    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:05:54.196124 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193387    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:05:54.196124 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193390    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:05:54.196124 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193392    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:05:54.196124 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193394    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:05:54.196124 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193398    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:05:54.196124 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193400    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:05:54.196124 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193928    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:05:54.196124 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193936    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:05:54.196124 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193940    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:05:54.196124 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193943    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:05:54.196124 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193947    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:05:54.196693 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193949    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:05:54.196693 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193952    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:05:54.196693 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193955    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:05:54.196693 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193958    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:05:54.196693 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193961    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:05:54.196693 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193964    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:05:54.196693 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193966    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:05:54.196693 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193969    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:05:54.196693 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193971    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:05:54.196693 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193974    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:05:54.196693 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193977    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:05:54.196693 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193982    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:05:54.196693 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193986    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:05:54.196693 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193990    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:05:54.196693 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193993    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:05:54.196693 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193996    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:05:54.196693 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.193999    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:05:54.196693 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194002    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:05:54.196693 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194005    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:05:54.196693 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194007    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:05:54.197180 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194010    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:05:54.197180 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194012    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:05:54.197180 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194015    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:05:54.197180 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194018    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:05:54.197180 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194020    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:05:54.197180 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194023    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:05:54.197180 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194025    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:05:54.197180 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194028    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:05:54.197180 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194031    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:05:54.197180 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194033    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:05:54.197180 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194036    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:05:54.197180 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194039    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:05:54.197180 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194042    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:05:54.197180 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194045    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:05:54.197180 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194048    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:05:54.197180 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194051    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:05:54.197180 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194053    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:05:54.197180 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194056    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:05:54.197180 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194058    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:05:54.197644 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194062    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:05:54.197644 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194066    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:05:54.197644 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194069    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:05:54.197644 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194071    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:05:54.197644 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194075    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:05:54.197644 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194078    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:05:54.197644 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194081    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:05:54.197644 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194083    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:05:54.197644 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194086    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:05:54.197644 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194088    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:05:54.197644 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194091    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:05:54.197644 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194093    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:05:54.197644 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194096    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:05:54.197644 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194099    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:05:54.197644 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194101    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:05:54.197644 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194103    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:05:54.197644 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194106    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:05:54.197644 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194108    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:05:54.197644 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194111    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:05:54.198083 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194114    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:05:54.198083 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194116    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:05:54.198083 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194118    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:05:54.198083 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194122    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:05:54.198083 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194124    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:05:54.198083 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194127    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:05:54.198083 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194129    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:05:54.198083 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194132    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:05:54.198083 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194134    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:05:54.198083 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194137    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:05:54.198083 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194139    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:05:54.198083 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194142    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:05:54.198083 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194144    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:05:54.198083 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194146    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:05:54.198083 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194149    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:05:54.198083 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194151    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:05:54.198083 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194154    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:05:54.198083 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194158    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:05:54.198083 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194161    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:05:54.198083 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194164    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:05:54.198544 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194166    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:05:54.198544 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194169    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:05:54.198544 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194171    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:05:54.198544 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194174    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:05:54.198544 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194177    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:05:54.198544 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194180    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:05:54.198544 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194182    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:05:54.198544 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194185    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:05:54.198544 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194188    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:05:54.198544 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194191    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:05:54.198544 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194194    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:05:54.198544 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194197    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:05:54.198544 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194199    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:05:54.198544 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194202    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:05:54.198544 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194206    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:05:54.198544 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194208    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:05:54.198544 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194211    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:05:54.198544 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194214    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:05:54.198544 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194216    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:05:54.198990 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194219    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:05:54.198990 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194221    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:05:54.198990 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194224    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:05:54.198990 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.194226    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:05:54.198990 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195300    2578 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 03:05:54.198990 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195311    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 03:05:54.198990 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195318    2578 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 03:05:54.198990 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195323    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 03:05:54.198990 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195328    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 03:05:54.198990 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195331    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 03:05:54.198990 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195336    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 03:05:54.198990 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195342    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 03:05:54.198990 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195345    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 03:05:54.198990 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195349    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 03:05:54.198990 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195352    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 03:05:54.198990 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195355    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 03:05:54.198990 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195358    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 03:05:54.198990 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195361    2578 flags.go:64] FLAG: --cgroup-root=""
Nov 01 03:05:54.198990 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195364    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 03:05:54.198990 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195367    2578 flags.go:64] FLAG: --client-ca-file=""
Nov 01 03:05:54.198990 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195370    2578 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 03:05:54.198990 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195373    2578 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 03:05:54.198990 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195377    2578 flags.go:64] FLAG: --cluster-domain=""
Nov 01 03:05:54.199591 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195380    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 03:05:54.199591 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195384    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 03:05:54.199591 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195387    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 03:05:54.199591 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195391    2578 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 03:05:54.199591 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195394    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 03:05:54.199591 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195398    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 03:05:54.199591 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195401    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 03:05:54.199591 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195405    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 03:05:54.199591 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195408    2578 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 03:05:54.199591 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195411    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 03:05:54.199591 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195414    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 03:05:54.199591 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195418    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 03:05:54.199591 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195420    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 03:05:54.199591 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195425    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 03:05:54.199591 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195428    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 03:05:54.199591 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195431    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 03:05:54.199591 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195434    2578 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 03:05:54.199591 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195437    2578 flags.go:64] FLAG: --enable-server="true"
Nov 01 03:05:54.199591 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195440    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 03:05:54.199591 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195445    2578 flags.go:64] FLAG: --event-burst="100"
Nov 01 03:05:54.199591 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195449    2578 flags.go:64] FLAG: --event-qps="50"
Nov 01 03:05:54.199591 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195452    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 03:05:54.199591 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195456    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 03:05:54.199591 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195458    2578 flags.go:64] FLAG: --eviction-hard=""
Nov 01 03:05:54.200226 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195463    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 03:05:54.200226 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195466    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 03:05:54.200226 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195469    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 03:05:54.200226 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195472    2578 flags.go:64] FLAG: --eviction-soft=""
Nov 01 03:05:54.200226 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195476    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 03:05:54.200226 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195479    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 03:05:54.200226 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195482    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 03:05:54.200226 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195485    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 03:05:54.200226 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195488    2578 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 03:05:54.200226 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195491    2578 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 03:05:54.200226 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195494    2578 flags.go:64] FLAG: --feature-gates=""
Nov 01 03:05:54.200226 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195498    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 03:05:54.200226 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195501    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 03:05:54.200226 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195504    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 03:05:54.200226 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195508    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 03:05:54.200226 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195511    2578 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 03:05:54.200226 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195515    2578 flags.go:64] FLAG: --help="false"
Nov 01 03:05:54.200226 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195519    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-141-122.us-west-1.compute.internal"
Nov 01 03:05:54.200226 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195522    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 03:05:54.200226 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195525    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 03:05:54.200226 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195528    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 03:05:54.200226 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195531    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 03:05:54.200226 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195535    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 03:05:54.200780 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195538    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 03:05:54.200780 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195541    2578 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 03:05:54.200780 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195544    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 03:05:54.200780 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195547    2578 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 03:05:54.200780 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195550    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 03:05:54.200780 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195566    2578 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 03:05:54.200780 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195569    2578 flags.go:64] FLAG: --kube-reserved=""
Nov 01 03:05:54.200780 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195572    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 03:05:54.200780 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195575    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 03:05:54.200780 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195578    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 03:05:54.200780 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195581    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 03:05:54.200780 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195584    2578 flags.go:64] FLAG: --lock-file=""
Nov 01 03:05:54.200780 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195587    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 03:05:54.200780 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195590    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 03:05:54.200780 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195594    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 03:05:54.200780 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195599    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 03:05:54.200780 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195602    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 03:05:54.200780 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195605    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 03:05:54.200780 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195608    2578 flags.go:64] FLAG: --logging-format="text"
Nov 01 03:05:54.200780 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195611    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 03:05:54.200780 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195615    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 03:05:54.200780 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195618    2578 flags.go:64] FLAG: --manifest-url=""
Nov 01 03:05:54.200780 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195621    2578 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 03:05:54.200780 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195625    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 03:05:54.200780 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195628    2578 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 03:05:54.201380 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195632    2578 flags.go:64] FLAG: --max-pods="110"
Nov 01 03:05:54.201380 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195636    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 03:05:54.201380 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195639    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 03:05:54.201380 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195643    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 03:05:54.201380 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195646    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 03:05:54.201380 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195649    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 03:05:54.201380 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195652    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 03:05:54.201380 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195655    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 03:05:54.201380 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195679    2578 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 03:05:54.201380 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195683    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 03:05:54.201380 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195686    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 03:05:54.201380 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195689    2578 flags.go:64] FLAG: --pod-cidr=""
Nov 01 03:05:54.201380 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195693    2578 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 03:05:54.201380 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195696    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 03:05:54.201380 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195699    2578 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 03:05:54.201380 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195702    2578 flags.go:64] FLAG: --port="10250"
Nov 01 03:05:54.201380 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195706    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 03:05:54.201380 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195708    2578 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0e2dc9ee5fdad2e52"
Nov 01 03:05:54.201380 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195712    2578 flags.go:64] FLAG: --qos-reserved=""
Nov 01 03:05:54.201380 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195715    2578 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 03:05:54.201380 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195718    2578 flags.go:64] FLAG: --register-node="true"
Nov 01 03:05:54.201380 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195721    2578 flags.go:64] FLAG: --register-with-taints=""
Nov 01 03:05:54.201380 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195725    2578 flags.go:64] FLAG: --registry-burst="10"
Nov 01 03:05:54.201380 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195728    2578 flags.go:64] FLAG: --registry-qps="5"
Nov 01 03:05:54.201380 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195731    2578 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 03:05:54.201380 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195734    2578 flags.go:64] FLAG: --reserved-memory=""
Nov 01 03:05:54.202013 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195737    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 03:05:54.202013 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195741    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 03:05:54.202013 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195744    2578 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 03:05:54.202013 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195747    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 03:05:54.202013 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195750    2578 flags.go:64] FLAG: --runonce="false"
Nov 01 03:05:54.202013 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195754    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 03:05:54.202013 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195758    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 03:05:54.202013 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195761    2578 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 03:05:54.202013 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195765    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 03:05:54.202013 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195768    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 03:05:54.202013 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195772    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 03:05:54.202013 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195776    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 03:05:54.202013 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195779    2578 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 03:05:54.202013 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195782    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 03:05:54.202013 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195785    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 03:05:54.202013 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195788    2578 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 03:05:54.202013 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195791    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 03:05:54.202013 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195795    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 03:05:54.202013 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195799    2578 flags.go:64] FLAG: --system-cgroups=""
Nov 01 03:05:54.202013 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195802    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 03:05:54.202013 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195808    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 03:05:54.202013 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195811    2578 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 03:05:54.202013 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195814    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 03:05:54.202013 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195818    2578 flags.go:64] FLAG: --tls-min-version=""
Nov 01 03:05:54.202013 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195821    2578 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 03:05:54.202625 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195835    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 03:05:54.202625 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195838    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 03:05:54.202625 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195841    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 03:05:54.202625 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195845    2578 flags.go:64] FLAG: --v="2"
Nov 01 03:05:54.202625 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195849    2578 flags.go:64] FLAG: --version="false"
Nov 01 03:05:54.202625 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195854    2578 flags.go:64] FLAG: --vmodule=""
Nov 01 03:05:54.202625 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195858    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 03:05:54.202625 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.195861    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 03:05:54.202625 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.195992    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:05:54.202625 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.195996    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:05:54.202625 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196000    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:05:54.202625 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196003    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:05:54.202625 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196007    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:05:54.202625 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196010    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:05:54.202625 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196013    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:05:54.202625 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196015    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:05:54.202625 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196018    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:05:54.202625 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196022    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:05:54.202625 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196026    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:05:54.202625 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196030    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:05:54.202625 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196032    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:05:54.203118 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196035    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:05:54.203118 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196038    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:05:54.203118 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196040    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:05:54.203118 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196043    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:05:54.203118 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196047    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:05:54.203118 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196050    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:05:54.203118 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196053    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:05:54.203118 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196055    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:05:54.203118 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196058    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:05:54.203118 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196061    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:05:54.203118 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196064    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:05:54.203118 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196067    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:05:54.203118 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196070    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:05:54.203118 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196073    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:05:54.203118 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196075    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:05:54.203118 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196078    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:05:54.203118 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196081    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:05:54.203118 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196083    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:05:54.203118 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196086    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:05:54.203590 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196088    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:05:54.203590 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196091    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:05:54.203590 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196093    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:05:54.203590 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196096    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:05:54.203590 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196100    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:05:54.203590 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196102    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:05:54.203590 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196105    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:05:54.203590 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196108    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:05:54.203590 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196110    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:05:54.203590 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196114    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:05:54.203590 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196116    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:05:54.203590 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196119    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:05:54.203590 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196122    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:05:54.203590 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196124    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:05:54.203590 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196127    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:05:54.203590 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196130    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:05:54.203590 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196133    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:05:54.203590 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196136    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:05:54.203590 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196139    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:05:54.203590 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196142    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:05:54.204061 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196145    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:05:54.204061 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196147    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:05:54.204061 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196150    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:05:54.204061 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196152    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:05:54.204061 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196155    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:05:54.204061 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196158    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:05:54.204061 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196160    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:05:54.204061 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196162    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:05:54.204061 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196165    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:05:54.204061 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196167    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:05:54.204061 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196170    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:05:54.204061 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196173    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:05:54.204061 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196176    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:05:54.204061 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196179    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:05:54.204061 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196181    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:05:54.204061 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196183    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:05:54.204061 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196187    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:05:54.204061 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196190    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:05:54.204061 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196192    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:05:54.204580 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196195    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:05:54.204580 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196197    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:05:54.204580 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196200    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:05:54.204580 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196203    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:05:54.204580 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196206    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:05:54.204580 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196208    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:05:54.204580 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196211    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:05:54.204580 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196214    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:05:54.204580 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196216    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:05:54.204580 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196219    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:05:54.204580 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196222    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:05:54.204580 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196224    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:05:54.204580 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196227    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:05:54.204580 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196230    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:05:54.204580 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196232    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:05:54.204580 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196235    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:05:54.204580 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196238    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:05:54.204580 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196240    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:05:54.204580 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196243    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:05:54.204580 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196245    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:05:54.205046 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196248    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:05:54.205046 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196250    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:05:54.205046 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196266    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:05:54.205046 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196270    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:05:54.205046 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196273    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:05:54.205046 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196276    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:05:54.205046 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196279    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:05:54.205046 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196282    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:05:54.205046 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196284    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:05:54.205046 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196288    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:05:54.205046 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196291    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:05:54.205046 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196295    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:05:54.205046 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196299    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:05:54.205046 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196302    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:05:54.205046 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196305    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:05:54.205046 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.196332    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 03:05:54.205418 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196371    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:05:54.205418 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196376    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:05:54.205418 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196379    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:05:54.205418 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196381    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:05:54.205418 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196384    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:05:54.205418 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196387    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:05:54.205418 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196389    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:05:54.205418 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196392    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:05:54.205418 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196395    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:05:54.205418 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196398    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:05:54.205418 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196401    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:05:54.205418 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196403    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:05:54.205418 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196406    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:05:54.205418 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196408    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:05:54.205418 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196411    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:05:54.205418 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196414    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:05:54.205418 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196416    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:05:54.205418 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196419    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:05:54.205418 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196422    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:05:54.205418 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196424    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:05:54.205895 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196427    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:05:54.205895 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196429    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:05:54.205895 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196432    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:05:54.205895 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196434    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:05:54.205895 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196438    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:05:54.205895 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196442    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:05:54.205895 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196448    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:05:54.205895 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196452    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:05:54.205895 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196455    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:05:54.205895 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196457    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:05:54.205895 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196460    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:05:54.205895 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196463    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:05:54.205895 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196466    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:05:54.205895 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196469    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:05:54.205895 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196472    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:05:54.205895 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196474    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:05:54.205895 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196477    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:05:54.205895 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196479    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:05:54.205895 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196483    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:05:54.206347 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196487    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:05:54.206347 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196490    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:05:54.206347 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196493    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:05:54.206347 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196496    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:05:54.206347 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196498    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:05:54.206347 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196501    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:05:54.206347 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196504    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:05:54.206347 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196506    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:05:54.206347 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196509    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:05:54.206347 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196511    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:05:54.206347 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196514    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:05:54.206347 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196517    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:05:54.206347 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196519    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:05:54.206347 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196522    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:05:54.206347 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196525    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:05:54.206347 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196529    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:05:54.206347 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196532    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:05:54.206347 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196534    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:05:54.206347 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196538    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:05:54.206347 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196542    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:05:54.206821 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196545    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:05:54.206821 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196548    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:05:54.206821 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196550    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:05:54.206821 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196566    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:05:54.206821 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196569    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:05:54.206821 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196572    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:05:54.206821 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196575    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:05:54.206821 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196579    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:05:54.206821 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196581    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:05:54.206821 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196584    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:05:54.206821 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196587    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:05:54.206821 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196589    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:05:54.206821 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196592    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:05:54.206821 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196595    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:05:54.206821 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196597    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:05:54.206821 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196600    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:05:54.206821 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196603    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:05:54.206821 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196605    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:05:54.206821 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196608    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:05:54.206821 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196610    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:05:54.207305 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196613    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:05:54.207305 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196616    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:05:54.207305 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196618    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:05:54.207305 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196621    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:05:54.207305 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196624    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:05:54.207305 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196627    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:05:54.207305 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196629    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:05:54.207305 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196633    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:05:54.207305 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196636    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:05:54.207305 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196638    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:05:54.207305 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196641    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:05:54.207305 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196645    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:05:54.207305 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196649    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:05:54.207305 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196652    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:05:54.207305 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196654    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:05:54.207305 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196657    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:05:54.207305 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196660    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:05:54.207305 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196662    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:05:54.207305 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196665    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:05:54.207813 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196667    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:05:54.207813 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196671    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:05:54.207813 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196673    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:05:54.207813 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196676    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:05:54.207813 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196679    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:05:54.207813 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196681    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:05:54.207813 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196684    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:05:54.207813 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.196686    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:05:54.207813 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.202947    2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 03:05:54.207813 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.202963    2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 03:05:54.207813 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.202980    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 03:05:54.207813 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.202984    2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 03:05:54.207813 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203038    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:05:54.207813 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203042    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:05:54.207813 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203047    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:05:54.207813 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203051    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:05:54.207813 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203054    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:05:54.207813 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203057    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:05:54.207813 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203060    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:05:54.208252 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203062    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:05:54.208252 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203065    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:05:54.208252 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203068    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:05:54.208252 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203070    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:05:54.208252 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203073    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:05:54.208252 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203076    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:05:54.208252 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203079    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:05:54.208252 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203082    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:05:54.208252 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203084    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:05:54.208252 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203087    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:05:54.208252 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203089    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:05:54.208252 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203092    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:05:54.208252 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203094    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:05:54.208252 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203097    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:05:54.208252 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203100    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:05:54.208252 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203103    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:05:54.208252 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203106    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:05:54.208252 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203108    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:05:54.208252 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203111    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:05:54.208252 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203113    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:05:54.208718 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203116    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:05:54.208718 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203119    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:05:54.208718 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203123    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:05:54.208718 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203125    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:05:54.208718 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203128    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:05:54.208718 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203131    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:05:54.208718 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203134    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:05:54.208718 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203136    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:05:54.208718 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203139    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:05:54.208718 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203142    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:05:54.208718 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203145    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:05:54.208718 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203148    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:05:54.208718 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203150    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:05:54.208718 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203153    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:05:54.208718 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203155    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:05:54.208718 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203158    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:05:54.208718 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203161    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:05:54.208718 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203163    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:05:54.208718 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203166    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:05:54.208718 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203171    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:05:54.209181 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203175    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:05:54.209181 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203178    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:05:54.209181 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203181    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:05:54.209181 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203184    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:05:54.209181 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203187    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:05:54.209181 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203190    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:05:54.209181 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203192    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:05:54.209181 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203195    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:05:54.209181 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203198    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:05:54.209181 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203200    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:05:54.209181 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203203    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:05:54.209181 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203207    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:05:54.209181 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203211    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:05:54.209181 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203214    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:05:54.209181 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203218    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:05:54.209181 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203221    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:05:54.209181 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203223    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:05:54.209181 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203226    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:05:54.209181 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203228    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:05:54.209630 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203231    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:05:54.209630 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203234    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:05:54.209630 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203236    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:05:54.209630 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203239    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:05:54.209630 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203241    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:05:54.209630 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203244    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:05:54.209630 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203246    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:05:54.209630 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203249    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:05:54.209630 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203252    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:05:54.209630 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203254    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:05:54.209630 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203257    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:05:54.209630 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203259    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:05:54.209630 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203262    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:05:54.209630 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203264    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:05:54.209630 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203267    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:05:54.209630 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203269    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:05:54.209630 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203272    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:05:54.209630 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203275    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:05:54.209630 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203277    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:05:54.209630 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203280    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:05:54.210125 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203282    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:05:54.210125 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203286    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:05:54.210125 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203289    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:05:54.210125 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203291    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:05:54.210125 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203294    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:05:54.210125 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203297    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:05:54.210125 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203299    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:05:54.210125 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203302    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:05:54.210125 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203305    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:05:54.210125 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203307    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:05:54.210125 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203310    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:05:54.210125 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203312    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:05:54.210125 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203315    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:05:54.210125 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203318    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:05:54.210125 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203320    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:05:54.210125 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203323    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:05:54.210125 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203325    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:05:54.210125 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203328    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:05:54.210125 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203330    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:05:54.210125 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203333    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:05:54.210604 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.203359    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 03:05:54.210604 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203509    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:05:54.210604 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203514    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:05:54.210604 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203517    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:05:54.210604 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203520    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:05:54.210604 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203523    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:05:54.210604 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203525    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:05:54.210604 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203528    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:05:54.210604 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203531    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:05:54.210604 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203533    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:05:54.210604 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203536    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:05:54.210604 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203538    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:05:54.210604 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203541    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:05:54.210604 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203544    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:05:54.210604 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203547    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:05:54.210604 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203549    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:05:54.210973 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203552    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:05:54.210973 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203570    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:05:54.210973 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203573    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:05:54.210973 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203576    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:05:54.210973 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203578    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:05:54.210973 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203581    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:05:54.210973 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203584    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:05:54.210973 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203586    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:05:54.210973 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203589    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:05:54.210973 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203591    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:05:54.210973 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203594    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:05:54.210973 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203597    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:05:54.210973 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203599    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:05:54.210973 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203602    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:05:54.210973 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203605    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:05:54.210973 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203607    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:05:54.210973 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203609    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:05:54.210973 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203612    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:05:54.210973 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203614    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:05:54.211411 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203617    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:05:54.211411 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203619    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:05:54.211411 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203622    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:05:54.211411 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203624    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:05:54.211411 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203626    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:05:54.211411 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203629    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:05:54.211411 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203631    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:05:54.211411 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203634    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:05:54.211411 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203636    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:05:54.211411 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203639    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:05:54.211411 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203641    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:05:54.211411 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203644    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:05:54.211411 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203647    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:05:54.211411 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203649    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:05:54.211411 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203652    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:05:54.211411 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203654    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:05:54.211411 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203657    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:05:54.211411 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203660    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:05:54.211411 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203662    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:05:54.211903 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203665    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:05:54.211903 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203667    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:05:54.211903 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203670    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:05:54.211903 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203672    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:05:54.211903 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203675    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:05:54.211903 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203677    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:05:54.211903 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203680    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:05:54.211903 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203682    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:05:54.211903 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203685    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:05:54.211903 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203687    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:05:54.211903 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203690    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:05:54.211903 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203692    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:05:54.211903 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203695    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:05:54.211903 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203697    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:05:54.211903 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203700    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:05:54.211903 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203702    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:05:54.211903 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203705    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:05:54.211903 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203709    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:05:54.211903 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203712    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:05:54.212344 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203716    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:05:54.212344 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203719    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:05:54.212344 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203722    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:05:54.212344 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203725    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:05:54.212344 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203728    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:05:54.212344 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203730    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:05:54.212344 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203733    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:05:54.212344 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203736    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:05:54.212344 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203738    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:05:54.212344 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203741    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:05:54.212344 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203744    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:05:54.212344 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203746    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:05:54.212344 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203749    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:05:54.212344 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203751    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:05:54.212344 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203754    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:05:54.212344 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203756    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:05:54.212344 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203759    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:05:54.212344 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203762    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:05:54.212344 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203764    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:05:54.212344 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203767    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:05:54.212885 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203769    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:05:54.212885 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203772    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:05:54.212885 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203774    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:05:54.212885 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203777    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:05:54.212885 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203779    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:05:54.212885 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203782    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:05:54.212885 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203785    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:05:54.212885 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203787    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:05:54.212885 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203789    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:05:54.212885 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203792    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:05:54.212885 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203794    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:05:54.212885 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203798    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:05:54.212885 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203801    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:05:54.212885 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203804    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:05:54.212885 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.203828    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 03:05:54.212885 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203858    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:05:54.213495 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203863    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:05:54.213495 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203865    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:05:54.213495 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203868    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:05:54.213495 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203871    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:05:54.213495 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203874    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:05:54.213495 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203877    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:05:54.213495 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203880    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:05:54.213495 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203882    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:05:54.213495 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203885    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:05:54.213495 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203887    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:05:54.213495 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203890    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:05:54.213495 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203892    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:05:54.213495 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203895    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:05:54.213495 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203898    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:05:54.213495 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203900    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:05:54.213495 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203903    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:05:54.213495 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203905    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:05:54.213495 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203908    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:05:54.213495 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203910    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:05:54.213495 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203913    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:05:54.214186 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203915    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:05:54.214186 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203918    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:05:54.214186 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203921    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:05:54.214186 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203925    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:05:54.214186 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203928    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:05:54.214186 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203931    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:05:54.214186 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203934    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:05:54.214186 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203936    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:05:54.214186 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203939    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:05:54.214186 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203941    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:05:54.214186 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203944    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:05:54.214186 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203947    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:05:54.214186 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203949    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:05:54.214186 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203952    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:05:54.214186 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203954    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:05:54.214186 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203957    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:05:54.214186 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203959    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:05:54.214186 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203963    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:05:54.214186 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203966    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:05:54.214708 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203968    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:05:54.214708 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203971    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:05:54.214708 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203973    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:05:54.214708 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203976    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:05:54.214708 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203978    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:05:54.214708 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203981    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:05:54.214708 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203983    2578 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:05:54.214708 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203986    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:05:54.214708 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203989    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:05:54.214708 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203991    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:05:54.214708 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203994    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:05:54.214708 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203996    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:05:54.214708 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.203999    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:05:54.214708 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204001    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:05:54.214708 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204004    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:05:54.214708 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204006    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:05:54.214708 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204009    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:05:54.214708 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204011    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:05:54.214708 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204014    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:05:54.214708 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204016    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:05:54.215178 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204019    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:05:54.215178 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204021    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:05:54.215178 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204024    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:05:54.215178 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204026    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:05:54.215178 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204029    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:05:54.215178 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204031    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:05:54.215178 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204034    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:05:54.215178 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204036    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:05:54.215178 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204039    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:05:54.215178 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204041    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:05:54.215178 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204046    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:05:54.215178 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204049    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:05:54.215178 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204052    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:05:54.215178 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204055    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:05:54.215178 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204057    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:05:54.215178 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204059    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:05:54.215178 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204062    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:05:54.215178 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204065    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:05:54.215178 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204068    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:05:54.215697 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204070    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:05:54.215697 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204073    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:05:54.215697 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204076    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:05:54.215697 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204078    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:05:54.215697 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204081    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:05:54.215697 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204083    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:05:54.215697 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204086    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:05:54.215697 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204088    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:05:54.215697 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204091    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:05:54.215697 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204093    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:05:54.215697 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204097    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:05:54.215697 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204099    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:05:54.215697 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204102    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:05:54.215697 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204104    2578 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:05:54.215697 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204107    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:05:54.215697 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204110    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:05:54.215697 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204112    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:05:54.215697 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204116    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:05:54.215697 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204120    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:05:54.216274 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204124    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:05:54.216274 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204127    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:05:54.216274 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204130    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:05:54.216274 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204132    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:05:54.216274 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204137    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:05:54.216274 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204140    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:05:54.216274 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204147    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:05:54.216274 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:54.204149    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:05:54.216274 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.204196    2578 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 03:05:54.216274 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.208535    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 01 03:05:54.216274 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.209519    2578 server.go:1014] "Starting client certificate rotation"
Nov 01 03:05:54.216274 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.209584    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 03:05:54.216274 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.209638    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 03:05:54.216274 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.211940    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 03:05:54.217977 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.217958    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 03:05:54.228883 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.228866    2578 log.go:25] "Validated CRI v1 runtime API"
Nov 01 03:05:54.231975 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.231963    2578 log.go:25] "Validated CRI v1 image API"
Nov 01 03:05:54.232749 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.232722    2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 03:05:54.235784 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.235762    2578 fs.go:135] Filesystem UUIDs: map[3c16cd3a-e64a-47c3-8190-467f00fdb4dc:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ac5ba056-168c-4877-a711-d4cf6e474e80:/dev/nvme0n1p4]
Nov 01 03:05:54.235828 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.235784    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 01 03:05:54.240679 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.240581    2578 manager.go:217] Machine: {Timestamp:2026-11-01 03:05:54.23928144 +0000 UTC m=+0.350533738 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100457 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b890a10dee2b78d4631eafe98a290 SystemUUID:ec2b890a-10de-e2b7-8d46-31eafe98a290 BootID:5c802453-d661-409f-b139-7fbd23e8d7cf Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:7e:47:ef:38:99 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:7e:47:ef:38:99 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3a:1b:7c:92:45:95 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 03:05:54.240679 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.240678    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 03:05:54.240785 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.240745    2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 03:05:54.241956 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.241932    2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 03:05:54.242077 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.241957    2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-122.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 03:05:54.242144 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.242085    2578 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 03:05:54.242144 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.242094    2578 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 03:05:54.242144 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.242130    2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 03:05:54.242223 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.242147    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 03:05:54.242919 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.242910    2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 03:05:54.246403 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.246391    2578 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 03:05:54.246520 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.246510    2578 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 03:05:54.247215 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.247206    2578 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 03:05:54.247244 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.247222    2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 03:05:54.247244 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.247233    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 03:05:54.247244 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.247244    2578 kubelet.go:405] "Adding apiserver pod source"
Nov 01 03:05:54.247353 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.247252    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 03:05:54.247579 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.247548    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 03:05:54.250117 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.250096    2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 03:05:54.254876 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.254861    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 03:05:54.255740 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.255728    2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 03:05:54.255777 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.255758    2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 03:05:54.255913 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.255903    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 03:05:54.255945 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.255918    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 03:05:54.255945 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.255924    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 03:05:54.255945 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.255930    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 03:05:54.255945 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.255942    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 03:05:54.256044 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.255950    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 03:05:54.256775 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.256766    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 03:05:54.256807 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.256776    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 03:05:54.256807 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.256783    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 03:05:54.256807 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.256789    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 03:05:54.256807 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.256803    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 03:05:54.257034 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.256812    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 03:05:54.257479 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.257470    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 03:05:54.257479 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.257479    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 03:05:54.261121 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.261107    2578 server.go:1263] "Started kubelet"
Nov 01 03:05:54.261226 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.261183    2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 03:05:54.261285 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.261200    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 03:05:54.261285 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.261272    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 03:05:54.261808 ip-10-0-141-122 systemd[1]: Started Kubernetes Kubelet.
Nov 01 03:05:54.263723 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.263699    2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 03:05:54.268573 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.266923    2578 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 03:05:54.270587 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.270548    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 03:05:54.270847 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:54.270815    2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 01 03:05:54.270917 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.270909    2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 03:05:54.271065 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:54.270992    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-122.us-west-1.compute.internal\" not found"
Nov 01 03:05:54.271496 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.271475    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 03:05:54.271496 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.271475    2578 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 03:05:54.271630 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.271504    2578 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 03:05:54.271702 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.271655    2578 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 03:05:54.271702 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.271693    2578 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 03:05:54.271994 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.271980    2578 factory.go:153] Registering CRI-O factory
Nov 01 03:05:54.272048 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.272014    2578 factory.go:223] Registration of the crio container factory successfully
Nov 01 03:05:54.272078 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.272053    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 03:05:54.272078 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.272060    2578 factory.go:55] Registering systemd factory
Nov 01 03:05:54.272078 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.272065    2578 factory.go:223] Registration of the systemd container factory successfully
Nov 01 03:05:54.272078 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.272076    2578 factory.go:103] Registering Raw factory
Nov 01 03:05:54.272197 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.272085    2578 manager.go:1199] Started watching for new ooms in manager
Nov 01 03:05:54.272924 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.272909    2578 manager.go:319] Starting recovery of all containers
Nov 01 03:05:54.282283 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.282265    2578 manager.go:324] Recovery completed
Nov 01 03:05:54.287126 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.287109    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 03:05:54.288961 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.288925    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 03:05:54.289038 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.288988    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 03:05:54.289038 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.289008    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 03:05:54.290247 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.289582    2578 cpu_manager.go:226] "Starting" policy="none"
Nov 01 03:05:54.290247 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.289593    2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 03:05:54.290247 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.289612    2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 03:05:54.290247 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.289586    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 03:05:54.290540 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.290309    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 03:05:54.290540 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.290335    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 03:05:54.290540 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.290344    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 03:05:54.371800 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:54.371772    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-122.us-west-1.compute.internal\" not found"
Nov 01 03:05:54.433360 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.416049    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 03:05:54.433360 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.417195    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 03:05:54.433360 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.417221    2578 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 03:05:54.433360 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.417242    2578 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 03:05:54.433360 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:54.417282    2578 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 01 03:05:54.433360 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.432167    2578 policy_none.go:50] "Start"
Nov 01 03:05:54.433360 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.432187    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 03:05:54.433360 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.432203    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 03:05:54.436774 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.433890    2578 policy_none.go:44] "Start"
Nov 01 03:05:54.457751 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.457618    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-122.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 01 03:05:54.457751 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:54.457625    2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-122.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 01 03:05:54.458838 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:54.457658    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-122.us-west-1.compute.internal.18e3cce3898efb54  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-122.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-122.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 03:05:54.261080916 +0000 UTC m=+0.372333214,LastTimestamp:2026-11-01 03:05:54.261080916 +0000 UTC m=+0.372333214,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-122.us-west-1.compute.internal,}"
Nov 01 03:05:54.471849 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:54.471828    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-122.us-west-1.compute.internal\" not found"
Nov 01 03:05:54.475127 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.475103    2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 03:05:54.477342 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.477326    2578 manager.go:341] "Starting Device Plugin manager"
Nov 01 03:05:54.477425 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:54.477357    2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 03:05:54.477425 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.477369    2578 server.go:87] "Starting device plugin registration server"
Nov 01 03:05:54.477602 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.477589    2578 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 03:05:54.477670 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.477601    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 03:05:54.477728 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.477695    2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 03:05:54.477796 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.477785    2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 03:05:54.477853 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.477804    2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 03:05:54.478278 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:54.478257    2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 03:05:54.478348 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:54.478300    2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-122.us-west-1.compute.internal\" not found"
Nov 01 03:05:54.495702 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.495682    2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 03:05:54.495771 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.495733    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-59j5k"
Nov 01 03:05:54.518232 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.518210    2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-122.us-west-1.compute.internal"]
Nov 01 03:05:54.518326 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.518269    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 03:05:54.519021 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.519007    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 03:05:54.519085 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.519033    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 03:05:54.519085 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.519043    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 03:05:54.521164 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.521152    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 03:05:54.521333 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.521306    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal"
Nov 01 03:05:54.521401 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.521335    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 03:05:54.521889 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.521869    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 03:05:54.521984 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.521872    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 03:05:54.521984 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.521899    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 03:05:54.521984 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.521915    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 03:05:54.521984 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.521923    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 03:05:54.521984 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.521947    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 03:05:54.524419 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.524405    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-122.us-west-1.compute.internal"
Nov 01 03:05:54.524486 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.524427    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 03:05:54.525082 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.525068    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 03:05:54.525158 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.525095    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 03:05:54.525158 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.525109    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 03:05:54.541716 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:54.541700    2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-122.us-west-1.compute.internal\" not found" node="ip-10-0-141-122.us-west-1.compute.internal"
Nov 01 03:05:54.544267 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:54.544248    2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-122.us-west-1.compute.internal\" not found" node="ip-10-0-141-122.us-west-1.compute.internal"
Nov 01 03:05:54.572817 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.572794    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bbcea9cdf04d326735591dc246925152-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal\" (UID: \"bbcea9cdf04d326735591dc246925152\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal"
Nov 01 03:05:54.572910 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.572819    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbcea9cdf04d326735591dc246925152-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal\" (UID: \"bbcea9cdf04d326735591dc246925152\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal"
Nov 01 03:05:54.572910 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.572836    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5391522643ff9514bed3da110ddd8b71-config\") pod \"kube-apiserver-proxy-ip-10-0-141-122.us-west-1.compute.internal\" (UID: \"5391522643ff9514bed3da110ddd8b71\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-122.us-west-1.compute.internal"
Nov 01 03:05:54.578096 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.578082    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 03:05:54.578788 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.578774    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 03:05:54.578864 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.578802    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 03:05:54.578864 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.578815    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 03:05:54.578864 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.578844    2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-122.us-west-1.compute.internal"
Nov 01 03:05:54.583542 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.583526    2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 03:05:54.673253 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.673225    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bbcea9cdf04d326735591dc246925152-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal\" (UID: \"bbcea9cdf04d326735591dc246925152\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal"
Nov 01 03:05:54.673253 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.673258    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbcea9cdf04d326735591dc246925152-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal\" (UID: \"bbcea9cdf04d326735591dc246925152\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal"
Nov 01 03:05:54.673364 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.673275    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5391522643ff9514bed3da110ddd8b71-config\") pod \"kube-apiserver-proxy-ip-10-0-141-122.us-west-1.compute.internal\" (UID: \"5391522643ff9514bed3da110ddd8b71\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-122.us-west-1.compute.internal"
Nov 01 03:05:54.673364 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.673326    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbcea9cdf04d326735591dc246925152-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal\" (UID: \"bbcea9cdf04d326735591dc246925152\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal"
Nov 01 03:05:54.673364 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.673329    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bbcea9cdf04d326735591dc246925152-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal\" (UID: \"bbcea9cdf04d326735591dc246925152\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal"
Nov 01 03:05:54.673454 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.673385    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5391522643ff9514bed3da110ddd8b71-config\") pod \"kube-apiserver-proxy-ip-10-0-141-122.us-west-1.compute.internal\" (UID: \"5391522643ff9514bed3da110ddd8b71\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-122.us-west-1.compute.internal"
Nov 01 03:05:54.676729 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.676712    2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 03:05:54.677363 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.677349    2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 03:05:54.678543 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.678529    2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 03:05:54.682017 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.682003    2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-122.us-west-1.compute.internal"
Nov 01 03:05:54.682077 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:54.682023    2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-122.us-west-1.compute.internal\": node \"ip-10-0-141-122.us-west-1.compute.internal\" not found"
Nov 01 03:05:54.771247 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.771212    2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-122.us-west-1.compute.internal"
Nov 01 03:05:54.841676 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.841662    2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 03:05:54.842336 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.842325    2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal"
Nov 01 03:05:54.844821 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.844808    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal"
Nov 01 03:05:54.847782 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.847768    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-122.us-west-1.compute.internal"
Nov 01 03:05:54.910771 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:54.910750    2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 03:05:55.209795 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.209767    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 01 03:05:55.210335 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.209928    2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 01 03:05:55.210335 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.209974    2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 01 03:05:55.210335 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.209998    2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 01 03:05:55.211120 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:55.209980    2578 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a28e94d7245cf43b9a1c835832cbb3a3-03ecbaf5388fcb74.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-141-122.us-west-1.compute.internal.18e3cce38b39376f\": read tcp 10.0.141.122:43162->52.15.232.74:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-141-122.us-west-1.compute.internal.18e3cce38b39376f  default   5849 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-122.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-141-122.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-141-122.us-west-1.compute.internal,},FirstTimestamp:2026-11-01 03:05:54 +0000 UTC,LastTimestamp:2026-11-01 03:05:54.519046908 +0000 UTC m=+0.630299209,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-122.us-west-1.compute.internal,}"
Nov 01 03:05:55.249034 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.249002    2578 apiserver.go:52] "Watching apiserver"
Nov 01 03:05:55.270722 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.270697    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 01 03:05:55.500577 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.500481    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 03:00:54 +0000 UTC" deadline="2028-06-16 11:50:04.141496688 +0000 UTC"
Nov 01 03:05:55.500577 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.500513    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14240h44m8.640986437s"
Nov 01 03:05:55.547538 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.547511    2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 03:05:55.547853 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.547834    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-p5ght","openshift-ovn-kubernetes/ovnkube-node-h5ksq","kube-system/konnectivity-agent-62ccn","kube-system/kube-apiserver-proxy-ip-10-0-141-122.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk","openshift-image-registry/node-ca-stpgr","kube-system/global-pull-secret-syncer-4bvb6","openshift-cluster-node-tuning-operator/tuned-l892q","openshift-multus/network-metrics-daemon-mzkbj","openshift-network-diagnostics/network-check-target-62mxc","openshift-network-operator/iptables-alerter-ksj2n","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-9nhh6","openshift-dns/node-resolver-77nf7"]
Nov 01 03:05:55.550489 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.550465    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-77nf7"
Nov 01 03:05:55.553446 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.553428    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.555884 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.555866    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bvb6"
Nov 01 03:05:55.555968 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:55.555942    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bvb6" podUID="a972ee28-29fa-4755-adcd-b40d476c22f0"
Nov 01 03:05:55.558130 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.558116    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk"
Nov 01 03:05:55.560351 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.560335    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-stpgr"
Nov 01 03:05:55.562540 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.562526    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.565592 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.565574    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.567881 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.567864    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzkbj"
Nov 01 03:05:55.567953 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:55.567929    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzkbj" podUID="0a66c33f-29bc-4920-9d7a-0c45215514a0"
Nov 01 03:05:55.570190 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.570173    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62mxc"
Nov 01 03:05:55.570261 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:55.570243    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62mxc" podUID="e09a58a7-120e-4ec5-9aba-f15863267643"
Nov 01 03:05:55.572439 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.572420    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ksj2n"
Nov 01 03:05:55.575067 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.575052    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9nhh6"
Nov 01 03:05:55.577425 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.577409    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-62ccn"
Nov 01 03:05:55.579998 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.579980    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjt9z\" (UniqueName: \"kubernetes.io/projected/73622c04-1428-4f7a-8faf-8eb17604f681-kube-api-access-hjt9z\") pod \"node-ca-stpgr\" (UID: \"73622c04-1428-4f7a-8faf-8eb17604f681\") " pod="openshift-image-registry/node-ca-stpgr"
Nov 01 03:05:55.580050 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.580009    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-var-lib-kubelet\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.580050 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.580032    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.580156 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.580062    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a66c33f-29bc-4920-9d7a-0c45215514a0-metrics-certs\") pod \"network-metrics-daemon-mzkbj\" (UID: \"0a66c33f-29bc-4920-9d7a-0c45215514a0\") " pod="openshift-multus/network-metrics-daemon-mzkbj"
Nov 01 03:05:55.580211 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.580167    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q546p\" (UniqueName: \"kubernetes.io/projected/8190fea3-4e71-4250-9a0f-3a1d78dd32db-kube-api-access-q546p\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.580261 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.580207    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbfdt\" (UniqueName: \"kubernetes.io/projected/ca0be51d-87a4-462d-9c83-fcfb8bf224e8-kube-api-access-cbfdt\") pod \"iptables-alerter-ksj2n\" (UID: \"ca0be51d-87a4-462d-9c83-fcfb8bf224e8\") " pod="openshift-network-operator/iptables-alerter-ksj2n"
Nov 01 03:05:55.580261 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.580236    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-var-lib-openvswitch\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.580327 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.580272    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-host\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.580358 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.580331    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-systemd-units\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.580393 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.580374    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-etc-openvswitch\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.580426 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.580405    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/998efdf0-7719-4768-bfd0-3da04d85d108-registration-dir\") pod \"aws-ebs-csi-driver-node-rjtwk\" (UID: \"998efdf0-7719-4768-bfd0-3da04d85d108\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk"
Nov 01 03:05:55.580459 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.580425    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-etc-sysctl-d\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.580494 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.580473    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-etc-systemd\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.580527 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.580510    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8190fea3-4e71-4250-9a0f-3a1d78dd32db-tmp\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.580563 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.580526    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-run-systemd\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.580596 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.580561    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9e212e37-6c34-4b3c-941c-b38621f1f184-env-overrides\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.580649 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.580615    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-host-var-lib-cni-multus\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.580649 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.580634    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnmwf\" (UniqueName: \"kubernetes.io/projected/0a66c33f-29bc-4920-9d7a-0c45215514a0-kube-api-access-xnmwf\") pod \"network-metrics-daemon-mzkbj\" (UID: \"0a66c33f-29bc-4920-9d7a-0c45215514a0\") " pod="openshift-multus/network-metrics-daemon-mzkbj"
Nov 01 03:05:55.580714 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.580655    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lbzv\" (UniqueName: \"kubernetes.io/projected/e09a58a7-120e-4ec5-9aba-f15863267643-kube-api-access-7lbzv\") pod \"network-check-target-62mxc\" (UID: \"e09a58a7-120e-4ec5-9aba-f15863267643\") " pod="openshift-network-diagnostics/network-check-target-62mxc"
Nov 01 03:05:55.580714 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.580688    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-node-log\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.580775 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.580716    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-multus-cni-dir\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.580775 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.580732    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-host-slash\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.580839 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.580765    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-host-run-k8s-cni-cncf-io\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.580839 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.580793    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/998efdf0-7719-4768-bfd0-3da04d85d108-sys-fs\") pod \"aws-ebs-csi-driver-node-rjtwk\" (UID: \"998efdf0-7719-4768-bfd0-3da04d85d108\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk"
Nov 01 03:05:55.580839 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.580810    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-host-run-netns\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.580930 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.580847    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/73622c04-1428-4f7a-8faf-8eb17604f681-host\") pod \"node-ca-stpgr\" (UID: \"73622c04-1428-4f7a-8faf-8eb17604f681\") " pod="openshift-image-registry/node-ca-stpgr"
Nov 01 03:05:55.580930 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.580875    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-run-openvswitch\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.580930 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.580893    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-etc-modprobe-d\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.580930 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.580919    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-etc-sysconfig\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.581048 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.580963    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca0be51d-87a4-462d-9c83-fcfb8bf224e8-host-slash\") pod \"iptables-alerter-ksj2n\" (UID: \"ca0be51d-87a4-462d-9c83-fcfb8bf224e8\") " pod="openshift-network-operator/iptables-alerter-ksj2n"
Nov 01 03:05:55.581080 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.581046    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-host-run-netns\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.581115 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.581088    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9e212e37-6c34-4b3c-941c-b38621f1f184-ovnkube-config\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.581152 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.581125    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-system-cni-dir\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.581185 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.581150    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-hostroot\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.581185 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.581173    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-etc-sysctl-conf\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.581250 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.581224    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-run\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.581280 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.581253    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-lib-modules\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.581312 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.581281    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-run-ovn\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.581312 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.581300    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwpzw\" (UniqueName: \"kubernetes.io/projected/9e212e37-6c34-4b3c-941c-b38621f1f184-kube-api-access-pwpzw\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.581372 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.581330    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-host-run-multus-certs\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.581407 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.581369    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-etc-kubernetes\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.581442 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.581401    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ca0be51d-87a4-462d-9c83-fcfb8bf224e8-iptables-alerter-script\") pod \"iptables-alerter-ksj2n\" (UID: \"ca0be51d-87a4-462d-9c83-fcfb8bf224e8\") " pod="openshift-network-operator/iptables-alerter-ksj2n"
Nov 01 03:05:55.581442 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.581428    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-host-kubelet\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.581511 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.581451    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-host-run-ovn-kubernetes\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.581511 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.581479    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a972ee28-29fa-4755-adcd-b40d476c22f0-dbus\") pod \"global-pull-secret-syncer-4bvb6\" (UID: \"a972ee28-29fa-4755-adcd-b40d476c22f0\") " pod="kube-system/global-pull-secret-syncer-4bvb6"
Nov 01 03:05:55.581612 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.581557    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfxfn\" (UniqueName: \"kubernetes.io/projected/998efdf0-7719-4768-bfd0-3da04d85d108-kube-api-access-qfxfn\") pod \"aws-ebs-csi-driver-node-rjtwk\" (UID: \"998efdf0-7719-4768-bfd0-3da04d85d108\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk"
Nov 01 03:05:55.581612 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.581604    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-etc-kubernetes\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.581754 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.581648    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/73622c04-1428-4f7a-8faf-8eb17604f681-serviceca\") pod \"node-ca-stpgr\" (UID: \"73622c04-1428-4f7a-8faf-8eb17604f681\") " pod="openshift-image-registry/node-ca-stpgr"
Nov 01 03:05:55.581754 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.581674    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9e212e37-6c34-4b3c-941c-b38621f1f184-ovnkube-script-lib\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.581754 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.581692    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-multus-daemon-config\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.581962 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.581794    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-host-cni-netd\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.581962 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.581828    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-cnibin\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.581962 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.581852    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkm29\" (UniqueName: \"kubernetes.io/projected/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-kube-api-access-vkm29\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.581962 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.581881    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-sys\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.581962 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.581905    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8190fea3-4e71-4250-9a0f-3a1d78dd32db-etc-tuned\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.581962 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.581925    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-log-socket\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.581962 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.581964    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/998efdf0-7719-4768-bfd0-3da04d85d108-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rjtwk\" (UID: \"998efdf0-7719-4768-bfd0-3da04d85d108\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk"
Nov 01 03:05:55.582169 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.581979    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-multus-conf-dir\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.582169 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.582028    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a972ee28-29fa-4755-adcd-b40d476c22f0-kubelet-config\") pod \"global-pull-secret-syncer-4bvb6\" (UID: \"a972ee28-29fa-4755-adcd-b40d476c22f0\") " pod="kube-system/global-pull-secret-syncer-4bvb6"
Nov 01 03:05:55.582169 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.582049    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/998efdf0-7719-4768-bfd0-3da04d85d108-socket-dir\") pod \"aws-ebs-csi-driver-node-rjtwk\" (UID: \"998efdf0-7719-4768-bfd0-3da04d85d108\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk"
Nov 01 03:05:55.582169 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.582064    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/998efdf0-7719-4768-bfd0-3da04d85d108-device-dir\") pod \"aws-ebs-csi-driver-node-rjtwk\" (UID: \"998efdf0-7719-4768-bfd0-3da04d85d108\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk"
Nov 01 03:05:55.582169 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.582080    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/998efdf0-7719-4768-bfd0-3da04d85d108-etc-selinux\") pod \"aws-ebs-csi-driver-node-rjtwk\" (UID: \"998efdf0-7719-4768-bfd0-3da04d85d108\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk"
Nov 01 03:05:55.582169 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.582094    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/482aebaa-cb7f-4be2-991a-7fb4cd801c7f-hosts-file\") pod \"node-resolver-77nf7\" (UID: \"482aebaa-cb7f-4be2-991a-7fb4cd801c7f\") " pod="openshift-dns/node-resolver-77nf7"
Nov 01 03:05:55.582169 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.582122    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/482aebaa-cb7f-4be2-991a-7fb4cd801c7f-tmp-dir\") pod \"node-resolver-77nf7\" (UID: \"482aebaa-cb7f-4be2-991a-7fb4cd801c7f\") " pod="openshift-dns/node-resolver-77nf7"
Nov 01 03:05:55.582382 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.582172    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm7m6\" (UniqueName: \"kubernetes.io/projected/482aebaa-cb7f-4be2-991a-7fb4cd801c7f-kube-api-access-mm7m6\") pod \"node-resolver-77nf7\" (UID: \"482aebaa-cb7f-4be2-991a-7fb4cd801c7f\") " pod="openshift-dns/node-resolver-77nf7"
Nov 01 03:05:55.582382 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.582219    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-multus-socket-dir-parent\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.582382 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.582259    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-host-var-lib-cni-bin\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.582382 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.582291    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-host-cni-bin\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.582382 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.582309    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9e212e37-6c34-4b3c-941c-b38621f1f184-ovn-node-metrics-cert\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.582382 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.582335    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a972ee28-29fa-4755-adcd-b40d476c22f0-original-pull-secret\") pod \"global-pull-secret-syncer-4bvb6\" (UID: \"a972ee28-29fa-4755-adcd-b40d476c22f0\") " pod="kube-system/global-pull-secret-syncer-4bvb6"
Nov 01 03:05:55.582382 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.582371    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-cni-binary-copy\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.582671 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.582390    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-host-var-lib-kubelet\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.614375 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.611504    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 03:05:55.614375 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.611667    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-77n5m\""
Nov 01 03:05:55.614375 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.611959    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 03:05:55.668759 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.668739    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 03:05:55.672356 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.672340    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 03:05:55.682769 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.682742    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-multus-socket-dir-parent\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.682880 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.682785    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-host-var-lib-cni-bin\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.682880 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.682818    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1d8dba89-5673-4cd3-a6ac-914765f228d6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9nhh6\" (UID: \"1d8dba89-5673-4cd3-a6ac-914765f228d6\") " pod="openshift-multus/multus-additional-cni-plugins-9nhh6"
Nov 01 03:05:55.682880 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.682847    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-host-cni-bin\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.682880 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.682872    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9e212e37-6c34-4b3c-941c-b38621f1f184-ovn-node-metrics-cert\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.683097 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.682878    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-host-var-lib-cni-bin\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.683097 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.682898    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a972ee28-29fa-4755-adcd-b40d476c22f0-original-pull-secret\") pod \"global-pull-secret-syncer-4bvb6\" (UID: \"a972ee28-29fa-4755-adcd-b40d476c22f0\") " pod="kube-system/global-pull-secret-syncer-4bvb6"
Nov 01 03:05:55.683097 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.682916    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-multus-socket-dir-parent\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.683097 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.682942    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-cni-binary-copy\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.683097 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.682940    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-host-cni-bin\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.683097 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.682982    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-host-var-lib-kubelet\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.683097 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.683014    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjt9z\" (UniqueName: \"kubernetes.io/projected/73622c04-1428-4f7a-8faf-8eb17604f681-kube-api-access-hjt9z\") pod \"node-ca-stpgr\" (UID: \"73622c04-1428-4f7a-8faf-8eb17604f681\") " pod="openshift-image-registry/node-ca-stpgr"
Nov 01 03:05:55.683097 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.683032    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-host-var-lib-kubelet\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.683097 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:55.683044    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 03:05:55.683097 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.683045    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-var-lib-kubelet\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.683097 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.683090    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-var-lib-kubelet\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.683649 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:55.683144    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a972ee28-29fa-4755-adcd-b40d476c22f0-original-pull-secret podName:a972ee28-29fa-4755-adcd-b40d476c22f0 nodeName:}" failed. No retries permitted until 2026-11-01 03:05:56.18309241 +0000 UTC m=+2.294344714 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a972ee28-29fa-4755-adcd-b40d476c22f0-original-pull-secret") pod "global-pull-secret-syncer-4bvb6" (UID: "a972ee28-29fa-4755-adcd-b40d476c22f0") : object "kube-system"/"original-pull-secret" not registered
Nov 01 03:05:55.683649 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.683180    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.683649 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.683213    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a66c33f-29bc-4920-9d7a-0c45215514a0-metrics-certs\") pod \"network-metrics-daemon-mzkbj\" (UID: \"0a66c33f-29bc-4920-9d7a-0c45215514a0\") " pod="openshift-multus/network-metrics-daemon-mzkbj"
Nov 01 03:05:55.683649 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.683232    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q546p\" (UniqueName: \"kubernetes.io/projected/8190fea3-4e71-4250-9a0f-3a1d78dd32db-kube-api-access-q546p\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.683649 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.683245    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.683649 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.683258    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbfdt\" (UniqueName: \"kubernetes.io/projected/ca0be51d-87a4-462d-9c83-fcfb8bf224e8-kube-api-access-cbfdt\") pod \"iptables-alerter-ksj2n\" (UID: \"ca0be51d-87a4-462d-9c83-fcfb8bf224e8\") " pod="openshift-network-operator/iptables-alerter-ksj2n"
Nov 01 03:05:55.683649 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.683304    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-var-lib-openvswitch\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.683649 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.683357    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1d8dba89-5673-4cd3-a6ac-914765f228d6-cni-binary-copy\") pod \"multus-additional-cni-plugins-9nhh6\" (UID: \"1d8dba89-5673-4cd3-a6ac-914765f228d6\") " pod="openshift-multus/multus-additional-cni-plugins-9nhh6"
Nov 01 03:05:55.683649 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.683385    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-host\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.683649 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.683420    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-var-lib-openvswitch\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.683649 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.683466    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-host\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.683649 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.683468    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-systemd-units\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.683649 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.683478    2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 03:05:55.683649 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.683516    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-etc-openvswitch\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.683649 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.683521    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-systemd-units\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.683649 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.683547    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/998efdf0-7719-4768-bfd0-3da04d85d108-registration-dir\") pod \"aws-ebs-csi-driver-node-rjtwk\" (UID: \"998efdf0-7719-4768-bfd0-3da04d85d108\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk"
Nov 01 03:05:55.684466 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.683581    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-etc-openvswitch\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.684466 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.683607    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c014e8ac-58ca-4161-a778-ba58b0e84147-konnectivity-ca\") pod \"konnectivity-agent-62ccn\" (UID: \"c014e8ac-58ca-4161-a778-ba58b0e84147\") " pod="kube-system/konnectivity-agent-62ccn"
Nov 01 03:05:55.684466 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.683638    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-etc-sysctl-d\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.684466 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.683666    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-etc-systemd\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.684466 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.683640    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/998efdf0-7719-4768-bfd0-3da04d85d108-registration-dir\") pod \"aws-ebs-csi-driver-node-rjtwk\" (UID: \"998efdf0-7719-4768-bfd0-3da04d85d108\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk"
Nov 01 03:05:55.684466 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.683720    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8190fea3-4e71-4250-9a0f-3a1d78dd32db-tmp\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.684466 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.683745    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-run-systemd\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.684466 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.683769    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9e212e37-6c34-4b3c-941c-b38621f1f184-env-overrides\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.684466 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.683770    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-etc-systemd\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.684466 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:55.683783    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 03:05:55.684466 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.683796    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-host-var-lib-cni-multus\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.684466 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.683824    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnmwf\" (UniqueName: \"kubernetes.io/projected/0a66c33f-29bc-4920-9d7a-0c45215514a0-kube-api-access-xnmwf\") pod \"network-metrics-daemon-mzkbj\" (UID: \"0a66c33f-29bc-4920-9d7a-0c45215514a0\") " pod="openshift-multus/network-metrics-daemon-mzkbj"
Nov 01 03:05:55.684466 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:55.683835    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a66c33f-29bc-4920-9d7a-0c45215514a0-metrics-certs podName:0a66c33f-29bc-4920-9d7a-0c45215514a0 nodeName:}" failed. No retries permitted until 2026-11-01 03:05:56.183817941 +0000 UTC m=+2.295070250 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a66c33f-29bc-4920-9d7a-0c45215514a0-metrics-certs") pod "network-metrics-daemon-mzkbj" (UID: "0a66c33f-29bc-4920-9d7a-0c45215514a0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 03:05:55.684466 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.683830    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-run-systemd\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.684466 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.683917    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-etc-sysctl-d\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.684466 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.683963    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-host-var-lib-cni-multus\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.684466 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.684029    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lbzv\" (UniqueName: \"kubernetes.io/projected/e09a58a7-120e-4ec5-9aba-f15863267643-kube-api-access-7lbzv\") pod \"network-check-target-62mxc\" (UID: \"e09a58a7-120e-4ec5-9aba-f15863267643\") " pod="openshift-network-diagnostics/network-check-target-62mxc"
Nov 01 03:05:55.685231 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.684078    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-node-log\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.685231 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.684105    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-multus-cni-dir\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.685231 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.684136    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-node-log\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.685231 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.684136    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1d8dba89-5673-4cd3-a6ac-914765f228d6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9nhh6\" (UID: \"1d8dba89-5673-4cd3-a6ac-914765f228d6\") " pod="openshift-multus/multus-additional-cni-plugins-9nhh6"
Nov 01 03:05:55.685231 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.684191    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-host-slash\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.685231 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.684222    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-host-run-k8s-cni-cncf-io\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.685231 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.684252    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1d8dba89-5673-4cd3-a6ac-914765f228d6-cnibin\") pod \"multus-additional-cni-plugins-9nhh6\" (UID: \"1d8dba89-5673-4cd3-a6ac-914765f228d6\") " pod="openshift-multus/multus-additional-cni-plugins-9nhh6"
Nov 01 03:05:55.685231 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.684278    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/998efdf0-7719-4768-bfd0-3da04d85d108-sys-fs\") pod \"aws-ebs-csi-driver-node-rjtwk\" (UID: \"998efdf0-7719-4768-bfd0-3da04d85d108\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk"
Nov 01 03:05:55.685231 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.684305    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-host-run-netns\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.685231 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.684327    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/73622c04-1428-4f7a-8faf-8eb17604f681-host\") pod \"node-ca-stpgr\" (UID: \"73622c04-1428-4f7a-8faf-8eb17604f681\") " pod="openshift-image-registry/node-ca-stpgr"
Nov 01 03:05:55.685231 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.684350    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-run-openvswitch\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.685231 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.684372    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-etc-modprobe-d\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.685231 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.684401    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-etc-sysconfig\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.685231 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.684428    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca0be51d-87a4-462d-9c83-fcfb8bf224e8-host-slash\") pod \"iptables-alerter-ksj2n\" (UID: \"ca0be51d-87a4-462d-9c83-fcfb8bf224e8\") " pod="openshift-network-operator/iptables-alerter-ksj2n"
Nov 01 03:05:55.685231 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.684481    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-host-run-netns\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.685231 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.684482    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-multus-cni-dir\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.685231 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.684511    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9e212e37-6c34-4b3c-941c-b38621f1f184-ovnkube-config\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.686068 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.684556    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-system-cni-dir\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.686068 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.684555    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-host-run-netns\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.686068 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.684587    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-hostroot\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.686068 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.684638    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-run-openvswitch\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.686068 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.684643    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-etc-sysctl-conf\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.686068 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.684679    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-run\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.686068 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.684686    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-host-slash\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.686068 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.684694    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-etc-modprobe-d\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.686068 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.684746    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-host-run-k8s-cni-cncf-io\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.686068 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.684770    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-etc-sysconfig\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.686068 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.684788    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-lib-modules\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.686068 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.684816    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca0be51d-87a4-462d-9c83-fcfb8bf224e8-host-slash\") pod \"iptables-alerter-ksj2n\" (UID: \"ca0be51d-87a4-462d-9c83-fcfb8bf224e8\") " pod="openshift-network-operator/iptables-alerter-ksj2n"
Nov 01 03:05:55.686068 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.684833    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-run-ovn\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.686068 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.684844    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/998efdf0-7719-4768-bfd0-3da04d85d108-sys-fs\") pod \"aws-ebs-csi-driver-node-rjtwk\" (UID: \"998efdf0-7719-4768-bfd0-3da04d85d108\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk"
Nov 01 03:05:55.686068 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.684864    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwpzw\" (UniqueName: \"kubernetes.io/projected/9e212e37-6c34-4b3c-941c-b38621f1f184-kube-api-access-pwpzw\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.686068 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.684866    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-host-run-netns\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.686068 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.684889    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-host-run-multus-certs\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.686068 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.684938    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/73622c04-1428-4f7a-8faf-8eb17604f681-host\") pod \"node-ca-stpgr\" (UID: \"73622c04-1428-4f7a-8faf-8eb17604f681\") " pod="openshift-image-registry/node-ca-stpgr"
Nov 01 03:05:55.686946 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.684930    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1d8dba89-5673-4cd3-a6ac-914765f228d6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9nhh6\" (UID: \"1d8dba89-5673-4cd3-a6ac-914765f228d6\") " pod="openshift-multus/multus-additional-cni-plugins-9nhh6"
Nov 01 03:05:55.686946 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.684972    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-etc-kubernetes\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.686946 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.684980    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-etc-sysctl-conf\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.686946 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.684991    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-system-cni-dir\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.686946 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.685000    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ca0be51d-87a4-462d-9c83-fcfb8bf224e8-iptables-alerter-script\") pod \"iptables-alerter-ksj2n\" (UID: \"ca0be51d-87a4-462d-9c83-fcfb8bf224e8\") " pod="openshift-network-operator/iptables-alerter-ksj2n"
Nov 01 03:05:55.686946 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.685011    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-hostroot\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.686946 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.685026    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-host-kubelet\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.686946 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.685032    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-run-ovn\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.686946 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.685053    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-host-run-ovn-kubernetes\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.686946 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.685080    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a972ee28-29fa-4755-adcd-b40d476c22f0-dbus\") pod \"global-pull-secret-syncer-4bvb6\" (UID: \"a972ee28-29fa-4755-adcd-b40d476c22f0\") " pod="kube-system/global-pull-secret-syncer-4bvb6"
Nov 01 03:05:55.686946 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.685084    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-run\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.686946 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.685094    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-lib-modules\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.686946 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.685115    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-etc-kubernetes\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.686946 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.685107    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfxfn\" (UniqueName: \"kubernetes.io/projected/998efdf0-7719-4768-bfd0-3da04d85d108-kube-api-access-qfxfn\") pod \"aws-ebs-csi-driver-node-rjtwk\" (UID: \"998efdf0-7719-4768-bfd0-3da04d85d108\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk"
Nov 01 03:05:55.686946 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.685192    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-etc-kubernetes\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.686946 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.685225    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/73622c04-1428-4f7a-8faf-8eb17604f681-serviceca\") pod \"node-ca-stpgr\" (UID: \"73622c04-1428-4f7a-8faf-8eb17604f681\") " pod="openshift-image-registry/node-ca-stpgr"
Nov 01 03:05:55.686946 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.685259    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9e212e37-6c34-4b3c-941c-b38621f1f184-ovnkube-script-lib\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.686946 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.685277    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a972ee28-29fa-4755-adcd-b40d476c22f0-dbus\") pod \"global-pull-secret-syncer-4bvb6\" (UID: \"a972ee28-29fa-4755-adcd-b40d476c22f0\") " pod="kube-system/global-pull-secret-syncer-4bvb6"
Nov 01 03:05:55.687738 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.685290    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-multus-daemon-config\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.687738 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.685321    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d8dba89-5673-4cd3-a6ac-914765f228d6-system-cni-dir\") pod \"multus-additional-cni-plugins-9nhh6\" (UID: \"1d8dba89-5673-4cd3-a6ac-914765f228d6\") " pod="openshift-multus/multus-additional-cni-plugins-9nhh6"
Nov 01 03:05:55.687738 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.685320    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-host-kubelet\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.687738 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.685197    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-host-run-ovn-kubernetes\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.687738 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.685368    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-host-run-multus-certs\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.687738 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.685441    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x5t7\" (UniqueName: \"kubernetes.io/projected/1d8dba89-5673-4cd3-a6ac-914765f228d6-kube-api-access-5x5t7\") pod \"multus-additional-cni-plugins-9nhh6\" (UID: \"1d8dba89-5673-4cd3-a6ac-914765f228d6\") " pod="openshift-multus/multus-additional-cni-plugins-9nhh6"
Nov 01 03:05:55.687738 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.685523    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-host-cni-netd\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.687738 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.685566    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-etc-kubernetes\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.687738 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.685569    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-cnibin\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.687738 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.685599    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-host-cni-netd\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.687738 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.685619    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkm29\" (UniqueName: \"kubernetes.io/projected/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-kube-api-access-vkm29\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.687738 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.685630    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-cnibin\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.687738 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.685737    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-sys\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.687738 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.685760    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8190fea3-4e71-4250-9a0f-3a1d78dd32db-etc-tuned\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.687738 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.685777    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-log-socket\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.687738 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.685803    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/998efdf0-7719-4768-bfd0-3da04d85d108-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rjtwk\" (UID: \"998efdf0-7719-4768-bfd0-3da04d85d108\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk"
Nov 01 03:05:55.687738 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.685860    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-log-socket\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.687738 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.685875    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-sys\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.688292 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.685872    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/998efdf0-7719-4768-bfd0-3da04d85d108-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rjtwk\" (UID: \"998efdf0-7719-4768-bfd0-3da04d85d108\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk"
Nov 01 03:05:55.688292 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.685894    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-multus-conf-dir\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.688292 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.685946    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a972ee28-29fa-4755-adcd-b40d476c22f0-kubelet-config\") pod \"global-pull-secret-syncer-4bvb6\" (UID: \"a972ee28-29fa-4755-adcd-b40d476c22f0\") " pod="kube-system/global-pull-secret-syncer-4bvb6"
Nov 01 03:05:55.688292 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.685963    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/998efdf0-7719-4768-bfd0-3da04d85d108-socket-dir\") pod \"aws-ebs-csi-driver-node-rjtwk\" (UID: \"998efdf0-7719-4768-bfd0-3da04d85d108\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk"
Nov 01 03:05:55.688292 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.686015    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-multus-conf-dir\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.688292 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.686023    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/998efdf0-7719-4768-bfd0-3da04d85d108-device-dir\") pod \"aws-ebs-csi-driver-node-rjtwk\" (UID: \"998efdf0-7719-4768-bfd0-3da04d85d108\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk"
Nov 01 03:05:55.688292 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.686039    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a972ee28-29fa-4755-adcd-b40d476c22f0-kubelet-config\") pod \"global-pull-secret-syncer-4bvb6\" (UID: \"a972ee28-29fa-4755-adcd-b40d476c22f0\") " pod="kube-system/global-pull-secret-syncer-4bvb6"
Nov 01 03:05:55.688292 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.686079    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/998efdf0-7719-4768-bfd0-3da04d85d108-etc-selinux\") pod \"aws-ebs-csi-driver-node-rjtwk\" (UID: \"998efdf0-7719-4768-bfd0-3da04d85d108\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk"
Nov 01 03:05:55.688292 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.686115    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/482aebaa-cb7f-4be2-991a-7fb4cd801c7f-hosts-file\") pod \"node-resolver-77nf7\" (UID: \"482aebaa-cb7f-4be2-991a-7fb4cd801c7f\") " pod="openshift-dns/node-resolver-77nf7"
Nov 01 03:05:55.688292 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.686137    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/998efdf0-7719-4768-bfd0-3da04d85d108-socket-dir\") pod \"aws-ebs-csi-driver-node-rjtwk\" (UID: \"998efdf0-7719-4768-bfd0-3da04d85d108\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk"
Nov 01 03:05:55.688292 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.686141    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/482aebaa-cb7f-4be2-991a-7fb4cd801c7f-tmp-dir\") pod \"node-resolver-77nf7\" (UID: \"482aebaa-cb7f-4be2-991a-7fb4cd801c7f\") " pod="openshift-dns/node-resolver-77nf7"
Nov 01 03:05:55.688292 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.686170    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/998efdf0-7719-4768-bfd0-3da04d85d108-device-dir\") pod \"aws-ebs-csi-driver-node-rjtwk\" (UID: \"998efdf0-7719-4768-bfd0-3da04d85d108\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk"
Nov 01 03:05:55.688292 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.686175    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm7m6\" (UniqueName: \"kubernetes.io/projected/482aebaa-cb7f-4be2-991a-7fb4cd801c7f-kube-api-access-mm7m6\") pod \"node-resolver-77nf7\" (UID: \"482aebaa-cb7f-4be2-991a-7fb4cd801c7f\") " pod="openshift-dns/node-resolver-77nf7"
Nov 01 03:05:55.688292 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.686235    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/998efdf0-7719-4768-bfd0-3da04d85d108-etc-selinux\") pod \"aws-ebs-csi-driver-node-rjtwk\" (UID: \"998efdf0-7719-4768-bfd0-3da04d85d108\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk"
Nov 01 03:05:55.688292 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.686238    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/482aebaa-cb7f-4be2-991a-7fb4cd801c7f-hosts-file\") pod \"node-resolver-77nf7\" (UID: \"482aebaa-cb7f-4be2-991a-7fb4cd801c7f\") " pod="openshift-dns/node-resolver-77nf7"
Nov 01 03:05:55.688292 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.686273    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c014e8ac-58ca-4161-a778-ba58b0e84147-agent-certs\") pod \"konnectivity-agent-62ccn\" (UID: \"c014e8ac-58ca-4161-a778-ba58b0e84147\") " pod="kube-system/konnectivity-agent-62ccn"
Nov 01 03:05:55.688292 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.686398    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/482aebaa-cb7f-4be2-991a-7fb4cd801c7f-tmp-dir\") pod \"node-resolver-77nf7\" (UID: \"482aebaa-cb7f-4be2-991a-7fb4cd801c7f\") " pod="openshift-dns/node-resolver-77nf7"
Nov 01 03:05:55.688798 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.687864    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8190fea3-4e71-4250-9a0f-3a1d78dd32db-tmp\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.688798 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.687919    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9e212e37-6c34-4b3c-941c-b38621f1f184-ovn-node-metrics-cert\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.688798 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.688127    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8190fea3-4e71-4250-9a0f-3a1d78dd32db-etc-tuned\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.721482 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.721456    2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 03:05:55.721592 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.721516    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tzw7z"
Nov 01 03:05:55.722398 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.722380    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 03:05:55.722969 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.722951    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 03:05:55.723593 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.723576    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 03:05:55.723688 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.723596    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mt5gq\""
Nov 01 03:05:55.723748 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.723724    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 03:05:55.723748 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.723734    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 03:05:55.723827 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.723745    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 03:05:55.723879 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.723848    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-62njk\""
Nov 01 03:05:55.723947 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.723929    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 03:05:55.723996 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.723986    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ksxvj\""
Nov 01 03:05:55.724046 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.723996    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 03:05:55.724046 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.724025    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 03:05:55.724144 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.724087    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4hc7l\""
Nov 01 03:05:55.724144 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.724137    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-vzcml\""
Nov 01 03:05:55.724265 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.724242    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6592m\""
Nov 01 03:05:55.724304 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.724253    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 03:05:55.724417 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.724403    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 03:05:55.724472 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.724454    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9e212e37-6c34-4b3c-941c-b38621f1f184-env-overrides\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.725380 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.725363    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9e212e37-6c34-4b3c-941c-b38621f1f184-ovnkube-config\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.726507 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.726491    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/73622c04-1428-4f7a-8faf-8eb17604f681-serviceca\") pod \"node-ca-stpgr\" (UID: \"73622c04-1428-4f7a-8faf-8eb17604f681\") " pod="openshift-image-registry/node-ca-stpgr"
Nov 01 03:05:55.776330 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.776275    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 03:05:55.776330 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.776279    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 03:05:55.776480 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.776377    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 03:05:55.776480 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.776386    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 03:05:55.776480 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.776426    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 03:05:55.776607 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.776554    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 03:05:55.776670 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.776613    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 03:05:55.776670 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.776623    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 03:05:55.776772 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.776685    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 03:05:55.777014 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.776993    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 03:05:55.777082 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.777029    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hrmfk\""
Nov 01 03:05:55.777082 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.777055    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 03:05:55.777082 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.777066    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-kbjpv\""
Nov 01 03:05:55.777082 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.777078    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 03:05:55.777572 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.777558    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 03:05:55.783191 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.783173    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfxfn\" (UniqueName: \"kubernetes.io/projected/998efdf0-7719-4768-bfd0-3da04d85d108-kube-api-access-qfxfn\") pod \"aws-ebs-csi-driver-node-rjtwk\" (UID: \"998efdf0-7719-4768-bfd0-3da04d85d108\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk"
Nov 01 03:05:55.783603 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.783577    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-cni-binary-copy\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.786985 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.786967    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ca0be51d-87a4-462d-9c83-fcfb8bf224e8-iptables-alerter-script\") pod \"iptables-alerter-ksj2n\" (UID: \"ca0be51d-87a4-462d-9c83-fcfb8bf224e8\") " pod="openshift-network-operator/iptables-alerter-ksj2n"
Nov 01 03:05:55.787062 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.787019    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-multus-daemon-config\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.787062 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.787021    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9e212e37-6c34-4b3c-941c-b38621f1f184-ovnkube-script-lib\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.787182 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.787168    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1d8dba89-5673-4cd3-a6ac-914765f228d6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9nhh6\" (UID: \"1d8dba89-5673-4cd3-a6ac-914765f228d6\") " pod="openshift-multus/multus-additional-cni-plugins-9nhh6"
Nov 01 03:05:55.787223 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.787195    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d8dba89-5673-4cd3-a6ac-914765f228d6-system-cni-dir\") pod \"multus-additional-cni-plugins-9nhh6\" (UID: \"1d8dba89-5673-4cd3-a6ac-914765f228d6\") " pod="openshift-multus/multus-additional-cni-plugins-9nhh6"
Nov 01 03:05:55.787223 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.787214    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x5t7\" (UniqueName: \"kubernetes.io/projected/1d8dba89-5673-4cd3-a6ac-914765f228d6-kube-api-access-5x5t7\") pod \"multus-additional-cni-plugins-9nhh6\" (UID: \"1d8dba89-5673-4cd3-a6ac-914765f228d6\") " pod="openshift-multus/multus-additional-cni-plugins-9nhh6"
Nov 01 03:05:55.787306 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.787248    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c014e8ac-58ca-4161-a778-ba58b0e84147-agent-certs\") pod \"konnectivity-agent-62ccn\" (UID: \"c014e8ac-58ca-4161-a778-ba58b0e84147\") " pod="kube-system/konnectivity-agent-62ccn"
Nov 01 03:05:55.787306 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.787274    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d8dba89-5673-4cd3-a6ac-914765f228d6-system-cni-dir\") pod \"multus-additional-cni-plugins-9nhh6\" (UID: \"1d8dba89-5673-4cd3-a6ac-914765f228d6\") " pod="openshift-multus/multus-additional-cni-plugins-9nhh6"
Nov 01 03:05:55.787401 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.787363    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1d8dba89-5673-4cd3-a6ac-914765f228d6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9nhh6\" (UID: \"1d8dba89-5673-4cd3-a6ac-914765f228d6\") " pod="openshift-multus/multus-additional-cni-plugins-9nhh6"
Nov 01 03:05:55.787456 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.787401    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1d8dba89-5673-4cd3-a6ac-914765f228d6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9nhh6\" (UID: \"1d8dba89-5673-4cd3-a6ac-914765f228d6\") " pod="openshift-multus/multus-additional-cni-plugins-9nhh6"
Nov 01 03:05:55.787509 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.787469    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1d8dba89-5673-4cd3-a6ac-914765f228d6-cni-binary-copy\") pod \"multus-additional-cni-plugins-9nhh6\" (UID: \"1d8dba89-5673-4cd3-a6ac-914765f228d6\") " pod="openshift-multus/multus-additional-cni-plugins-9nhh6"
Nov 01 03:05:55.787571 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.787510    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c014e8ac-58ca-4161-a778-ba58b0e84147-konnectivity-ca\") pod \"konnectivity-agent-62ccn\" (UID: \"c014e8ac-58ca-4161-a778-ba58b0e84147\") " pod="kube-system/konnectivity-agent-62ccn"
Nov 01 03:05:55.787571 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.787538    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1d8dba89-5673-4cd3-a6ac-914765f228d6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9nhh6\" (UID: \"1d8dba89-5673-4cd3-a6ac-914765f228d6\") " pod="openshift-multus/multus-additional-cni-plugins-9nhh6"
Nov 01 03:05:55.787571 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.787566    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1d8dba89-5673-4cd3-a6ac-914765f228d6-cnibin\") pod \"multus-additional-cni-plugins-9nhh6\" (UID: \"1d8dba89-5673-4cd3-a6ac-914765f228d6\") " pod="openshift-multus/multus-additional-cni-plugins-9nhh6"
Nov 01 03:05:55.787743 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.787667    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1d8dba89-5673-4cd3-a6ac-914765f228d6-cnibin\") pod \"multus-additional-cni-plugins-9nhh6\" (UID: \"1d8dba89-5673-4cd3-a6ac-914765f228d6\") " pod="openshift-multus/multus-additional-cni-plugins-9nhh6"
Nov 01 03:05:55.788497 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.788477    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c014e8ac-58ca-4161-a778-ba58b0e84147-konnectivity-ca\") pod \"konnectivity-agent-62ccn\" (UID: \"c014e8ac-58ca-4161-a778-ba58b0e84147\") " pod="kube-system/konnectivity-agent-62ccn"
Nov 01 03:05:55.788591 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.788478    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1d8dba89-5673-4cd3-a6ac-914765f228d6-cni-binary-copy\") pod \"multus-additional-cni-plugins-9nhh6\" (UID: \"1d8dba89-5673-4cd3-a6ac-914765f228d6\") " pod="openshift-multus/multus-additional-cni-plugins-9nhh6"
Nov 01 03:05:55.788591 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.788509    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1d8dba89-5673-4cd3-a6ac-914765f228d6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9nhh6\" (UID: \"1d8dba89-5673-4cd3-a6ac-914765f228d6\") " pod="openshift-multus/multus-additional-cni-plugins-9nhh6"
Nov 01 03:05:55.788677 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.788587    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1d8dba89-5673-4cd3-a6ac-914765f228d6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9nhh6\" (UID: \"1d8dba89-5673-4cd3-a6ac-914765f228d6\") " pod="openshift-multus/multus-additional-cni-plugins-9nhh6"
Nov 01 03:05:55.790009 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.789989    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c014e8ac-58ca-4161-a778-ba58b0e84147-agent-certs\") pod \"konnectivity-agent-62ccn\" (UID: \"c014e8ac-58ca-4161-a778-ba58b0e84147\") " pod="kube-system/konnectivity-agent-62ccn"
Nov 01 03:05:55.791595 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:55.791545    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 03:05:55.791878 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:55.791860    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 03:05:55.791961 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:55.791886    2578 projected.go:196] Error preparing data for projected volume kube-api-access-7lbzv for pod openshift-network-diagnostics/network-check-target-62mxc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 03:05:55.792029 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:55.791987    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e09a58a7-120e-4ec5-9aba-f15863267643-kube-api-access-7lbzv podName:e09a58a7-120e-4ec5-9aba-f15863267643 nodeName:}" failed. No retries permitted until 2026-11-01 03:05:56.291956631 +0000 UTC m=+2.403208937 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7lbzv" (UniqueName: "kubernetes.io/projected/e09a58a7-120e-4ec5-9aba-f15863267643-kube-api-access-7lbzv") pod "network-check-target-62mxc" (UID: "e09a58a7-120e-4ec5-9aba-f15863267643") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 03:05:55.794028 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.794005    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkm29\" (UniqueName: \"kubernetes.io/projected/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-kube-api-access-vkm29\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.794779 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.794755    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwpzw\" (UniqueName: \"kubernetes.io/projected/9e212e37-6c34-4b3c-941c-b38621f1f184-kube-api-access-pwpzw\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.795019 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.794992    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbfdt\" (UniqueName: \"kubernetes.io/projected/ca0be51d-87a4-462d-9c83-fcfb8bf224e8-kube-api-access-cbfdt\") pod \"iptables-alerter-ksj2n\" (UID: \"ca0be51d-87a4-462d-9c83-fcfb8bf224e8\") " pod="openshift-network-operator/iptables-alerter-ksj2n"
Nov 01 03:05:55.795186 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.795163    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm7m6\" (UniqueName: \"kubernetes.io/projected/482aebaa-cb7f-4be2-991a-7fb4cd801c7f-kube-api-access-mm7m6\") pod \"node-resolver-77nf7\" (UID: \"482aebaa-cb7f-4be2-991a-7fb4cd801c7f\") " pod="openshift-dns/node-resolver-77nf7"
Nov 01 03:05:55.796566 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.796539    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnmwf\" (UniqueName: \"kubernetes.io/projected/0a66c33f-29bc-4920-9d7a-0c45215514a0-kube-api-access-xnmwf\") pod \"network-metrics-daemon-mzkbj\" (UID: \"0a66c33f-29bc-4920-9d7a-0c45215514a0\") " pod="openshift-multus/network-metrics-daemon-mzkbj"
Nov 01 03:05:55.796805 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.796764    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjt9z\" (UniqueName: \"kubernetes.io/projected/73622c04-1428-4f7a-8faf-8eb17604f681-kube-api-access-hjt9z\") pod \"node-ca-stpgr\" (UID: \"73622c04-1428-4f7a-8faf-8eb17604f681\") " pod="openshift-image-registry/node-ca-stpgr"
Nov 01 03:05:55.797381 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.797363    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q546p\" (UniqueName: \"kubernetes.io/projected/8190fea3-4e71-4250-9a0f-3a1d78dd32db-kube-api-access-q546p\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.871416 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.871371    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x5t7\" (UniqueName: \"kubernetes.io/projected/1d8dba89-5673-4cd3-a6ac-914765f228d6-kube-api-access-5x5t7\") pod \"multus-additional-cni-plugins-9nhh6\" (UID: \"1d8dba89-5673-4cd3-a6ac-914765f228d6\") " pod="openshift-multus/multus-additional-cni-plugins-9nhh6"
Nov 01 03:05:55.887442 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.887413    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-77nf7"
Nov 01 03:05:55.926581 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.926551    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:05:55.932726 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.932702    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk"
Nov 01 03:05:55.934888 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.934860    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-stpgr"
Nov 01 03:05:55.947074 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.947053    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p5ght"
Nov 01 03:05:55.958371 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:55.958336    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbcea9cdf04d326735591dc246925152.slice/crio-b2223a9b09494c4f811b61916b2a8f35e16d4f4c5cde2281b3aa270664b436fa WatchSource:0}: Error finding container b2223a9b09494c4f811b61916b2a8f35e16d4f4c5cde2281b3aa270664b436fa: Status 404 returned error can't find the container with id b2223a9b09494c4f811b61916b2a8f35e16d4f4c5cde2281b3aa270664b436fa
Nov 01 03:05:55.959002 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:55.958979    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod998efdf0_7719_4768_bfd0_3da04d85d108.slice/crio-fde6d237ec81277da04848aed98d89c282c896aae61e77ba5ff427fe0e4e99c7 WatchSource:0}: Error finding container fde6d237ec81277da04848aed98d89c282c896aae61e77ba5ff427fe0e4e99c7: Status 404 returned error can't find the container with id fde6d237ec81277da04848aed98d89c282c896aae61e77ba5ff427fe0e4e99c7
Nov 01 03:05:55.959654 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:55.959615    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73622c04_1428_4f7a_8faf_8eb17604f681.slice/crio-5ba36b4263d1487516ad584e54be8f8ad2bc6b75bb923831ab769b92c1116d47 WatchSource:0}: Error finding container 5ba36b4263d1487516ad584e54be8f8ad2bc6b75bb923831ab769b92c1116d47: Status 404 returned error can't find the container with id 5ba36b4263d1487516ad584e54be8f8ad2bc6b75bb923831ab769b92c1116d47
Nov 01 03:05:55.960356 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:55.960332    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e212e37_6c34_4b3c_941c_b38621f1f184.slice/crio-c82f8c77dbea3089c4c6a24196b9841b6d15dcd34e64e9f297e5a0ba52f242f7 WatchSource:0}: Error finding container c82f8c77dbea3089c4c6a24196b9841b6d15dcd34e64e9f297e5a0ba52f242f7: Status 404 returned error can't find the container with id c82f8c77dbea3089c4c6a24196b9841b6d15dcd34e64e9f297e5a0ba52f242f7
Nov 01 03:05:55.964270 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.964253    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:05:55.966891 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.966868    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 03:05:55.975325 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:55.975305    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf308b2b4_03e5_4d52_97ff_f1eb3ff951b1.slice/crio-49fa91383c974839e7c75e8668112795f422c06c53aafa284bd3c532c9c7be31 WatchSource:0}: Error finding container 49fa91383c974839e7c75e8668112795f422c06c53aafa284bd3c532c9c7be31: Status 404 returned error can't find the container with id 49fa91383c974839e7c75e8668112795f422c06c53aafa284bd3c532c9c7be31
Nov 01 03:05:55.975922 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:55.975902    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5391522643ff9514bed3da110ddd8b71.slice/crio-a6ab6d91e787fd5d75e21c3a846dd81f99aeeae8c8cdd9c980045c29db5e19d1 WatchSource:0}: Error finding container a6ab6d91e787fd5d75e21c3a846dd81f99aeeae8c8cdd9c980045c29db5e19d1: Status 404 returned error can't find the container with id a6ab6d91e787fd5d75e21c3a846dd81f99aeeae8c8cdd9c980045c29db5e19d1
Nov 01 03:05:55.976885 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:55.976863    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8190fea3_4e71_4250_9a0f_3a1d78dd32db.slice/crio-695f59359b7b058daa772f98d3e3ce7b61104158a9dd00466a8d5863ecb8b20c WatchSource:0}: Error finding container 695f59359b7b058daa772f98d3e3ce7b61104158a9dd00466a8d5863ecb8b20c: Status 404 returned error can't find the container with id 695f59359b7b058daa772f98d3e3ce7b61104158a9dd00466a8d5863ecb8b20c
Nov 01 03:05:55.983474 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:55.983456    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ksj2n"
Nov 01 03:05:55.992455 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:55.992438    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca0be51d_87a4_462d_9c83_fcfb8bf224e8.slice/crio-a8e3518e155c194ecccfadb3976945cfcf143446b46944d5ec821842d71d5342 WatchSource:0}: Error finding container a8e3518e155c194ecccfadb3976945cfcf143446b46944d5ec821842d71d5342: Status 404 returned error can't find the container with id a8e3518e155c194ecccfadb3976945cfcf143446b46944d5ec821842d71d5342
Nov 01 03:05:56.013248 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:56.013225    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-62ccn"
Nov 01 03:05:56.018150 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:56.018136    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9nhh6"
Nov 01 03:05:56.022868 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:56.022848    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc014e8ac_58ca_4161_a778_ba58b0e84147.slice/crio-ad50c82576c60f43adea717755051e72585691739af4421c5b2c785460d1309e WatchSource:0}: Error finding container ad50c82576c60f43adea717755051e72585691739af4421c5b2c785460d1309e: Status 404 returned error can't find the container with id ad50c82576c60f43adea717755051e72585691739af4421c5b2c785460d1309e
Nov 01 03:05:56.026465 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:56.026418    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d8dba89_5673_4cd3_a6ac_914765f228d6.slice/crio-88ff02f45ff7b504af9f584fe1c578af34b1f29ec41a475a5e75cfe4eb18ed50 WatchSource:0}: Error finding container 88ff02f45ff7b504af9f584fe1c578af34b1f29ec41a475a5e75cfe4eb18ed50: Status 404 returned error can't find the container with id 88ff02f45ff7b504af9f584fe1c578af34b1f29ec41a475a5e75cfe4eb18ed50
Nov 01 03:05:56.191343 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:56.191311    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a972ee28-29fa-4755-adcd-b40d476c22f0-original-pull-secret\") pod \"global-pull-secret-syncer-4bvb6\" (UID: \"a972ee28-29fa-4755-adcd-b40d476c22f0\") " pod="kube-system/global-pull-secret-syncer-4bvb6"
Nov 01 03:05:56.191343 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:56.191348    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a66c33f-29bc-4920-9d7a-0c45215514a0-metrics-certs\") pod \"network-metrics-daemon-mzkbj\" (UID: \"0a66c33f-29bc-4920-9d7a-0c45215514a0\") " pod="openshift-multus/network-metrics-daemon-mzkbj"
Nov 01 03:05:56.191543 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:56.191465    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 03:05:56.191543 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:56.191466    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 03:05:56.191543 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:56.191518    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a66c33f-29bc-4920-9d7a-0c45215514a0-metrics-certs podName:0a66c33f-29bc-4920-9d7a-0c45215514a0 nodeName:}" failed. No retries permitted until 2026-11-01 03:05:57.1915052 +0000 UTC m=+3.302757486 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a66c33f-29bc-4920-9d7a-0c45215514a0-metrics-certs") pod "network-metrics-daemon-mzkbj" (UID: "0a66c33f-29bc-4920-9d7a-0c45215514a0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 03:05:56.191543 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:56.191533    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a972ee28-29fa-4755-adcd-b40d476c22f0-original-pull-secret podName:a972ee28-29fa-4755-adcd-b40d476c22f0 nodeName:}" failed. No retries permitted until 2026-11-01 03:05:57.191526187 +0000 UTC m=+3.302778473 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a972ee28-29fa-4755-adcd-b40d476c22f0-original-pull-secret") pod "global-pull-secret-syncer-4bvb6" (UID: "a972ee28-29fa-4755-adcd-b40d476c22f0") : object "kube-system"/"original-pull-secret" not registered
Nov 01 03:05:56.219418 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:56.219396    2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 03:05:56.292945 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:56.292868    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lbzv\" (UniqueName: \"kubernetes.io/projected/e09a58a7-120e-4ec5-9aba-f15863267643-kube-api-access-7lbzv\") pod \"network-check-target-62mxc\" (UID: \"e09a58a7-120e-4ec5-9aba-f15863267643\") " pod="openshift-network-diagnostics/network-check-target-62mxc"
Nov 01 03:05:56.293083 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:56.292999    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 03:05:56.293083 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:56.293021    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 03:05:56.293083 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:56.293034    2578 projected.go:196] Error preparing data for projected volume kube-api-access-7lbzv for pod openshift-network-diagnostics/network-check-target-62mxc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 03:05:56.293246 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:56.293093    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e09a58a7-120e-4ec5-9aba-f15863267643-kube-api-access-7lbzv podName:e09a58a7-120e-4ec5-9aba-f15863267643 nodeName:}" failed. No retries permitted until 2026-11-01 03:05:57.293073683 +0000 UTC m=+3.404325972 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7lbzv" (UniqueName: "kubernetes.io/projected/e09a58a7-120e-4ec5-9aba-f15863267643-kube-api-access-7lbzv") pod "network-check-target-62mxc" (UID: "e09a58a7-120e-4ec5-9aba-f15863267643") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 03:05:56.422308 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:56.422256    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-62ccn" event={"ID":"c014e8ac-58ca-4161-a778-ba58b0e84147","Type":"ContainerStarted","Data":"ad50c82576c60f43adea717755051e72585691739af4421c5b2c785460d1309e"}
Nov 01 03:05:56.423335 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:56.423310    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ksj2n" event={"ID":"ca0be51d-87a4-462d-9c83-fcfb8bf224e8","Type":"ContainerStarted","Data":"a8e3518e155c194ecccfadb3976945cfcf143446b46944d5ec821842d71d5342"}
Nov 01 03:05:56.424354 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:56.424333    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-l892q" event={"ID":"8190fea3-4e71-4250-9a0f-3a1d78dd32db","Type":"ContainerStarted","Data":"695f59359b7b058daa772f98d3e3ce7b61104158a9dd00466a8d5863ecb8b20c"}
Nov 01 03:05:56.425359 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:56.425337    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-122.us-west-1.compute.internal" event={"ID":"5391522643ff9514bed3da110ddd8b71","Type":"ContainerStarted","Data":"a6ab6d91e787fd5d75e21c3a846dd81f99aeeae8c8cdd9c980045c29db5e19d1"}
Nov 01 03:05:56.426418 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:56.426394    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk" event={"ID":"998efdf0-7719-4768-bfd0-3da04d85d108","Type":"ContainerStarted","Data":"fde6d237ec81277da04848aed98d89c282c896aae61e77ba5ff427fe0e4e99c7"}
Nov 01 03:05:56.427441 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:56.427424    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-stpgr" event={"ID":"73622c04-1428-4f7a-8faf-8eb17604f681","Type":"ContainerStarted","Data":"5ba36b4263d1487516ad584e54be8f8ad2bc6b75bb923831ab769b92c1116d47"}
Nov 01 03:05:56.428451 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:56.428429    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal" event={"ID":"bbcea9cdf04d326735591dc246925152","Type":"ContainerStarted","Data":"b2223a9b09494c4f811b61916b2a8f35e16d4f4c5cde2281b3aa270664b436fa"}
Nov 01 03:05:56.429572 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:56.429542    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq" event={"ID":"9e212e37-6c34-4b3c-941c-b38621f1f184","Type":"ContainerStarted","Data":"c82f8c77dbea3089c4c6a24196b9841b6d15dcd34e64e9f297e5a0ba52f242f7"}
Nov 01 03:05:56.430617 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:56.430594    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9nhh6" event={"ID":"1d8dba89-5673-4cd3-a6ac-914765f228d6","Type":"ContainerStarted","Data":"88ff02f45ff7b504af9f584fe1c578af34b1f29ec41a475a5e75cfe4eb18ed50"}
Nov 01 03:05:56.431605 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:56.431568    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p5ght" event={"ID":"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1","Type":"ContainerStarted","Data":"49fa91383c974839e7c75e8668112795f422c06c53aafa284bd3c532c9c7be31"}
Nov 01 03:05:56.605224 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:56.605145    2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 03:05:56.724899 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:56.724854    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 03:00:55 +0000 UTC" deadline="2028-07-06 03:00:54.49996899 +0000 UTC"
Nov 01 03:05:56.724899 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:56.724898    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14711h54m57.775073083s"
Nov 01 03:05:56.727067 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:56.727051    2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 03:05:56.756841 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:05:56.756813    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod482aebaa_cb7f_4be2_991a_7fb4cd801c7f.slice/crio-54484f3a947b1f4d94c67d7fa4f7025617444bc1dec9c1eca62fe9bf48d2808b WatchSource:0}: Error finding container 54484f3a947b1f4d94c67d7fa4f7025617444bc1dec9c1eca62fe9bf48d2808b: Status 404 returned error can't find the container with id 54484f3a947b1f4d94c67d7fa4f7025617444bc1dec9c1eca62fe9bf48d2808b
Nov 01 03:05:57.200939 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:57.200902    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a972ee28-29fa-4755-adcd-b40d476c22f0-original-pull-secret\") pod \"global-pull-secret-syncer-4bvb6\" (UID: \"a972ee28-29fa-4755-adcd-b40d476c22f0\") " pod="kube-system/global-pull-secret-syncer-4bvb6"
Nov 01 03:05:57.201158 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:57.200957    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a66c33f-29bc-4920-9d7a-0c45215514a0-metrics-certs\") pod \"network-metrics-daemon-mzkbj\" (UID: \"0a66c33f-29bc-4920-9d7a-0c45215514a0\") " pod="openshift-multus/network-metrics-daemon-mzkbj"
Nov 01 03:05:57.201158 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:57.201100    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 03:05:57.201158 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:57.201160    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a66c33f-29bc-4920-9d7a-0c45215514a0-metrics-certs podName:0a66c33f-29bc-4920-9d7a-0c45215514a0 nodeName:}" failed. No retries permitted until 2026-11-01 03:05:59.201140577 +0000 UTC m=+5.312392878 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a66c33f-29bc-4920-9d7a-0c45215514a0-metrics-certs") pod "network-metrics-daemon-mzkbj" (UID: "0a66c33f-29bc-4920-9d7a-0c45215514a0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 03:05:57.201350 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:57.201229    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 03:05:57.201350 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:57.201265    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a972ee28-29fa-4755-adcd-b40d476c22f0-original-pull-secret podName:a972ee28-29fa-4755-adcd-b40d476c22f0 nodeName:}" failed. No retries permitted until 2026-11-01 03:05:59.201252549 +0000 UTC m=+5.312504854 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a972ee28-29fa-4755-adcd-b40d476c22f0-original-pull-secret") pod "global-pull-secret-syncer-4bvb6" (UID: "a972ee28-29fa-4755-adcd-b40d476c22f0") : object "kube-system"/"original-pull-secret" not registered
Nov 01 03:05:57.301330 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:57.301292    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lbzv\" (UniqueName: \"kubernetes.io/projected/e09a58a7-120e-4ec5-9aba-f15863267643-kube-api-access-7lbzv\") pod \"network-check-target-62mxc\" (UID: \"e09a58a7-120e-4ec5-9aba-f15863267643\") " pod="openshift-network-diagnostics/network-check-target-62mxc"
Nov 01 03:05:57.301871 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:57.301476    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 03:05:57.301871 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:57.301499    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 03:05:57.301871 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:57.301514    2578 projected.go:196] Error preparing data for projected volume kube-api-access-7lbzv for pod openshift-network-diagnostics/network-check-target-62mxc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 03:05:57.301871 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:57.301575    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e09a58a7-120e-4ec5-9aba-f15863267643-kube-api-access-7lbzv podName:e09a58a7-120e-4ec5-9aba-f15863267643 nodeName:}" failed. No retries permitted until 2026-11-01 03:05:59.301555084 +0000 UTC m=+5.412807391 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7lbzv" (UniqueName: "kubernetes.io/projected/e09a58a7-120e-4ec5-9aba-f15863267643-kube-api-access-7lbzv") pod "network-check-target-62mxc" (UID: "e09a58a7-120e-4ec5-9aba-f15863267643") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 03:05:57.418652 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:57.418462    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62mxc"
Nov 01 03:05:57.418652 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:57.418572    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62mxc" podUID="e09a58a7-120e-4ec5-9aba-f15863267643"
Nov 01 03:05:57.418899 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:57.418716    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzkbj"
Nov 01 03:05:57.418899 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:57.418744    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bvb6"
Nov 01 03:05:57.419000 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:57.418892    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzkbj" podUID="0a66c33f-29bc-4920-9d7a-0c45215514a0"
Nov 01 03:05:57.419052 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:57.419013    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bvb6" podUID="a972ee28-29fa-4755-adcd-b40d476c22f0"
Nov 01 03:05:57.451519 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:57.451433    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-77nf7" event={"ID":"482aebaa-cb7f-4be2-991a-7fb4cd801c7f","Type":"ContainerStarted","Data":"54484f3a947b1f4d94c67d7fa4f7025617444bc1dec9c1eca62fe9bf48d2808b"}
Nov 01 03:05:57.725466 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:57.725353    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 03:00:55 +0000 UTC" deadline="2028-04-26 21:09:12.053158733 +0000 UTC"
Nov 01 03:05:57.725466 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:57.725418    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13026h3m14.327745745s"
Nov 01 03:05:59.216560 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:59.216518    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a972ee28-29fa-4755-adcd-b40d476c22f0-original-pull-secret\") pod \"global-pull-secret-syncer-4bvb6\" (UID: \"a972ee28-29fa-4755-adcd-b40d476c22f0\") " pod="kube-system/global-pull-secret-syncer-4bvb6"
Nov 01 03:05:59.217169 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:59.216572    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a66c33f-29bc-4920-9d7a-0c45215514a0-metrics-certs\") pod \"network-metrics-daemon-mzkbj\" (UID: \"0a66c33f-29bc-4920-9d7a-0c45215514a0\") " pod="openshift-multus/network-metrics-daemon-mzkbj"
Nov 01 03:05:59.217169 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:59.216727    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 03:05:59.217169 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:59.216795    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a66c33f-29bc-4920-9d7a-0c45215514a0-metrics-certs podName:0a66c33f-29bc-4920-9d7a-0c45215514a0 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:03.216774873 +0000 UTC m=+9.328027180 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a66c33f-29bc-4920-9d7a-0c45215514a0-metrics-certs") pod "network-metrics-daemon-mzkbj" (UID: "0a66c33f-29bc-4920-9d7a-0c45215514a0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 03:05:59.217169 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:59.216870    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 03:05:59.217169 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:59.216908    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a972ee28-29fa-4755-adcd-b40d476c22f0-original-pull-secret podName:a972ee28-29fa-4755-adcd-b40d476c22f0 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:03.216895292 +0000 UTC m=+9.328147583 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a972ee28-29fa-4755-adcd-b40d476c22f0-original-pull-secret") pod "global-pull-secret-syncer-4bvb6" (UID: "a972ee28-29fa-4755-adcd-b40d476c22f0") : object "kube-system"/"original-pull-secret" not registered
Nov 01 03:05:59.316954 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:59.316919    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lbzv\" (UniqueName: \"kubernetes.io/projected/e09a58a7-120e-4ec5-9aba-f15863267643-kube-api-access-7lbzv\") pod \"network-check-target-62mxc\" (UID: \"e09a58a7-120e-4ec5-9aba-f15863267643\") " pod="openshift-network-diagnostics/network-check-target-62mxc"
Nov 01 03:05:59.317118 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:59.317101    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 03:05:59.317180 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:59.317126    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 03:05:59.317180 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:59.317140    2578 projected.go:196] Error preparing data for projected volume kube-api-access-7lbzv for pod openshift-network-diagnostics/network-check-target-62mxc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 03:05:59.317304 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:59.317204    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e09a58a7-120e-4ec5-9aba-f15863267643-kube-api-access-7lbzv podName:e09a58a7-120e-4ec5-9aba-f15863267643 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:03.317185147 +0000 UTC m=+9.428437439 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7lbzv" (UniqueName: "kubernetes.io/projected/e09a58a7-120e-4ec5-9aba-f15863267643-kube-api-access-7lbzv") pod "network-check-target-62mxc" (UID: "e09a58a7-120e-4ec5-9aba-f15863267643") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 03:05:59.418556 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:59.418462    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzkbj"
Nov 01 03:05:59.418556 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:59.418494    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bvb6"
Nov 01 03:05:59.418556 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:05:59.418462    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62mxc"
Nov 01 03:05:59.418986 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:59.418656    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzkbj" podUID="0a66c33f-29bc-4920-9d7a-0c45215514a0"
Nov 01 03:05:59.418986 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:59.418765    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bvb6" podUID="a972ee28-29fa-4755-adcd-b40d476c22f0"
Nov 01 03:05:59.418986 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:05:59.418833    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62mxc" podUID="e09a58a7-120e-4ec5-9aba-f15863267643"
Nov 01 03:06:00.493015 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:00.492800    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9nhh6" event={"ID":"1d8dba89-5673-4cd3-a6ac-914765f228d6","Type":"ContainerStarted","Data":"4c9b0734d980feadfe29210ae9fea30b82dc90202bdee5a0602b62d54ada4365"}
Nov 01 03:06:01.418183 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:01.418153    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62mxc"
Nov 01 03:06:01.418350 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:01.418261    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62mxc" podUID="e09a58a7-120e-4ec5-9aba-f15863267643"
Nov 01 03:06:01.418350 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:01.418316    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bvb6"
Nov 01 03:06:01.418463 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:01.418362    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bvb6" podUID="a972ee28-29fa-4755-adcd-b40d476c22f0"
Nov 01 03:06:01.418463 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:01.418397    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzkbj"
Nov 01 03:06:01.418463 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:01.418444    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzkbj" podUID="0a66c33f-29bc-4920-9d7a-0c45215514a0"
Nov 01 03:06:01.517146 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:01.517106    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-122.us-west-1.compute.internal" event={"ID":"5391522643ff9514bed3da110ddd8b71","Type":"ContainerStarted","Data":"c68ca34bd87df0550778d481600d8de4ff2ef0b14f4bc9ff408644b5babc1647"}
Nov 01 03:06:01.520958 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:01.520930    2578 generic.go:356] "Generic (PLEG): container finished" podID="bbcea9cdf04d326735591dc246925152" containerID="754ac17bd698683907c39572acbf0deeb9f733504c9436719964fb0dcba06f13" exitCode=0
Nov 01 03:06:01.521063 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:01.521002    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal" event={"ID":"bbcea9cdf04d326735591dc246925152","Type":"ContainerDied","Data":"754ac17bd698683907c39572acbf0deeb9f733504c9436719964fb0dcba06f13"}
Nov 01 03:06:01.526447 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:01.526419    2578 generic.go:356] "Generic (PLEG): container finished" podID="1d8dba89-5673-4cd3-a6ac-914765f228d6" containerID="4c9b0734d980feadfe29210ae9fea30b82dc90202bdee5a0602b62d54ada4365" exitCode=0
Nov 01 03:06:01.526552 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:01.526501    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9nhh6" event={"ID":"1d8dba89-5673-4cd3-a6ac-914765f228d6","Type":"ContainerDied","Data":"4c9b0734d980feadfe29210ae9fea30b82dc90202bdee5a0602b62d54ada4365"}
Nov 01 03:06:01.534156 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:01.534130    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-62ccn" event={"ID":"c014e8ac-58ca-4161-a778-ba58b0e84147","Type":"ContainerStarted","Data":"500a786ffdc54880fc02f18e9287a692f5135d6a96ac30ab65f732b887cf9e09"}
Nov 01 03:06:01.634166 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:01.633963    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-122.us-west-1.compute.internal" podStartSLOduration=6.63394348 podStartE2EDuration="6.63394348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 03:05:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:06:01.63392076 +0000 UTC m=+7.745173067" watchObservedRunningTime="2026-11-01 03:06:01.63394348 +0000 UTC m=+7.745195787"
Nov 01 03:06:02.020258 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:02.019512    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-62ccn" podStartSLOduration=1.955943055 podStartE2EDuration="7.019490091s" totalImagesPullingTime="5.063547036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:05:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:06:01.88499419 +0000 UTC m=+7.996246497" watchObservedRunningTime="2026-11-01 03:06:02.019490091 +0000 UTC m=+8.130742399"
Nov 01 03:06:02.540202 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:02.539525    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-stpgr" event={"ID":"73622c04-1428-4f7a-8faf-8eb17604f681","Type":"ContainerStarted","Data":"b525a278f391c063b8495c31647bbed7a34442b5f8d7439c4537202d83dd142a"}
Nov 01 03:06:02.543342 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:02.543318    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal" event={"ID":"bbcea9cdf04d326735591dc246925152","Type":"ContainerStarted","Data":"635ad1341241bcdf3d52441407e1e9b11354792e7a1f5a3d402b4a5f631c6561"}
Nov 01 03:06:02.665218 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:02.665089    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-stpgr" podStartSLOduration=1.409341314 podStartE2EDuration="7.665070678s" totalImagesPullingTime="6.255729364s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 03:05:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:06:02.663903356 +0000 UTC m=+8.775155662" watchObservedRunningTime="2026-11-01 03:06:02.665070678 +0000 UTC m=+8.776322985"
Nov 01 03:06:03.258734 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:03.258690    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a972ee28-29fa-4755-adcd-b40d476c22f0-original-pull-secret\") pod \"global-pull-secret-syncer-4bvb6\" (UID: \"a972ee28-29fa-4755-adcd-b40d476c22f0\") " pod="kube-system/global-pull-secret-syncer-4bvb6"
Nov 01 03:06:03.258734 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:03.258739    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a66c33f-29bc-4920-9d7a-0c45215514a0-metrics-certs\") pod \"network-metrics-daemon-mzkbj\" (UID: \"0a66c33f-29bc-4920-9d7a-0c45215514a0\") " pod="openshift-multus/network-metrics-daemon-mzkbj"
Nov 01 03:06:03.258954 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:03.258905    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 03:06:03.259012 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:03.258975    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a66c33f-29bc-4920-9d7a-0c45215514a0-metrics-certs podName:0a66c33f-29bc-4920-9d7a-0c45215514a0 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:11.258956769 +0000 UTC m=+17.370209070 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a66c33f-29bc-4920-9d7a-0c45215514a0-metrics-certs") pod "network-metrics-daemon-mzkbj" (UID: "0a66c33f-29bc-4920-9d7a-0c45215514a0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 03:06:03.259285 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:03.259155    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 03:06:03.259285 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:03.259214    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a972ee28-29fa-4755-adcd-b40d476c22f0-original-pull-secret podName:a972ee28-29fa-4755-adcd-b40d476c22f0 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:11.259199365 +0000 UTC m=+17.370451661 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a972ee28-29fa-4755-adcd-b40d476c22f0-original-pull-secret") pod "global-pull-secret-syncer-4bvb6" (UID: "a972ee28-29fa-4755-adcd-b40d476c22f0") : object "kube-system"/"original-pull-secret" not registered
Nov 01 03:06:03.359585 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:03.359494    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lbzv\" (UniqueName: \"kubernetes.io/projected/e09a58a7-120e-4ec5-9aba-f15863267643-kube-api-access-7lbzv\") pod \"network-check-target-62mxc\" (UID: \"e09a58a7-120e-4ec5-9aba-f15863267643\") " pod="openshift-network-diagnostics/network-check-target-62mxc"
Nov 01 03:06:03.359781 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:03.359720    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 03:06:03.359781 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:03.359746    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 03:06:03.359781 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:03.359758    2578 projected.go:196] Error preparing data for projected volume kube-api-access-7lbzv for pod openshift-network-diagnostics/network-check-target-62mxc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 03:06:03.359950 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:03.359810    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e09a58a7-120e-4ec5-9aba-f15863267643-kube-api-access-7lbzv podName:e09a58a7-120e-4ec5-9aba-f15863267643 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:11.359792646 +0000 UTC m=+17.471044936 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7lbzv" (UniqueName: "kubernetes.io/projected/e09a58a7-120e-4ec5-9aba-f15863267643-kube-api-access-7lbzv") pod "network-check-target-62mxc" (UID: "e09a58a7-120e-4ec5-9aba-f15863267643") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 03:06:03.417821 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:03.417787    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62mxc"
Nov 01 03:06:03.417988 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:03.417897    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62mxc" podUID="e09a58a7-120e-4ec5-9aba-f15863267643"
Nov 01 03:06:03.417988 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:03.417955    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bvb6"
Nov 01 03:06:03.418109 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:03.418013    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bvb6" podUID="a972ee28-29fa-4755-adcd-b40d476c22f0"
Nov 01 03:06:03.418109 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:03.418070    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzkbj"
Nov 01 03:06:03.418210 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:03.418156    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzkbj" podUID="0a66c33f-29bc-4920-9d7a-0c45215514a0"
Nov 01 03:06:03.548279 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:03.548247    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk" event={"ID":"998efdf0-7719-4768-bfd0-3da04d85d108","Type":"ContainerStarted","Data":"f2995abd1969478c590ca98806ca45f4c86eb2ff68323db3a3a8a7275c7a218e"}
Nov 01 03:06:04.554161 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:04.553427    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p5ght" event={"ID":"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1","Type":"ContainerStarted","Data":"1ddcc3eb7f79aa5e51b26535f23c2be1d30c946bad1ee1df576b19a090a13483"}
Nov 01 03:06:04.673095 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:04.673045    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal" podStartSLOduration=9.673030727 podStartE2EDuration="9.673030727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 03:05:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:06:02.780333887 +0000 UTC m=+8.891586193" watchObservedRunningTime="2026-11-01 03:06:04.673030727 +0000 UTC m=+10.784283029"
Nov 01 03:06:04.673285 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:04.673260    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-p5ght" podStartSLOduration=1.364272725 podStartE2EDuration="9.673254822s" totalImagesPullingTime="8.308982097s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 03:05:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:06:04.672889723 +0000 UTC m=+10.784142054" watchObservedRunningTime="2026-11-01 03:06:04.673254822 +0000 UTC m=+10.784507128"
Nov 01 03:06:05.089570 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:05.089493    2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 03:06:05.208441 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:05.208411    2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-62ccn"
Nov 01 03:06:05.209248 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:05.209228    2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-62ccn"
Nov 01 03:06:05.418693 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:05.418636    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62mxc"
Nov 01 03:06:05.418693 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:05.418654    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzkbj"
Nov 01 03:06:05.418908 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:05.418822    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62mxc" podUID="e09a58a7-120e-4ec5-9aba-f15863267643"
Nov 01 03:06:05.419205 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:05.419177    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bvb6"
Nov 01 03:06:05.419324 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:05.419292    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bvb6" podUID="a972ee28-29fa-4755-adcd-b40d476c22f0"
Nov 01 03:06:05.419406 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:05.419386    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzkbj" podUID="0a66c33f-29bc-4920-9d7a-0c45215514a0"
Nov 01 03:06:05.483311 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:05.483128    2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T03:06:05.089514947Z","UUID":"c5a6002d-423f-4019-8a96-7d57062159a9","Handler":null,"Name":"","Endpoint":""}
Nov 01 03:06:05.485047 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:05.485022    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 03:06:05.485188 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:05.485056    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 03:06:05.558356 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:05.557837    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-77nf7" event={"ID":"482aebaa-cb7f-4be2-991a-7fb4cd801c7f","Type":"ContainerStarted","Data":"e44665db3955103af4ea4e35d039782625daa6aaedb22961bc58c99c4afb9a56"}
Nov 01 03:06:05.561154 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:05.561128    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk" event={"ID":"998efdf0-7719-4768-bfd0-3da04d85d108","Type":"ContainerStarted","Data":"c92ffa2b4dd161b7f10bff393e5df207e74935362d5fd600c2399fb21df47f47"}
Nov 01 03:06:05.674202 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:05.673981    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-77nf7" podStartSLOduration=3.318028929 podStartE2EDuration="11.673961336s" totalImagesPullingTime="8.355932407s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 03:05:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:06:05.673368151 +0000 UTC m=+11.784620457" watchObservedRunningTime="2026-11-01 03:06:05.673961336 +0000 UTC m=+11.785213643"
Nov 01 03:06:06.565835 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:06.565802    2578 generic.go:356] "Generic (PLEG): container finished" podID="1d8dba89-5673-4cd3-a6ac-914765f228d6" containerID="f2b517a8c456aa1b611d12ea901c5cdef22c2bed4f1e306b8e36ce6a1a6c1936" exitCode=0
Nov 01 03:06:06.566386 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:06.565889    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9nhh6" event={"ID":"1d8dba89-5673-4cd3-a6ac-914765f228d6","Type":"ContainerDied","Data":"f2b517a8c456aa1b611d12ea901c5cdef22c2bed4f1e306b8e36ce6a1a6c1936"}
Nov 01 03:06:06.568001 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:06.567976    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ksj2n" event={"ID":"ca0be51d-87a4-462d-9c83-fcfb8bf224e8","Type":"ContainerStarted","Data":"e14fb1d8dc1ee8a561017b9835eec5dd5650a413feacb4e324ea3697e8b8325e"}
Nov 01 03:06:06.889017 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:06.888923    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ksj2n" podStartSLOduration=3.285639726 podStartE2EDuration="11.888907977s" totalImagesPullingTime="8.603268251s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 03:05:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:06:06.888443557 +0000 UTC m=+12.999695866" watchObservedRunningTime="2026-11-01 03:06:06.888907977 +0000 UTC m=+13.000160282"
Nov 01 03:06:07.417730 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:07.417705    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzkbj"
Nov 01 03:06:07.417867 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:07.417743    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62mxc"
Nov 01 03:06:07.417867 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:07.417741    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bvb6"
Nov 01 03:06:07.417867 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:07.417821    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzkbj" podUID="0a66c33f-29bc-4920-9d7a-0c45215514a0"
Nov 01 03:06:07.417970 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:07.417887    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62mxc" podUID="e09a58a7-120e-4ec5-9aba-f15863267643"
Nov 01 03:06:07.417970 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:07.417932    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bvb6" podUID="a972ee28-29fa-4755-adcd-b40d476c22f0"
Nov 01 03:06:07.572273 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:07.572239    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-l892q" event={"ID":"8190fea3-4e71-4250-9a0f-3a1d78dd32db","Type":"ContainerStarted","Data":"c765426d0c72cf173a196ad3e499ee63753630991411800171769ba313a7eff0"}
Nov 01 03:06:07.574838 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:07.574807    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk" event={"ID":"998efdf0-7719-4768-bfd0-3da04d85d108","Type":"ContainerStarted","Data":"585a27bfcfe4258cd68c6602e9c62976a6d280bf01c37eed68fd52c2751566a8"}
Nov 01 03:06:07.689490 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:07.689403    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-l892q" podStartSLOduration=1.227513502 podStartE2EDuration="12.689390962s" totalImagesPullingTime="11.46187746s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 03:05:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:06:07.689254807 +0000 UTC m=+13.800507112" watchObservedRunningTime="2026-11-01 03:06:07.689390962 +0000 UTC m=+13.800643268"
Nov 01 03:06:07.806480 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:07.806435    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk" podStartSLOduration=2.562454406 podStartE2EDuration="12.806418655s" totalImagesPullingTime="10.243964249s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 03:05:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:06:07.806384255 +0000 UTC m=+13.917636560" watchObservedRunningTime="2026-11-01 03:06:07.806418655 +0000 UTC m=+13.917670960"
Nov 01 03:06:08.212461 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:08.212391    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-62ccn"
Nov 01 03:06:08.212629 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:08.212538    2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 03:06:08.213502 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:08.213475    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-62ccn"
Nov 01 03:06:08.579603 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:08.579507    2578 generic.go:356] "Generic (PLEG): container finished" podID="1d8dba89-5673-4cd3-a6ac-914765f228d6" containerID="423f3551e2412f97df3d5cc444e868096bf40e1be43578ff42f38b3d359477dc" exitCode=0
Nov 01 03:06:08.580129 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:08.579608    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9nhh6" event={"ID":"1d8dba89-5673-4cd3-a6ac-914765f228d6","Type":"ContainerDied","Data":"423f3551e2412f97df3d5cc444e868096bf40e1be43578ff42f38b3d359477dc"}
Nov 01 03:06:09.417858 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:09.417600    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62mxc"
Nov 01 03:06:09.418026 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:09.417624    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzkbj"
Nov 01 03:06:09.418026 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:09.417958    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62mxc" podUID="e09a58a7-120e-4ec5-9aba-f15863267643"
Nov 01 03:06:09.418026 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:09.417625    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bvb6"
Nov 01 03:06:09.418169 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:09.418029    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzkbj" podUID="0a66c33f-29bc-4920-9d7a-0c45215514a0"
Nov 01 03:06:09.418169 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:09.418095    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bvb6" podUID="a972ee28-29fa-4755-adcd-b40d476c22f0"
Nov 01 03:06:10.586541 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:10.586504    2578 generic.go:356] "Generic (PLEG): container finished" podID="1d8dba89-5673-4cd3-a6ac-914765f228d6" containerID="bc59703f6577f9703c11bd045736d591a4fff02da699cf4adea2a08b3130dc85" exitCode=0
Nov 01 03:06:10.586973 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:10.586589    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9nhh6" event={"ID":"1d8dba89-5673-4cd3-a6ac-914765f228d6","Type":"ContainerDied","Data":"bc59703f6577f9703c11bd045736d591a4fff02da699cf4adea2a08b3130dc85"}
Nov 01 03:06:11.325036 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:11.324963    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a972ee28-29fa-4755-adcd-b40d476c22f0-original-pull-secret\") pod \"global-pull-secret-syncer-4bvb6\" (UID: \"a972ee28-29fa-4755-adcd-b40d476c22f0\") " pod="kube-system/global-pull-secret-syncer-4bvb6"
Nov 01 03:06:11.325188 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:11.325053    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a66c33f-29bc-4920-9d7a-0c45215514a0-metrics-certs\") pod \"network-metrics-daemon-mzkbj\" (UID: \"0a66c33f-29bc-4920-9d7a-0c45215514a0\") " pod="openshift-multus/network-metrics-daemon-mzkbj"
Nov 01 03:06:11.325188 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:11.325113    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 03:06:11.325188 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:11.325155    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 03:06:11.325188 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:11.325179    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a972ee28-29fa-4755-adcd-b40d476c22f0-original-pull-secret podName:a972ee28-29fa-4755-adcd-b40d476c22f0 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:27.325160828 +0000 UTC m=+33.436413135 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a972ee28-29fa-4755-adcd-b40d476c22f0-original-pull-secret") pod "global-pull-secret-syncer-4bvb6" (UID: "a972ee28-29fa-4755-adcd-b40d476c22f0") : object "kube-system"/"original-pull-secret" not registered
Nov 01 03:06:11.325310 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:11.325195    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a66c33f-29bc-4920-9d7a-0c45215514a0-metrics-certs podName:0a66c33f-29bc-4920-9d7a-0c45215514a0 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:27.325187761 +0000 UTC m=+33.436440045 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a66c33f-29bc-4920-9d7a-0c45215514a0-metrics-certs") pod "network-metrics-daemon-mzkbj" (UID: "0a66c33f-29bc-4920-9d7a-0c45215514a0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 03:06:11.417908 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:11.417880    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62mxc"
Nov 01 03:06:11.418031 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:11.417963    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzkbj"
Nov 01 03:06:11.418076 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:11.418045    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bvb6"
Nov 01 03:06:11.418076 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:11.418055    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62mxc" podUID="e09a58a7-120e-4ec5-9aba-f15863267643"
Nov 01 03:06:11.418133 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:11.418113    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzkbj" podUID="0a66c33f-29bc-4920-9d7a-0c45215514a0"
Nov 01 03:06:11.418208 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:11.418186    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bvb6" podUID="a972ee28-29fa-4755-adcd-b40d476c22f0"
Nov 01 03:06:11.426065 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:11.426049    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lbzv\" (UniqueName: \"kubernetes.io/projected/e09a58a7-120e-4ec5-9aba-f15863267643-kube-api-access-7lbzv\") pod \"network-check-target-62mxc\" (UID: \"e09a58a7-120e-4ec5-9aba-f15863267643\") " pod="openshift-network-diagnostics/network-check-target-62mxc"
Nov 01 03:06:11.426171 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:11.426159    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 03:06:11.426206 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:11.426174    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 03:06:11.426206 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:11.426183    2578 projected.go:196] Error preparing data for projected volume kube-api-access-7lbzv for pod openshift-network-diagnostics/network-check-target-62mxc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 03:06:11.426267 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:11.426222    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e09a58a7-120e-4ec5-9aba-f15863267643-kube-api-access-7lbzv podName:e09a58a7-120e-4ec5-9aba-f15863267643 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:27.426209579 +0000 UTC m=+33.537461863 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7lbzv" (UniqueName: "kubernetes.io/projected/e09a58a7-120e-4ec5-9aba-f15863267643-kube-api-access-7lbzv") pod "network-check-target-62mxc" (UID: "e09a58a7-120e-4ec5-9aba-f15863267643") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 03:06:12.592801 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:12.592640    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq" event={"ID":"9e212e37-6c34-4b3c-941c-b38621f1f184","Type":"ContainerStarted","Data":"bca77a7a72be903c62ad7d90058114d623dbe0414a1b88bc17ddd2d0fe7a1174"}
Nov 01 03:06:12.593202 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:12.592833    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq" event={"ID":"9e212e37-6c34-4b3c-941c-b38621f1f184","Type":"ContainerStarted","Data":"0ba68a9b5afe9b37e0b7dc5bbb86191dbe19908e20a363e665997cd2503b022b"}
Nov 01 03:06:12.593202 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:12.592868    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq" event={"ID":"9e212e37-6c34-4b3c-941c-b38621f1f184","Type":"ContainerStarted","Data":"8e0f9156d6e68fd2173a55354b2b8c188a7b283ce9ff1b5aacaa967a8768867c"}
Nov 01 03:06:12.593202 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:12.592900    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq" event={"ID":"9e212e37-6c34-4b3c-941c-b38621f1f184","Type":"ContainerStarted","Data":"1a553f47810c1b72c435802eb69f6c4f5dc45f3a42e5e7bddd1922afc8afd8d8"}
Nov 01 03:06:12.593202 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:12.592928    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq" event={"ID":"9e212e37-6c34-4b3c-941c-b38621f1f184","Type":"ContainerStarted","Data":"eec850fb2e655fecfc5cf1ad1103f35e428de488e6d5e8802140472372998e41"}
Nov 01 03:06:12.593202 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:12.592958    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq" event={"ID":"9e212e37-6c34-4b3c-941c-b38621f1f184","Type":"ContainerStarted","Data":"7d085842f49a6340274261e3fe492e5ce420a61941dfc66e362eef9d6958474d"}
Nov 01 03:06:13.417463 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:13.417437    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62mxc"
Nov 01 03:06:13.417583 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:13.417515    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzkbj"
Nov 01 03:06:13.417583 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:13.417533    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bvb6"
Nov 01 03:06:13.417668 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:13.417632    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62mxc" podUID="e09a58a7-120e-4ec5-9aba-f15863267643"
Nov 01 03:06:13.417713 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:13.417699    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzkbj" podUID="0a66c33f-29bc-4920-9d7a-0c45215514a0"
Nov 01 03:06:13.417750 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:13.417742    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bvb6" podUID="a972ee28-29fa-4755-adcd-b40d476c22f0"
Nov 01 03:06:13.596383 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:13.596349    2578 generic.go:356] "Generic (PLEG): container finished" podID="1d8dba89-5673-4cd3-a6ac-914765f228d6" containerID="945ab6571529e03188314b47ba85dc7c4207fb40514db7d5b7d7674bddf16841" exitCode=0
Nov 01 03:06:13.596866 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:13.596403    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9nhh6" event={"ID":"1d8dba89-5673-4cd3-a6ac-914765f228d6","Type":"ContainerDied","Data":"945ab6571529e03188314b47ba85dc7c4207fb40514db7d5b7d7674bddf16841"}
Nov 01 03:06:14.601218 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:14.601185    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq" event={"ID":"9e212e37-6c34-4b3c-941c-b38621f1f184","Type":"ContainerStarted","Data":"abaacb4beaeb2a569426a570b11e199871d5d0cc0d2a248c77b377498fe057f6"}
Nov 01 03:06:14.603984 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:14.603958    2578 generic.go:356] "Generic (PLEG): container finished" podID="1d8dba89-5673-4cd3-a6ac-914765f228d6" containerID="d25851b3aff82b1af421546f23051319145235925704d99408a0fd13c6d4349a" exitCode=0
Nov 01 03:06:14.604084 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:14.604001    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9nhh6" event={"ID":"1d8dba89-5673-4cd3-a6ac-914765f228d6","Type":"ContainerDied","Data":"d25851b3aff82b1af421546f23051319145235925704d99408a0fd13c6d4349a"}
Nov 01 03:06:15.417474 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:15.417441    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bvb6"
Nov 01 03:06:15.417675 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:15.417544    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzkbj"
Nov 01 03:06:15.417675 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:15.417546    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bvb6" podUID="a972ee28-29fa-4755-adcd-b40d476c22f0"
Nov 01 03:06:15.417675 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:15.417577    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62mxc"
Nov 01 03:06:15.417805 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:15.417692    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzkbj" podUID="0a66c33f-29bc-4920-9d7a-0c45215514a0"
Nov 01 03:06:15.417805 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:15.417784    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62mxc" podUID="e09a58a7-120e-4ec5-9aba-f15863267643"
Nov 01 03:06:15.608819 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:15.608791    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9nhh6" event={"ID":"1d8dba89-5673-4cd3-a6ac-914765f228d6","Type":"ContainerStarted","Data":"7bb7e3dcf75791c207357c8ffccfed89a5320c2e4db17e0d3857f85e3ca651b2"}
Nov 01 03:06:15.731484 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:15.731409    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9nhh6" podStartSLOduration=7.585484879 podStartE2EDuration="20.731396741s" totalImagesPullingTime="13.145911862s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 03:05:55 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:06:15.730675088 +0000 UTC m=+21.841927393" watchObservedRunningTime="2026-11-01 03:06:15.731396741 +0000 UTC m=+21.842649046"
Nov 01 03:06:16.614000 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:16.613819    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq" event={"ID":"9e212e37-6c34-4b3c-941c-b38621f1f184","Type":"ContainerStarted","Data":"e11f9468fbeccc852887279ad49279c8930e091e5ee26a227a97352de44bdfe6"}
Nov 01 03:06:16.883008 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:16.882916    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq" podStartSLOduration=7.291498012 podStartE2EDuration="22.88290308s" totalImagesPullingTime="15.591405068s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 03:05:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:06:16.882591368 +0000 UTC m=+22.993843659" watchObservedRunningTime="2026-11-01 03:06:16.88290308 +0000 UTC m=+22.994155385"
Nov 01 03:06:17.417666 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:17.417634    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62mxc"
Nov 01 03:06:17.417805 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:17.417641    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bvb6"
Nov 01 03:06:17.417805 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:17.417745    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62mxc" podUID="e09a58a7-120e-4ec5-9aba-f15863267643"
Nov 01 03:06:17.417805 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:17.417758    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzkbj"
Nov 01 03:06:17.417924 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:17.417803    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bvb6" podUID="a972ee28-29fa-4755-adcd-b40d476c22f0"
Nov 01 03:06:17.417924 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:17.417862    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzkbj" podUID="0a66c33f-29bc-4920-9d7a-0c45215514a0"
Nov 01 03:06:17.616699 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:17.616670    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:17.617080 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:17.616758    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:17.617080 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:17.616808    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:17.631355 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:17.631333    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:17.631473 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:17.631431    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:18.791426 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:18.791390    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-62mxc"]
Nov 01 03:06:18.791906 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:18.791501    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62mxc"
Nov 01 03:06:18.791906 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:18.791619    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62mxc" podUID="e09a58a7-120e-4ec5-9aba-f15863267643"
Nov 01 03:06:18.792263 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:18.792241    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4bvb6"]
Nov 01 03:06:18.792364 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:18.792328    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bvb6"
Nov 01 03:06:18.792415 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:18.792405    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bvb6" podUID="a972ee28-29fa-4755-adcd-b40d476c22f0"
Nov 01 03:06:18.793692 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:18.793668    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mzkbj"]
Nov 01 03:06:18.793779 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:18.793765    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzkbj"
Nov 01 03:06:18.793883 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:18.793864    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzkbj" podUID="0a66c33f-29bc-4920-9d7a-0c45215514a0"
Nov 01 03:06:20.418621 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:20.418521    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bvb6"
Nov 01 03:06:20.418621 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:20.418575    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzkbj"
Nov 01 03:06:20.419020 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:20.418650    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bvb6" podUID="a972ee28-29fa-4755-adcd-b40d476c22f0"
Nov 01 03:06:20.419020 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:20.418721    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzkbj" podUID="0a66c33f-29bc-4920-9d7a-0c45215514a0"
Nov 01 03:06:20.419020 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:20.418743    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62mxc"
Nov 01 03:06:20.419020 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:20.418825    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62mxc" podUID="e09a58a7-120e-4ec5-9aba-f15863267643"
Nov 01 03:06:22.417746 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:22.417673    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bvb6"
Nov 01 03:06:22.417746 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:22.417687    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzkbj"
Nov 01 03:06:22.417746 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:22.417699    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62mxc"
Nov 01 03:06:22.418236 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:22.417770    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4bvb6" podUID="a972ee28-29fa-4755-adcd-b40d476c22f0"
Nov 01 03:06:22.418236 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:22.417845    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mzkbj" podUID="0a66c33f-29bc-4920-9d7a-0c45215514a0"
Nov 01 03:06:22.418236 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:22.417931    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-62mxc" podUID="e09a58a7-120e-4ec5-9aba-f15863267643"
Nov 01 03:06:24.124358 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.124200    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeReady"
Nov 01 03:06:24.124663 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.124450    2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 01 03:06:24.242918 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.242758    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-j6kdp"]
Nov 01 03:06:24.302281 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.302250    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fznsl"]
Nov 01 03:06:24.302457 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.302407    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-j6kdp"
Nov 01 03:06:24.320990 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.320968    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-62ws6"]
Nov 01 03:06:24.321092 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.321047    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-fznsl"
Nov 01 03:06:24.347163 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.347140    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-8plk2"]
Nov 01 03:06:24.347263 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.347244    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-62ws6"
Nov 01 03:06:24.363180 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.363159    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 01 03:06:24.363180 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.363169    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 01 03:06:24.368014 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.367997    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-48ssl"]
Nov 01 03:06:24.368132 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.368118    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-8plk2"
Nov 01 03:06:24.381033 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.381018    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-t6tw7"]
Nov 01 03:06:24.381126 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.381113    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-48ssl"
Nov 01 03:06:24.403205 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.403184    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chqll"]
Nov 01 03:06:24.403300 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.403285    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-t6tw7"
Nov 01 03:06:24.410886 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.410866    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-95fqh\""
Nov 01 03:06:24.419537 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.419516    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chqll"
Nov 01 03:06:24.422339 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.422316    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxfvf\" (UniqueName: \"kubernetes.io/projected/ffdeddae-4086-4fb5-b9f8-c8ffcb765925-kube-api-access-fxfvf\") pod \"kas-connection-checker-685895c7c9-j6kdp\" (UID: \"ffdeddae-4086-4fb5-b9f8-c8ffcb765925\") " pod="kube-system/kas-connection-checker-685895c7c9-j6kdp"
Nov 01 03:06:24.422543 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.422515    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5fh9\" (UniqueName: \"kubernetes.io/projected/bcbdd4ff-e252-4aee-9418-9b1b05eed574-kube-api-access-z5fh9\") pod \"volume-data-source-validator-58c9f5f46d-62ws6\" (UID: \"bcbdd4ff-e252-4aee-9418-9b1b05eed574\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-62ws6"
Nov 01 03:06:24.422674 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.422582    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbhvz\" (UniqueName: \"kubernetes.io/projected/e60132c0-948e-4f84-883e-d22564f210e1-kube-api-access-wbhvz\") pod \"kas-connection-checker-685895c7c9-fznsl\" (UID: \"e60132c0-948e-4f84-883e-d22564f210e1\") " pod="kube-system/kas-connection-checker-685895c7c9-fznsl"
Nov 01 03:06:24.424712 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.424694    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-xbdmd"]
Nov 01 03:06:24.447576 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.447537    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-jh955"]
Nov 01 03:06:24.447672 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.447641    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xbdmd"
Nov 01 03:06:24.460435 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.460416    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 01 03:06:24.460726 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.460551    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 03:06:24.460726 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.460634    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 01 03:06:24.460726 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.460700    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 01 03:06:24.461135 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.461102    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 03:06:24.461745 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.461728    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gtk6f"]
Nov 01 03:06:24.461837 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.461805    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62mxc"
Nov 01 03:06:24.461837 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.461805    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bvb6"
Nov 01 03:06:24.461837 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.461830    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-jh955"
Nov 01 03:06:24.462398 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.462385    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzkbj"
Nov 01 03:06:24.477650 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.477634    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 01 03:06:24.481160 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.481145    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-cbzc6"]
Nov 01 03:06:24.481271 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.481256    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gtk6f"
Nov 01 03:06:24.497442 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.497423    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-58bc694f-l7vm9"]
Nov 01 03:06:24.497539 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.497527    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbzc6"
Nov 01 03:06:24.510674 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.510646    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 01 03:06:24.510674 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.510654    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 01 03:06:24.510862 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.510849    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 01 03:06:24.510906 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.510850    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 01 03:06:24.511357 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.511343    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-pn9xg\""
Nov 01 03:06:24.511445 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.511402    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-cdgq2\""
Nov 01 03:06:24.511445 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.511407    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-bt25w\""
Nov 01 03:06:24.511521 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.511467    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-n9bgf\""
Nov 01 03:06:24.513408 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.513393    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6f44999c56-64k9p"]
Nov 01 03:06:24.513596 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.513582    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:24.517472 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.517457    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 03:06:24.518374 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.518360    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-tdml4\""
Nov 01 03:06:24.518449 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.518384    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 01 03:06:24.518508 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.518461    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 03:06:24.518508 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.518465    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z9gxp\""
Nov 01 03:06:24.518627 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.518530    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wgrq6\""
Nov 01 03:06:24.518627 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.518578    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-74fv2\""
Nov 01 03:06:24.522848 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.522832    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5fh9\" (UniqueName: \"kubernetes.io/projected/bcbdd4ff-e252-4aee-9418-9b1b05eed574-kube-api-access-z5fh9\") pod \"volume-data-source-validator-58c9f5f46d-62ws6\" (UID: \"bcbdd4ff-e252-4aee-9418-9b1b05eed574\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-62ws6"
Nov 01 03:06:24.522950 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.522861    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbhvz\" (UniqueName: \"kubernetes.io/projected/e60132c0-948e-4f84-883e-d22564f210e1-kube-api-access-wbhvz\") pod \"kas-connection-checker-685895c7c9-fznsl\" (UID: \"e60132c0-948e-4f84-883e-d22564f210e1\") " pod="kube-system/kas-connection-checker-685895c7c9-fznsl"
Nov 01 03:06:24.522950 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.522893    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7dbf71ad-8399-4a40-87d7-98746e830fa9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-t6tw7\" (UID: \"7dbf71ad-8399-4a40-87d7-98746e830fa9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-t6tw7"
Nov 01 03:06:24.522950 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.522922    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eee1338d-e2a0-4a45-8f74-b81eb7d23019-serving-cert\") pod \"insights-operator-7c4986d449-48ssl\" (UID: \"eee1338d-e2a0-4a45-8f74-b81eb7d23019\") " pod="openshift-insights/insights-operator-7c4986d449-48ssl"
Nov 01 03:06:24.523062 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.522957    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/14e2ac70-be2e-4e08-b093-e8a93d6f985a-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-chqll\" (UID: \"14e2ac70-be2e-4e08-b093-e8a93d6f985a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chqll"
Nov 01 03:06:24.523062 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.523026    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67zjh\" (UniqueName: \"kubernetes.io/projected/581612ca-5a0b-4426-9d07-ee65f85e0412-kube-api-access-67zjh\") pod \"kas-connection-checker-685895c7c9-8plk2\" (UID: \"581612ca-5a0b-4426-9d07-ee65f85e0412\") " pod="kube-system/kas-connection-checker-685895c7c9-8plk2"
Nov 01 03:06:24.523139 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.523107    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eee1338d-e2a0-4a45-8f74-b81eb7d23019-tmp\") pod \"insights-operator-7c4986d449-48ssl\" (UID: \"eee1338d-e2a0-4a45-8f74-b81eb7d23019\") " pod="openshift-insights/insights-operator-7c4986d449-48ssl"
Nov 01 03:06:24.523139 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.523131    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eee1338d-e2a0-4a45-8f74-b81eb7d23019-service-ca-bundle\") pod \"insights-operator-7c4986d449-48ssl\" (UID: \"eee1338d-e2a0-4a45-8f74-b81eb7d23019\") " pod="openshift-insights/insights-operator-7c4986d449-48ssl"
Nov 01 03:06:24.523217 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.523168    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7dbf71ad-8399-4a40-87d7-98746e830fa9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-t6tw7\" (UID: \"7dbf71ad-8399-4a40-87d7-98746e830fa9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-t6tw7"
Nov 01 03:06:24.523288 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.523273    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fecfe177-e63f-490f-94bc-c307041640d5-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-xbdmd\" (UID: \"fecfe177-e63f-490f-94bc-c307041640d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xbdmd"
Nov 01 03:06:24.523334 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.523322    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/14e2ac70-be2e-4e08-b093-e8a93d6f985a-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-chqll\" (UID: \"14e2ac70-be2e-4e08-b093-e8a93d6f985a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chqll"
Nov 01 03:06:24.523372 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.523356    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/eee1338d-e2a0-4a45-8f74-b81eb7d23019-snapshots\") pod \"insights-operator-7c4986d449-48ssl\" (UID: \"eee1338d-e2a0-4a45-8f74-b81eb7d23019\") " pod="openshift-insights/insights-operator-7c4986d449-48ssl"
Nov 01 03:06:24.523411 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.523376    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eee1338d-e2a0-4a45-8f74-b81eb7d23019-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-48ssl\" (UID: \"eee1338d-e2a0-4a45-8f74-b81eb7d23019\") " pod="openshift-insights/insights-operator-7c4986d449-48ssl"
Nov 01 03:06:24.523411 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.523401    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b7cm\" (UniqueName: \"kubernetes.io/projected/eee1338d-e2a0-4a45-8f74-b81eb7d23019-kube-api-access-8b7cm\") pod \"insights-operator-7c4986d449-48ssl\" (UID: \"eee1338d-e2a0-4a45-8f74-b81eb7d23019\") " pod="openshift-insights/insights-operator-7c4986d449-48ssl"
Nov 01 03:06:24.523486 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.523425    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fecfe177-e63f-490f-94bc-c307041640d5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xbdmd\" (UID: \"fecfe177-e63f-490f-94bc-c307041640d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xbdmd"
Nov 01 03:06:24.523486 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.523465    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqgps\" (UniqueName: \"kubernetes.io/projected/fecfe177-e63f-490f-94bc-c307041640d5-kube-api-access-gqgps\") pod \"cluster-monitoring-operator-884576c57-xbdmd\" (UID: \"fecfe177-e63f-490f-94bc-c307041640d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xbdmd"
Nov 01 03:06:24.523551 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.523536    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxfvf\" (UniqueName: \"kubernetes.io/projected/ffdeddae-4086-4fb5-b9f8-c8ffcb765925-kube-api-access-fxfvf\") pod \"kas-connection-checker-685895c7c9-j6kdp\" (UID: \"ffdeddae-4086-4fb5-b9f8-c8ffcb765925\") " pod="kube-system/kas-connection-checker-685895c7c9-j6kdp"
Nov 01 03:06:24.523624 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.523573    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5z8d\" (UniqueName: \"kubernetes.io/projected/14e2ac70-be2e-4e08-b093-e8a93d6f985a-kube-api-access-p5z8d\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-chqll\" (UID: \"14e2ac70-be2e-4e08-b093-e8a93d6f985a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chqll"
Nov 01 03:06:24.529711 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.529691    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-ddbff58bc-5dlt4"]
Nov 01 03:06:24.529819 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.529803    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:24.538229 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.538208    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 01 03:06:24.538316 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.538298    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-vtrwc\""
Nov 01 03:06:24.551741 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.551721    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-fsr8j"]
Nov 01 03:06:24.551845 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.551831    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:24.561531 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.561513    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 01 03:06:24.561634 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.561597    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 01 03:06:24.561690 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.561640    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 01 03:06:24.561690 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.561679    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 01 03:06:24.561792 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.561702    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 01 03:06:24.561792 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.561732    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 01 03:06:24.561792 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.561747    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 01 03:06:24.561792 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.561752    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 03:06:24.561942 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.561803    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 01 03:06:24.561942 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.561842    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 01 03:06:24.562018 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.561999    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 03:06:24.562068 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.562030    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 01 03:06:24.562099 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.562058    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 01 03:06:24.562137 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.562102    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 01 03:06:24.562166 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.562146    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 01 03:06:24.562166 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.562161    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 01 03:06:24.562622 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.562607    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 01 03:06:24.562685 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.562632    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 01 03:06:24.562932 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.562919    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 01 03:06:24.565360 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.565348    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wps29"]
Nov 01 03:06:24.565457 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.565445    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fsr8j"
Nov 01 03:06:24.579708 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.579691    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zdrpv"]
Nov 01 03:06:24.579813 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.579800    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wps29"
Nov 01 03:06:24.586507 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.586489    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbhvz\" (UniqueName: \"kubernetes.io/projected/e60132c0-948e-4f84-883e-d22564f210e1-kube-api-access-wbhvz\") pod \"kas-connection-checker-685895c7c9-fznsl\" (UID: \"e60132c0-948e-4f84-883e-d22564f210e1\") " pod="kube-system/kas-connection-checker-685895c7c9-fznsl"
Nov 01 03:06:24.586603 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.586497    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxfvf\" (UniqueName: \"kubernetes.io/projected/ffdeddae-4086-4fb5-b9f8-c8ffcb765925-kube-api-access-fxfvf\") pod \"kas-connection-checker-685895c7c9-j6kdp\" (UID: \"ffdeddae-4086-4fb5-b9f8-c8ffcb765925\") " pod="kube-system/kas-connection-checker-685895c7c9-j6kdp"
Nov 01 03:06:24.586691 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.586672    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5fh9\" (UniqueName: \"kubernetes.io/projected/bcbdd4ff-e252-4aee-9418-9b1b05eed574-kube-api-access-z5fh9\") pod \"volume-data-source-validator-58c9f5f46d-62ws6\" (UID: \"bcbdd4ff-e252-4aee-9418-9b1b05eed574\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-62ws6"
Nov 01 03:06:24.599897 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.599878    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-j6kdp"]
Nov 01 03:06:24.599973 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.599908    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fznsl"]
Nov 01 03:06:24.599973 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.599924    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-62ws6"]
Nov 01 03:06:24.599973 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.599939    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-48ssl"]
Nov 01 03:06:24.599973 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.599953    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chqll"]
Nov 01 03:06:24.599973 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.599968    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gtk6f"]
Nov 01 03:06:24.600149 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.599971    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zdrpv"
Nov 01 03:06:24.600149 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.599985    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-xbdmd"]
Nov 01 03:06:24.600149 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.599999    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f44999c56-64k9p"]
Nov 01 03:06:24.600149 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.600016    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58bc694f-l7vm9"]
Nov 01 03:06:24.600149 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.600100    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-fsr8j"]
Nov 01 03:06:24.600149 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.600131    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-ddbff58bc-5dlt4"]
Nov 01 03:06:24.600149 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.600149    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-8plk2"]
Nov 01 03:06:24.600375 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.600169    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-cbzc6"]
Nov 01 03:06:24.600375 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.600190    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-jh955"]
Nov 01 03:06:24.600375 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.600205    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-t6tw7"]
Nov 01 03:06:24.600375 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.600220    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zdrpv"]
Nov 01 03:06:24.600375 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.600239    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wps29"]
Nov 01 03:06:24.612099 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.612084    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 01 03:06:24.612264 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.612251    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 01 03:06:24.612336 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.612322    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 01 03:06:24.612734 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.612724    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 01 03:06:24.612831 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.612819    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 01 03:06:24.613137 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.613126    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-p789z\""
Nov 01 03:06:24.613186 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.613175    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 01 03:06:24.613223 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.613214    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8s4ww\""
Nov 01 03:06:24.613332 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.613320    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-tmf7b\""
Nov 01 03:06:24.613362 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.613348    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 01 03:06:24.617200 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.617188    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-dsb6b\""
Nov 01 03:06:24.624465 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.624449    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtt7q\" (UniqueName: \"kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-kube-api-access-dtt7q\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:24.624510 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.624485    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7dbf71ad-8399-4a40-87d7-98746e830fa9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-t6tw7\" (UID: \"7dbf71ad-8399-4a40-87d7-98746e830fa9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-t6tw7"
Nov 01 03:06:24.624510 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.624501    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-tls\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:24.624582 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.624515    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9acc0734-3562-41fc-a05a-8c32cbf36443-installation-pull-secrets\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:24.624582 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.624550    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phjdn\" (UniqueName: \"kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-kube-api-access-phjdn\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:24.624684 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:24.624622    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 03:06:24.624684 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.624643    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fecfe177-e63f-490f-94bc-c307041640d5-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-xbdmd\" (UID: \"fecfe177-e63f-490f-94bc-c307041640d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xbdmd"
Nov 01 03:06:24.624684 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:24.624680    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dbf71ad-8399-4a40-87d7-98746e830fa9-networking-console-plugin-cert podName:7dbf71ad-8399-4a40-87d7-98746e830fa9 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:25.124666911 +0000 UTC m=+31.235919196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7dbf71ad-8399-4a40-87d7-98746e830fa9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-t6tw7" (UID: "7dbf71ad-8399-4a40-87d7-98746e830fa9") : secret "networking-console-plugin-cert" not found
Nov 01 03:06:24.624833 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.624727    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h78fl\" (UniqueName: \"kubernetes.io/projected/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-kube-api-access-h78fl\") pod \"router-default-ddbff58bc-5dlt4\" (UID: \"be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88\") " pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:24.624833 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.624794    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/14e2ac70-be2e-4e08-b093-e8a93d6f985a-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-chqll\" (UID: \"14e2ac70-be2e-4e08-b093-e8a93d6f985a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chqll"
Nov 01 03:06:24.624833 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.624821    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-tls\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:24.625003 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.624847    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/bbdccd97-7806-4669-a9cb-6aef70599c54-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-gtk6f\" (UID: \"bbdccd97-7806-4669-a9cb-6aef70599c54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gtk6f"
Nov 01 03:06:24.625003 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.624909    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/eee1338d-e2a0-4a45-8f74-b81eb7d23019-snapshots\") pod \"insights-operator-7c4986d449-48ssl\" (UID: \"eee1338d-e2a0-4a45-8f74-b81eb7d23019\") " pod="openshift-insights/insights-operator-7c4986d449-48ssl"
Nov 01 03:06:24.625003 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.624934    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d28ff750-94d5-4eff-a8ef-58215bfb99e3-trusted-ca\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:24.625003 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.624960    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9acc0734-3562-41fc-a05a-8c32cbf36443-trusted-ca\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:24.625003 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.624985    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5z8d\" (UniqueName: \"kubernetes.io/projected/14e2ac70-be2e-4e08-b093-e8a93d6f985a-kube-api-access-p5z8d\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-chqll\" (UID: \"14e2ac70-be2e-4e08-b093-e8a93d6f985a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chqll"
Nov 01 03:06:24.625247 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.625022    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/14e2ac70-be2e-4e08-b093-e8a93d6f985a-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-chqll\" (UID: \"14e2ac70-be2e-4e08-b093-e8a93d6f985a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chqll"
Nov 01 03:06:24.625247 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.625063    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/45c60c0c-dcff-421d-a9f2-4dc53e7637c9-config\") pod \"service-ca-operator-544d749c8c-cbzc6\" (UID: \"45c60c0c-dcff-421d-a9f2-4dc53e7637c9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbzc6"
Nov 01 03:06:24.625247 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.625107    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7dbf71ad-8399-4a40-87d7-98746e830fa9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-t6tw7\" (UID: \"7dbf71ad-8399-4a40-87d7-98746e830fa9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-t6tw7"
Nov 01 03:06:24.625247 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.625135    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f29ea61e-2a78-4c11-9566-f52497e135ff-serving-cert\") pod \"console-operator-7cd56b8dbc-jh955\" (UID: \"f29ea61e-2a78-4c11-9566-f52497e135ff\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jh955"
Nov 01 03:06:24.625247 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.625164    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-metrics-certs\") pod \"router-default-ddbff58bc-5dlt4\" (UID: \"be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88\") " pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:24.625247 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.625193    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67zjh\" (UniqueName: \"kubernetes.io/projected/581612ca-5a0b-4426-9d07-ee65f85e0412-kube-api-access-67zjh\") pod \"kas-connection-checker-685895c7c9-8plk2\" (UID: \"581612ca-5a0b-4426-9d07-ee65f85e0412\") " pod="kube-system/kas-connection-checker-685895c7c9-8plk2"
Nov 01 03:06:24.625247 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.625218    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-stats-auth\") pod \"router-default-ddbff58bc-5dlt4\" (UID: \"be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88\") " pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:24.625247 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.625244    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-default-certificate\") pod \"router-default-ddbff58bc-5dlt4\" (UID: \"be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88\") " pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:24.625734 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.625273    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eee1338d-e2a0-4a45-8f74-b81eb7d23019-tmp\") pod \"insights-operator-7c4986d449-48ssl\" (UID: \"eee1338d-e2a0-4a45-8f74-b81eb7d23019\") " pod="openshift-insights/insights-operator-7c4986d449-48ssl"
Nov 01 03:06:24.625734 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.625294    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d28ff750-94d5-4eff-a8ef-58215bfb99e3-installation-pull-secrets\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:24.625734 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.625320    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9acc0734-3562-41fc-a05a-8c32cbf36443-image-registry-private-configuration\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:24.625734 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.625361    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d28ff750-94d5-4eff-a8ef-58215bfb99e3-ca-trust-extracted\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:24.625734 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.625363    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/14e2ac70-be2e-4e08-b093-e8a93d6f985a-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-chqll\" (UID: \"14e2ac70-be2e-4e08-b093-e8a93d6f985a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chqll"
Nov 01 03:06:24.625734 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.625385    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-bound-sa-token\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:24.625734 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.625390    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/eee1338d-e2a0-4a45-8f74-b81eb7d23019-snapshots\") pod \"insights-operator-7c4986d449-48ssl\" (UID: \"eee1338d-e2a0-4a45-8f74-b81eb7d23019\") " pod="openshift-insights/insights-operator-7c4986d449-48ssl"
Nov 01 03:06:24.625734 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.625424    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/45c60c0c-dcff-421d-a9f2-4dc53e7637c9-serving-cert\") pod \"service-ca-operator-544d749c8c-cbzc6\" (UID: \"45c60c0c-dcff-421d-a9f2-4dc53e7637c9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbzc6"
Nov 01 03:06:24.625734 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.625471    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fecfe177-e63f-490f-94bc-c307041640d5-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-xbdmd\" (UID: \"fecfe177-e63f-490f-94bc-c307041640d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xbdmd"
Nov 01 03:06:24.625734 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.625515    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-service-ca-bundle\") pod \"router-default-ddbff58bc-5dlt4\" (UID: \"be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88\") " pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:24.625734 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.625575    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-certificates\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:24.625734 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.625636    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eee1338d-e2a0-4a45-8f74-b81eb7d23019-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-48ssl\" (UID: \"eee1338d-e2a0-4a45-8f74-b81eb7d23019\") " pod="openshift-insights/insights-operator-7c4986d449-48ssl"
Nov 01 03:06:24.625734 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.625655    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eee1338d-e2a0-4a45-8f74-b81eb7d23019-tmp\") pod \"insights-operator-7c4986d449-48ssl\" (UID: \"eee1338d-e2a0-4a45-8f74-b81eb7d23019\") " pod="openshift-insights/insights-operator-7c4986d449-48ssl"
Nov 01 03:06:24.625734 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.625682    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b7cm\" (UniqueName: \"kubernetes.io/projected/eee1338d-e2a0-4a45-8f74-b81eb7d23019-kube-api-access-8b7cm\") pod \"insights-operator-7c4986d449-48ssl\" (UID: \"eee1338d-e2a0-4a45-8f74-b81eb7d23019\") " pod="openshift-insights/insights-operator-7c4986d449-48ssl"
Nov 01 03:06:24.625734 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.625719    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5xx5\" (UniqueName: \"kubernetes.io/projected/45c60c0c-dcff-421d-a9f2-4dc53e7637c9-kube-api-access-m5xx5\") pod \"service-ca-operator-544d749c8c-cbzc6\" (UID: \"45c60c0c-dcff-421d-a9f2-4dc53e7637c9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbzc6"
Nov 01 03:06:24.626298 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.625798    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7dbf71ad-8399-4a40-87d7-98746e830fa9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-t6tw7\" (UID: \"7dbf71ad-8399-4a40-87d7-98746e830fa9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-t6tw7"
Nov 01 03:06:24.626298 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.625842    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fecfe177-e63f-490f-94bc-c307041640d5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xbdmd\" (UID: \"fecfe177-e63f-490f-94bc-c307041640d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xbdmd"
Nov 01 03:06:24.626298 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.625918    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f29ea61e-2a78-4c11-9566-f52497e135ff-trusted-ca\") pod \"console-operator-7cd56b8dbc-jh955\" (UID: \"f29ea61e-2a78-4c11-9566-f52497e135ff\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jh955"
Nov 01 03:06:24.626298 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:24.625936    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 03:06:24.626298 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:24.625990    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fecfe177-e63f-490f-94bc-c307041640d5-cluster-monitoring-operator-tls podName:fecfe177-e63f-490f-94bc-c307041640d5 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:25.125973094 +0000 UTC m=+31.237225382 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fecfe177-e63f-490f-94bc-c307041640d5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xbdmd" (UID: "fecfe177-e63f-490f-94bc-c307041640d5") : secret "cluster-monitoring-operator-tls" not found
Nov 01 03:06:24.626298 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.626013    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqgps\" (UniqueName: \"kubernetes.io/projected/fecfe177-e63f-490f-94bc-c307041640d5-kube-api-access-gqgps\") pod \"cluster-monitoring-operator-884576c57-xbdmd\" (UID: \"fecfe177-e63f-490f-94bc-c307041640d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xbdmd"
Nov 01 03:06:24.626298 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.626061    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvjgq\" (UniqueName: \"kubernetes.io/projected/f29ea61e-2a78-4c11-9566-f52497e135ff-kube-api-access-tvjgq\") pod \"console-operator-7cd56b8dbc-jh955\" (UID: \"f29ea61e-2a78-4c11-9566-f52497e135ff\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jh955"
Nov 01 03:06:24.626298 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.626099    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d28ff750-94d5-4eff-a8ef-58215bfb99e3-image-registry-private-configuration\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:24.626298 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.626188    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9acc0734-3562-41fc-a05a-8c32cbf36443-ca-trust-extracted\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:24.626298 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.626223    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eee1338d-e2a0-4a45-8f74-b81eb7d23019-serving-cert\") pod \"insights-operator-7c4986d449-48ssl\" (UID: \"eee1338d-e2a0-4a45-8f74-b81eb7d23019\") " pod="openshift-insights/insights-operator-7c4986d449-48ssl"
Nov 01 03:06:24.626298 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.626265    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f29ea61e-2a78-4c11-9566-f52497e135ff-config\") pod \"console-operator-7cd56b8dbc-jh955\" (UID: \"f29ea61e-2a78-4c11-9566-f52497e135ff\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jh955"
Nov 01 03:06:24.626686 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.626312    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-certificates\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:24.626686 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.626318    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eee1338d-e2a0-4a45-8f74-b81eb7d23019-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-48ssl\" (UID: \"eee1338d-e2a0-4a45-8f74-b81eb7d23019\") " pod="openshift-insights/insights-operator-7c4986d449-48ssl"
Nov 01 03:06:24.626686 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.626342    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bbdccd97-7806-4669-a9cb-6aef70599c54-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gtk6f\" (UID: \"bbdccd97-7806-4669-a9cb-6aef70599c54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gtk6f"
Nov 01 03:06:24.626686 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.626371    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlznq\" (UniqueName: \"kubernetes.io/projected/bbdccd97-7806-4669-a9cb-6aef70599c54-kube-api-access-qlznq\") pod \"cluster-samples-operator-6ddb6c8b94-gtk6f\" (UID: \"bbdccd97-7806-4669-a9cb-6aef70599c54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gtk6f"
Nov 01 03:06:24.626686 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.626425    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-bound-sa-token\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:24.626686 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.626472    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eee1338d-e2a0-4a45-8f74-b81eb7d23019-service-ca-bundle\") pod \"insights-operator-7c4986d449-48ssl\" (UID: \"eee1338d-e2a0-4a45-8f74-b81eb7d23019\") " pod="openshift-insights/insights-operator-7c4986d449-48ssl"
Nov 01 03:06:24.626877 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.626862    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eee1338d-e2a0-4a45-8f74-b81eb7d23019-service-ca-bundle\") pod \"insights-operator-7c4986d449-48ssl\" (UID: \"eee1338d-e2a0-4a45-8f74-b81eb7d23019\") " pod="openshift-insights/insights-operator-7c4986d449-48ssl"
Nov 01 03:06:24.627180 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.627160    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/14e2ac70-be2e-4e08-b093-e8a93d6f985a-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-chqll\" (UID: \"14e2ac70-be2e-4e08-b093-e8a93d6f985a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chqll"
Nov 01 03:06:24.627251 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.627241    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-j6kdp"
Nov 01 03:06:24.628591 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.628574    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eee1338d-e2a0-4a45-8f74-b81eb7d23019-serving-cert\") pod \"insights-operator-7c4986d449-48ssl\" (UID: \"eee1338d-e2a0-4a45-8f74-b81eb7d23019\") " pod="openshift-insights/insights-operator-7c4986d449-48ssl"
Nov 01 03:06:24.648058 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.648037    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-fznsl"
Nov 01 03:06:24.662500 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.662418    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 03:06:24.662500 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.662453    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 01 03:06:24.662989 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.662503    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 03:06:24.662989 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.662521    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 01 03:06:24.662989 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.662593    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 03:06:24.662989 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.662690    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 01 03:06:24.663968 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.663952    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 01 03:06:24.677678 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.677655    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-62ws6"
Nov 01 03:06:24.689338 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.689308    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5z8d\" (UniqueName: \"kubernetes.io/projected/14e2ac70-be2e-4e08-b093-e8a93d6f985a-kube-api-access-p5z8d\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-chqll\" (UID: \"14e2ac70-be2e-4e08-b093-e8a93d6f985a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chqll"
Nov 01 03:06:24.690495 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.690456    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b7cm\" (UniqueName: \"kubernetes.io/projected/eee1338d-e2a0-4a45-8f74-b81eb7d23019-kube-api-access-8b7cm\") pod \"insights-operator-7c4986d449-48ssl\" (UID: \"eee1338d-e2a0-4a45-8f74-b81eb7d23019\") " pod="openshift-insights/insights-operator-7c4986d449-48ssl"
Nov 01 03:06:24.694498 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.694476    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqgps\" (UniqueName: \"kubernetes.io/projected/fecfe177-e63f-490f-94bc-c307041640d5-kube-api-access-gqgps\") pod \"cluster-monitoring-operator-884576c57-xbdmd\" (UID: \"fecfe177-e63f-490f-94bc-c307041640d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xbdmd"
Nov 01 03:06:24.694736 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.694718    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67zjh\" (UniqueName: \"kubernetes.io/projected/581612ca-5a0b-4426-9d07-ee65f85e0412-kube-api-access-67zjh\") pod \"kas-connection-checker-685895c7c9-8plk2\" (UID: \"581612ca-5a0b-4426-9d07-ee65f85e0412\") " pod="kube-system/kas-connection-checker-685895c7c9-8plk2"
Nov 01 03:06:24.697476 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.697452    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-8plk2"
Nov 01 03:06:24.710058 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.710038    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kkkrb\""
Nov 01 03:06:24.710058 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.710052    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 03:06:24.710190 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.710101    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 03:06:24.710190 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.710177    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-fdfjz\""
Nov 01 03:06:24.714504 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.714481    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-48ssl"
Nov 01 03:06:24.727459 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.727439    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h78fl\" (UniqueName: \"kubernetes.io/projected/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-kube-api-access-h78fl\") pod \"router-default-ddbff58bc-5dlt4\" (UID: \"be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88\") " pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:24.727585 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.727468    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-tls\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:24.727585 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.727489    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/bbdccd97-7806-4669-a9cb-6aef70599c54-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-gtk6f\" (UID: \"bbdccd97-7806-4669-a9cb-6aef70599c54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gtk6f"
Nov 01 03:06:24.727702 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:24.727594    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 03:06:24.727702 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:24.727609    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58bc694f-l7vm9: secret "image-registry-tls" not found
Nov 01 03:06:24.727702 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:24.727651    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-tls podName:d28ff750-94d5-4eff-a8ef-58215bfb99e3 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:25.227637912 +0000 UTC m=+31.338890197 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-tls") pod "image-registry-58bc694f-l7vm9" (UID: "d28ff750-94d5-4eff-a8ef-58215bfb99e3") : secret "image-registry-tls" not found
Nov 01 03:06:24.727702 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.727674    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d28ff750-94d5-4eff-a8ef-58215bfb99e3-trusted-ca\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:24.727702 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.727696    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9acc0734-3562-41fc-a05a-8c32cbf36443-trusted-ca\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:24.727942 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.727724    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/45c60c0c-dcff-421d-a9f2-4dc53e7637c9-config\") pod \"service-ca-operator-544d749c8c-cbzc6\" (UID: \"45c60c0c-dcff-421d-a9f2-4dc53e7637c9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbzc6"
Nov 01 03:06:24.727942 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.727745    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f29ea61e-2a78-4c11-9566-f52497e135ff-serving-cert\") pod \"console-operator-7cd56b8dbc-jh955\" (UID: \"f29ea61e-2a78-4c11-9566-f52497e135ff\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jh955"
Nov 01 03:06:24.727942 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.727766    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19871ddf-e177-4442-b592-8f1c7df24aaf-tmp-dir\") pod \"dns-default-wps29\" (UID: \"19871ddf-e177-4442-b592-8f1c7df24aaf\") " pod="openshift-dns/dns-default-wps29"
Nov 01 03:06:24.727942 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.727783    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-metrics-certs\") pod \"router-default-ddbff58bc-5dlt4\" (UID: \"be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88\") " pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:24.727942 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.727796    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/19871ddf-e177-4442-b592-8f1c7df24aaf-config-volume\") pod \"dns-default-wps29\" (UID: \"19871ddf-e177-4442-b592-8f1c7df24aaf\") " pod="openshift-dns/dns-default-wps29"
Nov 01 03:06:24.727942 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.727810    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19871ddf-e177-4442-b592-8f1c7df24aaf-metrics-tls\") pod \"dns-default-wps29\" (UID: \"19871ddf-e177-4442-b592-8f1c7df24aaf\") " pod="openshift-dns/dns-default-wps29"
Nov 01 03:06:24.727942 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.727828    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-stats-auth\") pod \"router-default-ddbff58bc-5dlt4\" (UID: \"be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88\") " pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:24.727942 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.727844    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-default-certificate\") pod \"router-default-ddbff58bc-5dlt4\" (UID: \"be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88\") " pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:24.727942 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.727867    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d28ff750-94d5-4eff-a8ef-58215bfb99e3-installation-pull-secrets\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:24.727942 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.727886    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9acc0734-3562-41fc-a05a-8c32cbf36443-image-registry-private-configuration\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:24.727942 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.727902    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9ppn\" (UniqueName: \"kubernetes.io/projected/2ab1baec-cb71-4942-a246-7920cd44526c-kube-api-access-t9ppn\") pod \"network-check-source-6f78d59f99-fsr8j\" (UID: \"2ab1baec-cb71-4942-a246-7920cd44526c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fsr8j"
Nov 01 03:06:24.727942 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.727928    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d28ff750-94d5-4eff-a8ef-58215bfb99e3-ca-trust-extracted\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:24.727942 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.727944    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-bound-sa-token\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:24.728544 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.727958    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/45c60c0c-dcff-421d-a9f2-4dc53e7637c9-serving-cert\") pod \"service-ca-operator-544d749c8c-cbzc6\" (UID: \"45c60c0c-dcff-421d-a9f2-4dc53e7637c9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbzc6"
Nov 01 03:06:24.728544 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.727972    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-service-ca-bundle\") pod \"router-default-ddbff58bc-5dlt4\" (UID: \"be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88\") " pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:24.728544 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.727988    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-certificates\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:24.728544 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.728008    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5xx5\" (UniqueName: \"kubernetes.io/projected/45c60c0c-dcff-421d-a9f2-4dc53e7637c9-kube-api-access-m5xx5\") pod \"service-ca-operator-544d749c8c-cbzc6\" (UID: \"45c60c0c-dcff-421d-a9f2-4dc53e7637c9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbzc6"
Nov 01 03:06:24.728544 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.728024    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpmsp\" (UniqueName: \"kubernetes.io/projected/a22c3f93-43f6-4908-9549-b966fb3c04e5-kube-api-access-dpmsp\") pod \"ingress-canary-zdrpv\" (UID: \"a22c3f93-43f6-4908-9549-b966fb3c04e5\") " pod="openshift-ingress-canary/ingress-canary-zdrpv"
Nov 01 03:06:24.728544 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.728046    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f29ea61e-2a78-4c11-9566-f52497e135ff-trusted-ca\") pod \"console-operator-7cd56b8dbc-jh955\" (UID: \"f29ea61e-2a78-4c11-9566-f52497e135ff\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jh955"
Nov 01 03:06:24.728544 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.728066    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvjgq\" (UniqueName: \"kubernetes.io/projected/f29ea61e-2a78-4c11-9566-f52497e135ff-kube-api-access-tvjgq\") pod \"console-operator-7cd56b8dbc-jh955\" (UID: \"f29ea61e-2a78-4c11-9566-f52497e135ff\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jh955"
Nov 01 03:06:24.728544 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.728086    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d28ff750-94d5-4eff-a8ef-58215bfb99e3-image-registry-private-configuration\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:24.728544 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.728103    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a22c3f93-43f6-4908-9549-b966fb3c04e5-cert\") pod \"ingress-canary-zdrpv\" (UID: \"a22c3f93-43f6-4908-9549-b966fb3c04e5\") " pod="openshift-ingress-canary/ingress-canary-zdrpv"
Nov 01 03:06:24.728544 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.728122    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9acc0734-3562-41fc-a05a-8c32cbf36443-ca-trust-extracted\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:24.728544 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.728140    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f29ea61e-2a78-4c11-9566-f52497e135ff-config\") pod \"console-operator-7cd56b8dbc-jh955\" (UID: \"f29ea61e-2a78-4c11-9566-f52497e135ff\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jh955"
Nov 01 03:06:24.728544 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.728157    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-certificates\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:24.728544 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.728174    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bbdccd97-7806-4669-a9cb-6aef70599c54-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gtk6f\" (UID: \"bbdccd97-7806-4669-a9cb-6aef70599c54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gtk6f"
Nov 01 03:06:24.728544 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.728189    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlznq\" (UniqueName: \"kubernetes.io/projected/bbdccd97-7806-4669-a9cb-6aef70599c54-kube-api-access-qlznq\") pod \"cluster-samples-operator-6ddb6c8b94-gtk6f\" (UID: \"bbdccd97-7806-4669-a9cb-6aef70599c54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gtk6f"
Nov 01 03:06:24.728544 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.728204    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-bound-sa-token\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:24.728544 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.728225    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtt7q\" (UniqueName: \"kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-kube-api-access-dtt7q\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:24.729310 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.728263    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-tls\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:24.729310 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.728278    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9acc0734-3562-41fc-a05a-8c32cbf36443-installation-pull-secrets\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:24.729310 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.728292    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phjdn\" (UniqueName: \"kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-kube-api-access-phjdn\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:24.729310 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.728309    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z96gl\" (UniqueName: \"kubernetes.io/projected/19871ddf-e177-4442-b592-8f1c7df24aaf-kube-api-access-z96gl\") pod \"dns-default-wps29\" (UID: \"19871ddf-e177-4442-b592-8f1c7df24aaf\") " pod="openshift-dns/dns-default-wps29"
Nov 01 03:06:24.729310 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.728311    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/bbdccd97-7806-4669-a9cb-6aef70599c54-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-gtk6f\" (UID: \"bbdccd97-7806-4669-a9cb-6aef70599c54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gtk6f"
Nov 01 03:06:24.729310 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.728790    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-certificates\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:24.729310 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.728891    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/45c60c0c-dcff-421d-a9f2-4dc53e7637c9-config\") pod \"service-ca-operator-544d749c8c-cbzc6\" (UID: \"45c60c0c-dcff-421d-a9f2-4dc53e7637c9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbzc6"
Nov 01 03:06:24.729759 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:24.729485    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 03:06:24.729759 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:24.729535    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbdccd97-7806-4669-a9cb-6aef70599c54-samples-operator-tls podName:bbdccd97-7806-4669-a9cb-6aef70599c54 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:25.229519305 +0000 UTC m=+31.340771616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bbdccd97-7806-4669-a9cb-6aef70599c54-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-gtk6f" (UID: "bbdccd97-7806-4669-a9cb-6aef70599c54") : secret "samples-operator-tls" not found
Nov 01 03:06:24.731825 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.730532    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f29ea61e-2a78-4c11-9566-f52497e135ff-trusted-ca\") pod \"console-operator-7cd56b8dbc-jh955\" (UID: \"f29ea61e-2a78-4c11-9566-f52497e135ff\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jh955"
Nov 01 03:06:24.731825 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.730803    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9acc0734-3562-41fc-a05a-8c32cbf36443-trusted-ca\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:24.731825 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.731413    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-certificates\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:24.731825 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:24.731450    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 03:06:24.731825 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:24.731467    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f44999c56-64k9p: secret "image-registry-tls" not found
Nov 01 03:06:24.731825 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:24.731525    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 03:06:24.732148 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:24.731961    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-tls podName:9acc0734-3562-41fc-a05a-8c32cbf36443 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:25.231505176 +0000 UTC m=+31.342757477 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-tls") pod "image-registry-6f44999c56-64k9p" (UID: "9acc0734-3562-41fc-a05a-8c32cbf36443") : secret "image-registry-tls" not found
Nov 01 03:06:24.732148 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:24.731985    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-metrics-certs podName:be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:25.231972928 +0000 UTC m=+31.343225236 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-metrics-certs") pod "router-default-ddbff58bc-5dlt4" (UID: "be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88") : secret "router-metrics-certs-default" not found
Nov 01 03:06:24.732302 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.732194    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9acc0734-3562-41fc-a05a-8c32cbf36443-ca-trust-extracted\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:24.732506 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.732476    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d28ff750-94d5-4eff-a8ef-58215bfb99e3-trusted-ca\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:24.732610 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.732584    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f29ea61e-2a78-4c11-9566-f52497e135ff-config\") pod \"console-operator-7cd56b8dbc-jh955\" (UID: \"f29ea61e-2a78-4c11-9566-f52497e135ff\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jh955"
Nov 01 03:06:24.733309 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.732816    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d28ff750-94d5-4eff-a8ef-58215bfb99e3-ca-trust-extracted\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:24.734438 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:24.733751    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-service-ca-bundle podName:be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:25.233730187 +0000 UTC m=+31.344982489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-service-ca-bundle") pod "router-default-ddbff58bc-5dlt4" (UID: "be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88") : configmap references non-existent config key: service-ca.crt
Nov 01 03:06:24.734438 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.734366    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f29ea61e-2a78-4c11-9566-f52497e135ff-serving-cert\") pod \"console-operator-7cd56b8dbc-jh955\" (UID: \"f29ea61e-2a78-4c11-9566-f52497e135ff\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jh955"
Nov 01 03:06:24.738089 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.734780    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-stats-auth\") pod \"router-default-ddbff58bc-5dlt4\" (UID: \"be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88\") " pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:24.738089 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.736613    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d28ff750-94d5-4eff-a8ef-58215bfb99e3-installation-pull-secrets\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:24.738922 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.738701    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-default-certificate\") pod \"router-default-ddbff58bc-5dlt4\" (UID: \"be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88\") " pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:24.739623 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.739594    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/45c60c0c-dcff-421d-a9f2-4dc53e7637c9-serving-cert\") pod \"service-ca-operator-544d749c8c-cbzc6\" (UID: \"45c60c0c-dcff-421d-a9f2-4dc53e7637c9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbzc6"
Nov 01 03:06:24.739742 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.739723    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d28ff750-94d5-4eff-a8ef-58215bfb99e3-image-registry-private-configuration\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:24.739942 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.739922    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9acc0734-3562-41fc-a05a-8c32cbf36443-image-registry-private-configuration\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:24.740639 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.740620    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9acc0734-3562-41fc-a05a-8c32cbf36443-installation-pull-secrets\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:24.757720 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.755993    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chqll"
Nov 01 03:06:24.797171 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.796508    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h78fl\" (UniqueName: \"kubernetes.io/projected/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-kube-api-access-h78fl\") pod \"router-default-ddbff58bc-5dlt4\" (UID: \"be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88\") " pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:24.812857 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.807180    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-bound-sa-token\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:24.812857 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.812041    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5xx5\" (UniqueName: \"kubernetes.io/projected/45c60c0c-dcff-421d-a9f2-4dc53e7637c9-kube-api-access-m5xx5\") pod \"service-ca-operator-544d749c8c-cbzc6\" (UID: \"45c60c0c-dcff-421d-a9f2-4dc53e7637c9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbzc6"
Nov 01 03:06:24.813070 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.812698    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlznq\" (UniqueName: \"kubernetes.io/projected/bbdccd97-7806-4669-a9cb-6aef70599c54-kube-api-access-qlznq\") pod \"cluster-samples-operator-6ddb6c8b94-gtk6f\" (UID: \"bbdccd97-7806-4669-a9cb-6aef70599c54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gtk6f"
Nov 01 03:06:24.814678 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.814628    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtt7q\" (UniqueName: \"kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-kube-api-access-dtt7q\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:24.816733 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.816688    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-bound-sa-token\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:24.816844 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.816822    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvjgq\" (UniqueName: \"kubernetes.io/projected/f29ea61e-2a78-4c11-9566-f52497e135ff-kube-api-access-tvjgq\") pod \"console-operator-7cd56b8dbc-jh955\" (UID: \"f29ea61e-2a78-4c11-9566-f52497e135ff\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jh955"
Nov 01 03:06:24.818214 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.818194    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phjdn\" (UniqueName: \"kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-kube-api-access-phjdn\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:24.831231 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.830677    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z96gl\" (UniqueName: \"kubernetes.io/projected/19871ddf-e177-4442-b592-8f1c7df24aaf-kube-api-access-z96gl\") pod \"dns-default-wps29\" (UID: \"19871ddf-e177-4442-b592-8f1c7df24aaf\") " pod="openshift-dns/dns-default-wps29"
Nov 01 03:06:24.831231 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.830782    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19871ddf-e177-4442-b592-8f1c7df24aaf-tmp-dir\") pod \"dns-default-wps29\" (UID: \"19871ddf-e177-4442-b592-8f1c7df24aaf\") " pod="openshift-dns/dns-default-wps29"
Nov 01 03:06:24.831231 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.830823    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/19871ddf-e177-4442-b592-8f1c7df24aaf-config-volume\") pod \"dns-default-wps29\" (UID: \"19871ddf-e177-4442-b592-8f1c7df24aaf\") " pod="openshift-dns/dns-default-wps29"
Nov 01 03:06:24.831231 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.830845    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19871ddf-e177-4442-b592-8f1c7df24aaf-metrics-tls\") pod \"dns-default-wps29\" (UID: \"19871ddf-e177-4442-b592-8f1c7df24aaf\") " pod="openshift-dns/dns-default-wps29"
Nov 01 03:06:24.831231 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.830889    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9ppn\" (UniqueName: \"kubernetes.io/projected/2ab1baec-cb71-4942-a246-7920cd44526c-kube-api-access-t9ppn\") pod \"network-check-source-6f78d59f99-fsr8j\" (UID: \"2ab1baec-cb71-4942-a246-7920cd44526c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fsr8j"
Nov 01 03:06:24.831231 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.830959    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpmsp\" (UniqueName: \"kubernetes.io/projected/a22c3f93-43f6-4908-9549-b966fb3c04e5-kube-api-access-dpmsp\") pod \"ingress-canary-zdrpv\" (UID: \"a22c3f93-43f6-4908-9549-b966fb3c04e5\") " pod="openshift-ingress-canary/ingress-canary-zdrpv"
Nov 01 03:06:24.831231 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.831006    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a22c3f93-43f6-4908-9549-b966fb3c04e5-cert\") pod \"ingress-canary-zdrpv\" (UID: \"a22c3f93-43f6-4908-9549-b966fb3c04e5\") " pod="openshift-ingress-canary/ingress-canary-zdrpv"
Nov 01 03:06:24.831615 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:24.831441    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 03:06:24.831615 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:24.831506    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19871ddf-e177-4442-b592-8f1c7df24aaf-metrics-tls podName:19871ddf-e177-4442-b592-8f1c7df24aaf nodeName:}" failed. No retries permitted until 2026-11-01 03:06:25.331487901 +0000 UTC m=+31.442740207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19871ddf-e177-4442-b592-8f1c7df24aaf-metrics-tls") pod "dns-default-wps29" (UID: "19871ddf-e177-4442-b592-8f1c7df24aaf") : secret "dns-default-metrics-tls" not found
Nov 01 03:06:24.844509 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.844181    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19871ddf-e177-4442-b592-8f1c7df24aaf-tmp-dir\") pod \"dns-default-wps29\" (UID: \"19871ddf-e177-4442-b592-8f1c7df24aaf\") " pod="openshift-dns/dns-default-wps29"
Nov 01 03:06:24.845681 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.845641    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/19871ddf-e177-4442-b592-8f1c7df24aaf-config-volume\") pod \"dns-default-wps29\" (UID: \"19871ddf-e177-4442-b592-8f1c7df24aaf\") " pod="openshift-dns/dns-default-wps29"
Nov 01 03:06:24.846073 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.846025    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a22c3f93-43f6-4908-9549-b966fb3c04e5-cert\") pod \"ingress-canary-zdrpv\" (UID: \"a22c3f93-43f6-4908-9549-b966fb3c04e5\") " pod="openshift-ingress-canary/ingress-canary-zdrpv"
Nov 01 03:06:24.874441 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.874414    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbzc6"
Nov 01 03:06:24.894872 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.894851    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z96gl\" (UniqueName: \"kubernetes.io/projected/19871ddf-e177-4442-b592-8f1c7df24aaf-kube-api-access-z96gl\") pod \"dns-default-wps29\" (UID: \"19871ddf-e177-4442-b592-8f1c7df24aaf\") " pod="openshift-dns/dns-default-wps29"
Nov 01 03:06:24.895051 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.895033    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9ppn\" (UniqueName: \"kubernetes.io/projected/2ab1baec-cb71-4942-a246-7920cd44526c-kube-api-access-t9ppn\") pod \"network-check-source-6f78d59f99-fsr8j\" (UID: \"2ab1baec-cb71-4942-a246-7920cd44526c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fsr8j"
Nov 01 03:06:24.899912 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.899895    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpmsp\" (UniqueName: \"kubernetes.io/projected/a22c3f93-43f6-4908-9549-b966fb3c04e5-kube-api-access-dpmsp\") pod \"ingress-canary-zdrpv\" (UID: \"a22c3f93-43f6-4908-9549-b966fb3c04e5\") " pod="openshift-ingress-canary/ingress-canary-zdrpv"
Nov 01 03:06:24.953691 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.953664    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fsr8j"
Nov 01 03:06:24.990937 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:24.990906    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zdrpv"
Nov 01 03:06:25.078183 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:25.078160    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-j6kdp"]
Nov 01 03:06:25.081333 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:06:25.081303    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffdeddae_4086_4fb5_b9f8_c8ffcb765925.slice/crio-7a916b464e271033e5650cc96ca39397503853853c9742d5600a5f956a7d7d47 WatchSource:0}: Error finding container 7a916b464e271033e5650cc96ca39397503853853c9742d5600a5f956a7d7d47: Status 404 returned error can't find the container with id 7a916b464e271033e5650cc96ca39397503853853c9742d5600a5f956a7d7d47
Nov 01 03:06:25.083984 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:25.083960    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-62ws6"]
Nov 01 03:06:25.084926 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:25.084903    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fznsl"]
Nov 01 03:06:25.087133 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:06:25.087111    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode60132c0_948e_4f84_883e_d22564f210e1.slice/crio-4bbef179152e161a50c2a59d9194272f6473ebc2b3c2ece917e5743eb50a63f5 WatchSource:0}: Error finding container 4bbef179152e161a50c2a59d9194272f6473ebc2b3c2ece917e5743eb50a63f5: Status 404 returned error can't find the container with id 4bbef179152e161a50c2a59d9194272f6473ebc2b3c2ece917e5743eb50a63f5
Nov 01 03:06:25.087724 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:06:25.087696    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcbdd4ff_e252_4aee_9418_9b1b05eed574.slice/crio-f37006ff7bdb9b2342fb996deec818e8a3e5f535a2cbb2a0fbcd4cd18fd1771c WatchSource:0}: Error finding container f37006ff7bdb9b2342fb996deec818e8a3e5f535a2cbb2a0fbcd4cd18fd1771c: Status 404 returned error can't find the container with id f37006ff7bdb9b2342fb996deec818e8a3e5f535a2cbb2a0fbcd4cd18fd1771c
Nov 01 03:06:25.108535 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:25.108262    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-jh955"
Nov 01 03:06:25.128305 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:25.128277    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-8plk2"]
Nov 01 03:06:25.132365 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:06:25.132328    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod581612ca_5a0b_4426_9d07_ee65f85e0412.slice/crio-a77fcbed07b128d61f26fdab0b3b9290df54cd8244c91c2a2c52d9c5cd27331c WatchSource:0}: Error finding container a77fcbed07b128d61f26fdab0b3b9290df54cd8244c91c2a2c52d9c5cd27331c: Status 404 returned error can't find the container with id a77fcbed07b128d61f26fdab0b3b9290df54cd8244c91c2a2c52d9c5cd27331c
Nov 01 03:06:25.133944 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:25.133528    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fecfe177-e63f-490f-94bc-c307041640d5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xbdmd\" (UID: \"fecfe177-e63f-490f-94bc-c307041640d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xbdmd"
Nov 01 03:06:25.133944 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:25.133633    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7dbf71ad-8399-4a40-87d7-98746e830fa9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-t6tw7\" (UID: \"7dbf71ad-8399-4a40-87d7-98746e830fa9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-t6tw7"
Nov 01 03:06:25.133944 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:25.133766    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 03:06:25.133944 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:25.133825    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dbf71ad-8399-4a40-87d7-98746e830fa9-networking-console-plugin-cert podName:7dbf71ad-8399-4a40-87d7-98746e830fa9 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:26.133805421 +0000 UTC m=+32.245057722 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7dbf71ad-8399-4a40-87d7-98746e830fa9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-t6tw7" (UID: "7dbf71ad-8399-4a40-87d7-98746e830fa9") : secret "networking-console-plugin-cert" not found
Nov 01 03:06:25.133944 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:25.133851    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 03:06:25.133944 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:25.133906    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fecfe177-e63f-490f-94bc-c307041640d5-cluster-monitoring-operator-tls podName:fecfe177-e63f-490f-94bc-c307041640d5 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:26.133885829 +0000 UTC m=+32.245138132 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fecfe177-e63f-490f-94bc-c307041640d5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xbdmd" (UID: "fecfe177-e63f-490f-94bc-c307041640d5") : secret "cluster-monitoring-operator-tls" not found
Nov 01 03:06:25.166536 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:25.166470    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-48ssl"]
Nov 01 03:06:25.177698 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:06:25.177650    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeee1338d_e2a0_4a45_8f74_b81eb7d23019.slice/crio-7ff97e9cf55bf6cc1b4f494df68854b3713a287490b20ae897e75f2a656355c9 WatchSource:0}: Error finding container 7ff97e9cf55bf6cc1b4f494df68854b3713a287490b20ae897e75f2a656355c9: Status 404 returned error can't find the container with id 7ff97e9cf55bf6cc1b4f494df68854b3713a287490b20ae897e75f2a656355c9
Nov 01 03:06:25.234452 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:25.234196    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bbdccd97-7806-4669-a9cb-6aef70599c54-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gtk6f\" (UID: \"bbdccd97-7806-4669-a9cb-6aef70599c54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gtk6f"
Nov 01 03:06:25.234452 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:25.234280    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-tls\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:25.234452 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:25.234329    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-tls\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:25.234452 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:25.234410    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-metrics-certs\") pod \"router-default-ddbff58bc-5dlt4\" (UID: \"be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88\") " pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:25.234948 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:25.234461    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 03:06:25.234948 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:25.234521    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbdccd97-7806-4669-a9cb-6aef70599c54-samples-operator-tls podName:bbdccd97-7806-4669-a9cb-6aef70599c54 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:26.234501273 +0000 UTC m=+32.345753578 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bbdccd97-7806-4669-a9cb-6aef70599c54-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-gtk6f" (UID: "bbdccd97-7806-4669-a9cb-6aef70599c54") : secret "samples-operator-tls" not found
Nov 01 03:06:25.234948 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:25.234541    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-service-ca-bundle\") pod \"router-default-ddbff58bc-5dlt4\" (UID: \"be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88\") " pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:25.234948 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:25.234620    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 03:06:25.234948 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:25.234679    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-metrics-certs podName:be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:26.234661532 +0000 UTC m=+32.345913818 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-metrics-certs") pod "router-default-ddbff58bc-5dlt4" (UID: "be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88") : secret "router-metrics-certs-default" not found
Nov 01 03:06:25.234948 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:25.234696    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-service-ca-bundle podName:be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:26.234687687 +0000 UTC m=+32.345939980 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-service-ca-bundle") pod "router-default-ddbff58bc-5dlt4" (UID: "be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88") : configmap references non-existent config key: service-ca.crt
Nov 01 03:06:25.234948 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:25.234732    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 03:06:25.234948 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:25.234746    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f44999c56-64k9p: secret "image-registry-tls" not found
Nov 01 03:06:25.234948 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:25.234760    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 03:06:25.234948 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:25.234772    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58bc694f-l7vm9: secret "image-registry-tls" not found
Nov 01 03:06:25.234948 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:25.234773    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-tls podName:9acc0734-3562-41fc-a05a-8c32cbf36443 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:26.234764712 +0000 UTC m=+32.346017038 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-tls") pod "image-registry-6f44999c56-64k9p" (UID: "9acc0734-3562-41fc-a05a-8c32cbf36443") : secret "image-registry-tls" not found
Nov 01 03:06:25.234948 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:25.234814    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-tls podName:d28ff750-94d5-4eff-a8ef-58215bfb99e3 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:26.234804348 +0000 UTC m=+32.346056632 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-tls") pod "image-registry-58bc694f-l7vm9" (UID: "d28ff750-94d5-4eff-a8ef-58215bfb99e3") : secret "image-registry-tls" not found
Nov 01 03:06:25.270004 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:25.269976    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-cbzc6"]
Nov 01 03:06:25.276135 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:06:25.275266    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45c60c0c_dcff_421d_a9f2_4dc53e7637c9.slice/crio-3697c9672a50903bf381fe097bc80913031101f10a9832dd057ba5be41978ee4 WatchSource:0}: Error finding container 3697c9672a50903bf381fe097bc80913031101f10a9832dd057ba5be41978ee4: Status 404 returned error can't find the container with id 3697c9672a50903bf381fe097bc80913031101f10a9832dd057ba5be41978ee4
Nov 01 03:06:25.301533 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:25.301509    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chqll"]
Nov 01 03:06:25.304020 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:06:25.303998    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14e2ac70_be2e_4e08_b093_e8a93d6f985a.slice/crio-6ae3ed4f2fb47fac082d2e201eb9d84583e7293300d9cf621e865522115ad012 WatchSource:0}: Error finding container 6ae3ed4f2fb47fac082d2e201eb9d84583e7293300d9cf621e865522115ad012: Status 404 returned error can't find the container with id 6ae3ed4f2fb47fac082d2e201eb9d84583e7293300d9cf621e865522115ad012
Nov 01 03:06:25.335927 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:25.335901    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19871ddf-e177-4442-b592-8f1c7df24aaf-metrics-tls\") pod \"dns-default-wps29\" (UID: \"19871ddf-e177-4442-b592-8f1c7df24aaf\") " pod="openshift-dns/dns-default-wps29"
Nov 01 03:06:25.336060 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:25.336043    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 03:06:25.336105 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:25.336096    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19871ddf-e177-4442-b592-8f1c7df24aaf-metrics-tls podName:19871ddf-e177-4442-b592-8f1c7df24aaf nodeName:}" failed. No retries permitted until 2026-11-01 03:06:26.336081575 +0000 UTC m=+32.447333860 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19871ddf-e177-4442-b592-8f1c7df24aaf-metrics-tls") pod "dns-default-wps29" (UID: "19871ddf-e177-4442-b592-8f1c7df24aaf") : secret "dns-default-metrics-tls" not found
Nov 01 03:06:25.355121 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:25.355100    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-fsr8j"]
Nov 01 03:06:25.357678 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:06:25.357659    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ab1baec_cb71_4942_a246_7920cd44526c.slice/crio-cae338536d142cad1185c0ffac4012f88a6180d3352778fbaea998cf592a298e WatchSource:0}: Error finding container cae338536d142cad1185c0ffac4012f88a6180d3352778fbaea998cf592a298e: Status 404 returned error can't find the container with id cae338536d142cad1185c0ffac4012f88a6180d3352778fbaea998cf592a298e
Nov 01 03:06:25.377847 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:25.377742    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zdrpv"]
Nov 01 03:06:25.379524 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:06:25.379503    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda22c3f93_43f6_4908_9549_b966fb3c04e5.slice/crio-63ac8c7636952aa5d2fb015f80cd4690f2cf189a60bb36eb8136f47cbf331941 WatchSource:0}: Error finding container 63ac8c7636952aa5d2fb015f80cd4690f2cf189a60bb36eb8136f47cbf331941: Status 404 returned error can't find the container with id 63ac8c7636952aa5d2fb015f80cd4690f2cf189a60bb36eb8136f47cbf331941
Nov 01 03:06:25.422289 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:25.422243    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:25.540284 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:25.540215    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-jh955"]
Nov 01 03:06:25.542715 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:06:25.542690    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29ea61e_2a78_4c11_9566_f52497e135ff.slice/crio-18b2e76dfc01da8daf9dbf0daef2c0616a2343e95f4faea38329ce5563c0fbe4 WatchSource:0}: Error finding container 18b2e76dfc01da8daf9dbf0daef2c0616a2343e95f4faea38329ce5563c0fbe4: Status 404 returned error can't find the container with id 18b2e76dfc01da8daf9dbf0daef2c0616a2343e95f4faea38329ce5563c0fbe4
Nov 01 03:06:25.635227 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:25.635199    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fsr8j" event={"ID":"2ab1baec-cb71-4942-a246-7920cd44526c","Type":"ContainerStarted","Data":"cae338536d142cad1185c0ffac4012f88a6180d3352778fbaea998cf592a298e"}
Nov 01 03:06:25.636339 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:25.636318    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-62ws6" event={"ID":"bcbdd4ff-e252-4aee-9418-9b1b05eed574","Type":"ContainerStarted","Data":"f37006ff7bdb9b2342fb996deec818e8a3e5f535a2cbb2a0fbcd4cd18fd1771c"}
Nov 01 03:06:25.637353 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:25.637332    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbzc6" event={"ID":"45c60c0c-dcff-421d-a9f2-4dc53e7637c9","Type":"ContainerStarted","Data":"3697c9672a50903bf381fe097bc80913031101f10a9832dd057ba5be41978ee4"}
Nov 01 03:06:25.638544 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:25.638498    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chqll" event={"ID":"14e2ac70-be2e-4e08-b093-e8a93d6f985a","Type":"ContainerStarted","Data":"6ae3ed4f2fb47fac082d2e201eb9d84583e7293300d9cf621e865522115ad012"}
Nov 01 03:06:25.639898 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:25.639869    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-48ssl" event={"ID":"eee1338d-e2a0-4a45-8f74-b81eb7d23019","Type":"ContainerStarted","Data":"7ff97e9cf55bf6cc1b4f494df68854b3713a287490b20ae897e75f2a656355c9"}
Nov 01 03:06:25.641589 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:25.641568    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-fznsl" event={"ID":"e60132c0-948e-4f84-883e-d22564f210e1","Type":"ContainerStarted","Data":"08925acb2f924a81001c7f4b362ce6be84e7a9650992bd9d100bf1840c1cf097"}
Nov 01 03:06:25.641668 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:25.641607    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-fznsl" event={"ID":"e60132c0-948e-4f84-883e-d22564f210e1","Type":"ContainerStarted","Data":"4bbef179152e161a50c2a59d9194272f6473ebc2b3c2ece917e5743eb50a63f5"}
Nov 01 03:06:25.642922 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:25.642901    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-j6kdp" event={"ID":"ffdeddae-4086-4fb5-b9f8-c8ffcb765925","Type":"ContainerStarted","Data":"3c48bb218c0eb501f33b0ac8ff9f72562fcd2878bca986a0a8d82bfe2b56185d"}
Nov 01 03:06:25.643014 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:25.642938    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-j6kdp" event={"ID":"ffdeddae-4086-4fb5-b9f8-c8ffcb765925","Type":"ContainerStarted","Data":"7a916b464e271033e5650cc96ca39397503853853c9742d5600a5f956a7d7d47"}
Nov 01 03:06:25.644355 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:25.644332    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-8plk2" event={"ID":"581612ca-5a0b-4426-9d07-ee65f85e0412","Type":"ContainerStarted","Data":"16a359668136f785d687c415f79aca7d052ef4b2ce0e1cb45410032ac8aef8f8"}
Nov 01 03:06:25.644447 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:25.644372    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-8plk2" event={"ID":"581612ca-5a0b-4426-9d07-ee65f85e0412","Type":"ContainerStarted","Data":"a77fcbed07b128d61f26fdab0b3b9290df54cd8244c91c2a2c52d9c5cd27331c"}
Nov 01 03:06:25.645519 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:25.645493    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jh955" event={"ID":"f29ea61e-2a78-4c11-9566-f52497e135ff","Type":"ContainerStarted","Data":"18b2e76dfc01da8daf9dbf0daef2c0616a2343e95f4faea38329ce5563c0fbe4"}
Nov 01 03:06:25.646634 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:25.646615    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zdrpv" event={"ID":"a22c3f93-43f6-4908-9549-b966fb3c04e5","Type":"ContainerStarted","Data":"63ac8c7636952aa5d2fb015f80cd4690f2cf189a60bb36eb8136f47cbf331941"}
Nov 01 03:06:25.653064 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:25.653046    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:26.143146 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:26.143115    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fecfe177-e63f-490f-94bc-c307041640d5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xbdmd\" (UID: \"fecfe177-e63f-490f-94bc-c307041640d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xbdmd"
Nov 01 03:06:26.143581 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:26.143186    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7dbf71ad-8399-4a40-87d7-98746e830fa9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-t6tw7\" (UID: \"7dbf71ad-8399-4a40-87d7-98746e830fa9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-t6tw7"
Nov 01 03:06:26.143581 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:26.143265    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 03:06:26.143581 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:26.143322    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fecfe177-e63f-490f-94bc-c307041640d5-cluster-monitoring-operator-tls podName:fecfe177-e63f-490f-94bc-c307041640d5 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:28.143306623 +0000 UTC m=+34.254558922 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fecfe177-e63f-490f-94bc-c307041640d5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xbdmd" (UID: "fecfe177-e63f-490f-94bc-c307041640d5") : secret "cluster-monitoring-operator-tls" not found
Nov 01 03:06:26.143581 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:26.143337    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 03:06:26.143581 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:26.143417    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dbf71ad-8399-4a40-87d7-98746e830fa9-networking-console-plugin-cert podName:7dbf71ad-8399-4a40-87d7-98746e830fa9 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:28.143397394 +0000 UTC m=+34.254649696 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7dbf71ad-8399-4a40-87d7-98746e830fa9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-t6tw7" (UID: "7dbf71ad-8399-4a40-87d7-98746e830fa9") : secret "networking-console-plugin-cert" not found
Nov 01 03:06:26.243828 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:26.243787    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-tls\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:26.243995 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:26.243872    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-metrics-certs\") pod \"router-default-ddbff58bc-5dlt4\" (UID: \"be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88\") " pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:26.243995 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:26.243925    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-service-ca-bundle\") pod \"router-default-ddbff58bc-5dlt4\" (UID: \"be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88\") " pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:26.243995 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:26.243940    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 03:06:26.243995 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:26.243962    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58bc694f-l7vm9: secret "image-registry-tls" not found
Nov 01 03:06:26.243995 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:26.243981    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bbdccd97-7806-4669-a9cb-6aef70599c54-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gtk6f\" (UID: \"bbdccd97-7806-4669-a9cb-6aef70599c54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gtk6f"
Nov 01 03:06:26.243995 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:26.243994    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 03:06:26.244291 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:26.244015    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-tls podName:d28ff750-94d5-4eff-a8ef-58215bfb99e3 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:28.243996205 +0000 UTC m=+34.355248492 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-tls") pod "image-registry-58bc694f-l7vm9" (UID: "d28ff750-94d5-4eff-a8ef-58215bfb99e3") : secret "image-registry-tls" not found
Nov 01 03:06:26.244291 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:26.244036    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-metrics-certs podName:be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:28.244027275 +0000 UTC m=+34.355279562 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-metrics-certs") pod "router-default-ddbff58bc-5dlt4" (UID: "be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88") : secret "router-metrics-certs-default" not found
Nov 01 03:06:26.244291 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:26.244065    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 03:06:26.244291 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:26.244090    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-tls\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:26.244291 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:26.244117    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbdccd97-7806-4669-a9cb-6aef70599c54-samples-operator-tls podName:bbdccd97-7806-4669-a9cb-6aef70599c54 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:28.244102737 +0000 UTC m=+34.355355024 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bbdccd97-7806-4669-a9cb-6aef70599c54-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-gtk6f" (UID: "bbdccd97-7806-4669-a9cb-6aef70599c54") : secret "samples-operator-tls" not found
Nov 01 03:06:26.244291 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:26.244138    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-service-ca-bundle podName:be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:28.244129263 +0000 UTC m=+34.355381552 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-service-ca-bundle") pod "router-default-ddbff58bc-5dlt4" (UID: "be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88") : configmap references non-existent config key: service-ca.crt
Nov 01 03:06:26.244291 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:26.244171    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 03:06:26.244291 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:26.244186    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f44999c56-64k9p: secret "image-registry-tls" not found
Nov 01 03:06:26.244291 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:26.244236    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-tls podName:9acc0734-3562-41fc-a05a-8c32cbf36443 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:28.244224523 +0000 UTC m=+34.355476829 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-tls") pod "image-registry-6f44999c56-64k9p" (UID: "9acc0734-3562-41fc-a05a-8c32cbf36443") : secret "image-registry-tls" not found
Nov 01 03:06:26.345094 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:26.345057    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19871ddf-e177-4442-b592-8f1c7df24aaf-metrics-tls\") pod \"dns-default-wps29\" (UID: \"19871ddf-e177-4442-b592-8f1c7df24aaf\") " pod="openshift-dns/dns-default-wps29"
Nov 01 03:06:26.345377 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:26.345361    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 03:06:26.345456 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:26.345415    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19871ddf-e177-4442-b592-8f1c7df24aaf-metrics-tls podName:19871ddf-e177-4442-b592-8f1c7df24aaf nodeName:}" failed. No retries permitted until 2026-11-01 03:06:28.345395621 +0000 UTC m=+34.456647905 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19871ddf-e177-4442-b592-8f1c7df24aaf-metrics-tls") pod "dns-default-wps29" (UID: "19871ddf-e177-4442-b592-8f1c7df24aaf") : secret "dns-default-metrics-tls" not found
Nov 01 03:06:26.547846 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:26.547755    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-8plk2" podStartSLOduration=391.547736677 podStartE2EDuration="6m31.547736677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 02:59:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:06:26.546545948 +0000 UTC m=+32.657798256" watchObservedRunningTime="2026-11-01 03:06:26.547736677 +0000 UTC m=+32.658988987"
Nov 01 03:06:26.680875 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:26.680826    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-fznsl" podStartSLOduration=391.680812202 podStartE2EDuration="6m31.680812202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 02:59:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:06:26.679801391 +0000 UTC m=+32.791053699" watchObservedRunningTime="2026-11-01 03:06:26.680812202 +0000 UTC m=+32.792064507"
Nov 01 03:06:26.800646 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:26.800530    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-j6kdp" podStartSLOduration=391.80051136 podStartE2EDuration="6m31.80051136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 02:59:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:06:26.799839651 +0000 UTC m=+32.911091957" watchObservedRunningTime="2026-11-01 03:06:26.80051136 +0000 UTC m=+32.911763667"
Nov 01 03:06:27.356060 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:27.356022    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a66c33f-29bc-4920-9d7a-0c45215514a0-metrics-certs\") pod \"network-metrics-daemon-mzkbj\" (UID: \"0a66c33f-29bc-4920-9d7a-0c45215514a0\") " pod="openshift-multus/network-metrics-daemon-mzkbj"
Nov 01 03:06:27.356531 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:27.356144    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a972ee28-29fa-4755-adcd-b40d476c22f0-original-pull-secret\") pod \"global-pull-secret-syncer-4bvb6\" (UID: \"a972ee28-29fa-4755-adcd-b40d476c22f0\") " pod="kube-system/global-pull-secret-syncer-4bvb6"
Nov 01 03:06:27.356835 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:27.356812    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 03:06:27.356927 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:27.356900    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a66c33f-29bc-4920-9d7a-0c45215514a0-metrics-certs podName:0a66c33f-29bc-4920-9d7a-0c45215514a0 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:59.356871973 +0000 UTC m=+65.468124277 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a66c33f-29bc-4920-9d7a-0c45215514a0-metrics-certs") pod "network-metrics-daemon-mzkbj" (UID: "0a66c33f-29bc-4920-9d7a-0c45215514a0") : secret "metrics-daemon-secret" not found
Nov 01 03:06:27.364852 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:27.364824    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a972ee28-29fa-4755-adcd-b40d476c22f0-original-pull-secret\") pod \"global-pull-secret-syncer-4bvb6\" (UID: \"a972ee28-29fa-4755-adcd-b40d476c22f0\") " pod="kube-system/global-pull-secret-syncer-4bvb6"
Nov 01 03:06:27.458224 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:27.457614    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lbzv\" (UniqueName: \"kubernetes.io/projected/e09a58a7-120e-4ec5-9aba-f15863267643-kube-api-access-7lbzv\") pod \"network-check-target-62mxc\" (UID: \"e09a58a7-120e-4ec5-9aba-f15863267643\") " pod="openshift-network-diagnostics/network-check-target-62mxc"
Nov 01 03:06:27.462474 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:27.462423    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lbzv\" (UniqueName: \"kubernetes.io/projected/e09a58a7-120e-4ec5-9aba-f15863267643-kube-api-access-7lbzv\") pod \"network-check-target-62mxc\" (UID: \"e09a58a7-120e-4ec5-9aba-f15863267643\") " pod="openshift-network-diagnostics/network-check-target-62mxc"
Nov 01 03:06:27.492746 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:27.492487    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-62mxc"
Nov 01 03:06:27.610973 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:27.610735    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4bvb6"
Nov 01 03:06:28.083223 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:28.083153    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4bvb6"]
Nov 01 03:06:28.133507 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:28.133091    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-62mxc"]
Nov 01 03:06:28.138692 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:06:28.138665    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode09a58a7_120e_4ec5_9aba_f15863267643.slice/crio-12af2981349a6df9f1a2182daa931d656f31626994e1b49fb30a715a93b95b48 WatchSource:0}: Error finding container 12af2981349a6df9f1a2182daa931d656f31626994e1b49fb30a715a93b95b48: Status 404 returned error can't find the container with id 12af2981349a6df9f1a2182daa931d656f31626994e1b49fb30a715a93b95b48
Nov 01 03:06:28.165192 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:28.165149    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7dbf71ad-8399-4a40-87d7-98746e830fa9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-t6tw7\" (UID: \"7dbf71ad-8399-4a40-87d7-98746e830fa9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-t6tw7"
Nov 01 03:06:28.165365 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:28.165343    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fecfe177-e63f-490f-94bc-c307041640d5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xbdmd\" (UID: \"fecfe177-e63f-490f-94bc-c307041640d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xbdmd"
Nov 01 03:06:28.165532 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:28.165513    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 03:06:28.165630 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:28.165595    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fecfe177-e63f-490f-94bc-c307041640d5-cluster-monitoring-operator-tls podName:fecfe177-e63f-490f-94bc-c307041640d5 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:32.165575957 +0000 UTC m=+38.276828254 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fecfe177-e63f-490f-94bc-c307041640d5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xbdmd" (UID: "fecfe177-e63f-490f-94bc-c307041640d5") : secret "cluster-monitoring-operator-tls" not found
Nov 01 03:06:28.165708 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:28.165664    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 03:06:28.165708 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:28.165697    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dbf71ad-8399-4a40-87d7-98746e830fa9-networking-console-plugin-cert podName:7dbf71ad-8399-4a40-87d7-98746e830fa9 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:32.165686708 +0000 UTC m=+38.276939006 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7dbf71ad-8399-4a40-87d7-98746e830fa9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-t6tw7" (UID: "7dbf71ad-8399-4a40-87d7-98746e830fa9") : secret "networking-console-plugin-cert" not found
Nov 01 03:06:28.266217 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:28.266185    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bbdccd97-7806-4669-a9cb-6aef70599c54-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gtk6f\" (UID: \"bbdccd97-7806-4669-a9cb-6aef70599c54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gtk6f"
Nov 01 03:06:28.266369 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:28.266257    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-tls\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:28.266369 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:28.266298    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-tls\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:28.266369 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:28.266337    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-metrics-certs\") pod \"router-default-ddbff58bc-5dlt4\" (UID: \"be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88\") " pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:28.266710 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:28.266394    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-service-ca-bundle\") pod \"router-default-ddbff58bc-5dlt4\" (UID: \"be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88\") " pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:28.266710 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:28.266593    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-service-ca-bundle podName:be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:32.26657391 +0000 UTC m=+38.377826197 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-service-ca-bundle") pod "router-default-ddbff58bc-5dlt4" (UID: "be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88") : configmap references non-existent config key: service-ca.crt
Nov 01 03:06:28.266710 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:28.266689    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 03:06:28.266864 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:28.266729    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbdccd97-7806-4669-a9cb-6aef70599c54-samples-operator-tls podName:bbdccd97-7806-4669-a9cb-6aef70599c54 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:32.266717339 +0000 UTC m=+38.377969629 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bbdccd97-7806-4669-a9cb-6aef70599c54-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-gtk6f" (UID: "bbdccd97-7806-4669-a9cb-6aef70599c54") : secret "samples-operator-tls" not found
Nov 01 03:06:28.266864 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:28.266799    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 03:06:28.266864 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:28.266813    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f44999c56-64k9p: secret "image-registry-tls" not found
Nov 01 03:06:28.266864 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:28.266848    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-tls podName:9acc0734-3562-41fc-a05a-8c32cbf36443 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:32.266836659 +0000 UTC m=+38.378088961 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-tls") pod "image-registry-6f44999c56-64k9p" (UID: "9acc0734-3562-41fc-a05a-8c32cbf36443") : secret "image-registry-tls" not found
Nov 01 03:06:28.267051 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:28.266913    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 03:06:28.267051 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:28.266923    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58bc694f-l7vm9: secret "image-registry-tls" not found
Nov 01 03:06:28.267051 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:28.266954    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-tls podName:d28ff750-94d5-4eff-a8ef-58215bfb99e3 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:32.266943486 +0000 UTC m=+38.378195775 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-tls") pod "image-registry-58bc694f-l7vm9" (UID: "d28ff750-94d5-4eff-a8ef-58215bfb99e3") : secret "image-registry-tls" not found
Nov 01 03:06:28.267051 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:28.267013    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 03:06:28.267051 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:28.267045    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-metrics-certs podName:be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:32.267035131 +0000 UTC m=+38.378287429 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-metrics-certs") pod "router-default-ddbff58bc-5dlt4" (UID: "be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88") : secret "router-metrics-certs-default" not found
Nov 01 03:06:28.367303 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:28.367271    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19871ddf-e177-4442-b592-8f1c7df24aaf-metrics-tls\") pod \"dns-default-wps29\" (UID: \"19871ddf-e177-4442-b592-8f1c7df24aaf\") " pod="openshift-dns/dns-default-wps29"
Nov 01 03:06:28.367718 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:28.367436    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 03:06:28.367718 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:28.367491    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19871ddf-e177-4442-b592-8f1c7df24aaf-metrics-tls podName:19871ddf-e177-4442-b592-8f1c7df24aaf nodeName:}" failed. No retries permitted until 2026-11-01 03:06:32.367475625 +0000 UTC m=+38.478727914 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19871ddf-e177-4442-b592-8f1c7df24aaf-metrics-tls") pod "dns-default-wps29" (UID: "19871ddf-e177-4442-b592-8f1c7df24aaf") : secret "dns-default-metrics-tls" not found
Nov 01 03:06:28.667024 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:28.666943    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-62ws6" event={"ID":"bcbdd4ff-e252-4aee-9418-9b1b05eed574","Type":"ContainerStarted","Data":"a7cdf7da945309b844f49684707c4b3deff6829022a11bf5f4bcbf386dc89528"}
Nov 01 03:06:28.670410 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:28.670382    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4bvb6" event={"ID":"a972ee28-29fa-4755-adcd-b40d476c22f0","Type":"ContainerStarted","Data":"0b7cef567f173ef3b6d0baafd5015f8074a46e4fb248df070231b3edd177ed2f"}
Nov 01 03:06:28.671844 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:28.671820    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-62mxc" event={"ID":"e09a58a7-120e-4ec5-9aba-f15863267643","Type":"ContainerStarted","Data":"12af2981349a6df9f1a2182daa931d656f31626994e1b49fb30a715a93b95b48"}
Nov 01 03:06:28.784580 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:28.784352    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-62ws6" podStartSLOduration=347.187430031 podStartE2EDuration="5m49.783463097s" totalImagesPullingTime="2.596033066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:00:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:06:28.782767385 +0000 UTC m=+34.894019691" watchObservedRunningTime="2026-11-01 03:06:28.783463097 +0000 UTC m=+34.894715407"
Nov 01 03:06:28.907622 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:28.907591    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:29.058336 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:29.058258    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:30.692571 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:30.692504    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-48ssl" event={"ID":"eee1338d-e2a0-4a45-8f74-b81eb7d23019","Type":"ContainerStarted","Data":"2aded05819eae7519ba2246448f4f7ae3bc7689c18225f58266e75d32c3fa6e0"}
Nov 01 03:06:31.302069 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:31.301977    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-48ssl" podStartSLOduration=345.817950395 podStartE2EDuration="5m50.301961411s" totalImagesPullingTime="4.484011016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:00:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:06:30.814907648 +0000 UTC m=+36.926159954" watchObservedRunningTime="2026-11-01 03:06:31.301961411 +0000 UTC m=+37.413213711"
Nov 01 03:06:31.302222 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:31.302114    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2vfgr"]
Nov 01 03:06:31.318427 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:31.318382    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2vfgr"
Nov 01 03:06:31.368252 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:31.368215    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2vfgr"]
Nov 01 03:06:31.370952 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:31.370930    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 03:06:31.397760 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:31.397736    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-crio-socket\") pod \"insights-runtime-extractor-2vfgr\" (UID: \"caebbba6-44f3-40c1-95ce-b8a5a3f2667b\") " pod="openshift-insights/insights-runtime-extractor-2vfgr"
Nov 01 03:06:31.397877 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:31.397816    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2vfgr\" (UID: \"caebbba6-44f3-40c1-95ce-b8a5a3f2667b\") " pod="openshift-insights/insights-runtime-extractor-2vfgr"
Nov 01 03:06:31.397959 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:31.397926    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2vfgr\" (UID: \"caebbba6-44f3-40c1-95ce-b8a5a3f2667b\") " pod="openshift-insights/insights-runtime-extractor-2vfgr"
Nov 01 03:06:31.398095 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:31.398080    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlvp2\" (UniqueName: \"kubernetes.io/projected/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-kube-api-access-wlvp2\") pod \"insights-runtime-extractor-2vfgr\" (UID: \"caebbba6-44f3-40c1-95ce-b8a5a3f2667b\") " pod="openshift-insights/insights-runtime-extractor-2vfgr"
Nov 01 03:06:31.398162 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:31.398128    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-data-volume\") pod \"insights-runtime-extractor-2vfgr\" (UID: \"caebbba6-44f3-40c1-95ce-b8a5a3f2667b\") " pod="openshift-insights/insights-runtime-extractor-2vfgr"
Nov 01 03:06:31.467491 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:31.467448    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 03:06:31.467660 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:31.467599    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dg4zg\""
Nov 01 03:06:31.499116 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:31.499081    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-data-volume\") pod \"insights-runtime-extractor-2vfgr\" (UID: \"caebbba6-44f3-40c1-95ce-b8a5a3f2667b\") " pod="openshift-insights/insights-runtime-extractor-2vfgr"
Nov 01 03:06:31.499262 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:31.499123    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-crio-socket\") pod \"insights-runtime-extractor-2vfgr\" (UID: \"caebbba6-44f3-40c1-95ce-b8a5a3f2667b\") " pod="openshift-insights/insights-runtime-extractor-2vfgr"
Nov 01 03:06:31.499262 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:31.499198    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2vfgr\" (UID: \"caebbba6-44f3-40c1-95ce-b8a5a3f2667b\") " pod="openshift-insights/insights-runtime-extractor-2vfgr"
Nov 01 03:06:31.499262 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:31.499253    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2vfgr\" (UID: \"caebbba6-44f3-40c1-95ce-b8a5a3f2667b\") " pod="openshift-insights/insights-runtime-extractor-2vfgr"
Nov 01 03:06:31.499452 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:31.499376    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlvp2\" (UniqueName: \"kubernetes.io/projected/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-kube-api-access-wlvp2\") pod \"insights-runtime-extractor-2vfgr\" (UID: \"caebbba6-44f3-40c1-95ce-b8a5a3f2667b\") " pod="openshift-insights/insights-runtime-extractor-2vfgr"
Nov 01 03:06:31.500003 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:31.499978    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-data-volume\") pod \"insights-runtime-extractor-2vfgr\" (UID: \"caebbba6-44f3-40c1-95ce-b8a5a3f2667b\") " pod="openshift-insights/insights-runtime-extractor-2vfgr"
Nov 01 03:06:31.500201 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:31.500181    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-crio-socket\") pod \"insights-runtime-extractor-2vfgr\" (UID: \"caebbba6-44f3-40c1-95ce-b8a5a3f2667b\") " pod="openshift-insights/insights-runtime-extractor-2vfgr"
Nov 01 03:06:31.500299 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:31.500288    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 03:06:31.500361 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:31.500349    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-insights-runtime-extractor-tls podName:caebbba6-44f3-40c1-95ce-b8a5a3f2667b nodeName:}" failed. No retries permitted until 2026-11-01 03:06:32.000329189 +0000 UTC m=+38.111581478 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2vfgr" (UID: "caebbba6-44f3-40c1-95ce-b8a5a3f2667b") : secret "insights-runtime-extractor-tls" not found
Nov 01 03:06:31.500608 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:31.500591    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 03:06:31.500662 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:31.500643    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-kube-rbac-proxy-cm podName:caebbba6-44f3-40c1-95ce-b8a5a3f2667b nodeName:}" failed. No retries permitted until 2026-11-01 03:06:32.000628542 +0000 UTC m=+38.111880828 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2vfgr" (UID: "caebbba6-44f3-40c1-95ce-b8a5a3f2667b") : configmap "kube-rbac-proxy" not found
Nov 01 03:06:31.536896 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:31.536867    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:31.562918 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:31.562862    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlvp2\" (UniqueName: \"kubernetes.io/projected/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-kube-api-access-wlvp2\") pod \"insights-runtime-extractor-2vfgr\" (UID: \"caebbba6-44f3-40c1-95ce-b8a5a3f2667b\") " pod="openshift-insights/insights-runtime-extractor-2vfgr"
Nov 01 03:06:31.697152 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:31.697130    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jh955_f29ea61e-2a78-4c11-9566-f52497e135ff/console-operator/0.log"
Nov 01 03:06:31.697541 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:31.697170    2578 generic.go:356] "Generic (PLEG): container finished" podID="f29ea61e-2a78-4c11-9566-f52497e135ff" containerID="7f9a8382d9252f54f838d79cc0027d36b865b97b31dc521e10bec53313737ed5" exitCode=255
Nov 01 03:06:31.697541 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:31.697260    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jh955" event={"ID":"f29ea61e-2a78-4c11-9566-f52497e135ff","Type":"ContainerDied","Data":"7f9a8382d9252f54f838d79cc0027d36b865b97b31dc521e10bec53313737ed5"}
Nov 01 03:06:31.697541 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:31.697500    2578 scope.go:122] "RemoveContainer" containerID="7f9a8382d9252f54f838d79cc0027d36b865b97b31dc521e10bec53313737ed5"
Nov 01 03:06:31.738505 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:31.738480    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:32.003551 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:32.003510    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2vfgr\" (UID: \"caebbba6-44f3-40c1-95ce-b8a5a3f2667b\") " pod="openshift-insights/insights-runtime-extractor-2vfgr"
Nov 01 03:06:32.003737 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:32.003594    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2vfgr\" (UID: \"caebbba6-44f3-40c1-95ce-b8a5a3f2667b\") " pod="openshift-insights/insights-runtime-extractor-2vfgr"
Nov 01 03:06:32.003794 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:32.003767    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 03:06:32.003847 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:32.003829    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-kube-rbac-proxy-cm podName:caebbba6-44f3-40c1-95ce-b8a5a3f2667b nodeName:}" failed. No retries permitted until 2026-11-01 03:06:33.003809629 +0000 UTC m=+39.115061928 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2vfgr" (UID: "caebbba6-44f3-40c1-95ce-b8a5a3f2667b") : configmap "kube-rbac-proxy" not found
Nov 01 03:06:32.003918 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:32.003910    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 03:06:32.003968 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:32.003940    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-insights-runtime-extractor-tls podName:caebbba6-44f3-40c1-95ce-b8a5a3f2667b nodeName:}" failed. No retries permitted until 2026-11-01 03:06:33.00392999 +0000 UTC m=+39.115182277 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2vfgr" (UID: "caebbba6-44f3-40c1-95ce-b8a5a3f2667b") : secret "insights-runtime-extractor-tls" not found
Nov 01 03:06:32.206011 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:32.205977    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fecfe177-e63f-490f-94bc-c307041640d5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xbdmd\" (UID: \"fecfe177-e63f-490f-94bc-c307041640d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xbdmd"
Nov 01 03:06:32.206154 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:32.206062    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7dbf71ad-8399-4a40-87d7-98746e830fa9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-t6tw7\" (UID: \"7dbf71ad-8399-4a40-87d7-98746e830fa9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-t6tw7"
Nov 01 03:06:32.206154 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:32.206124    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 03:06:32.206265 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:32.206178    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 03:06:32.206265 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:32.206192    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fecfe177-e63f-490f-94bc-c307041640d5-cluster-monitoring-operator-tls podName:fecfe177-e63f-490f-94bc-c307041640d5 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:40.206172019 +0000 UTC m=+46.317424326 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fecfe177-e63f-490f-94bc-c307041640d5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xbdmd" (UID: "fecfe177-e63f-490f-94bc-c307041640d5") : secret "cluster-monitoring-operator-tls" not found
Nov 01 03:06:32.206265 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:32.206215    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dbf71ad-8399-4a40-87d7-98746e830fa9-networking-console-plugin-cert podName:7dbf71ad-8399-4a40-87d7-98746e830fa9 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:40.206204801 +0000 UTC m=+46.317457121 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7dbf71ad-8399-4a40-87d7-98746e830fa9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-t6tw7" (UID: "7dbf71ad-8399-4a40-87d7-98746e830fa9") : secret "networking-console-plugin-cert" not found
Nov 01 03:06:32.307899 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:32.307450    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-service-ca-bundle\") pod \"router-default-ddbff58bc-5dlt4\" (UID: \"be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88\") " pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:32.307899 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:32.307535    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bbdccd97-7806-4669-a9cb-6aef70599c54-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gtk6f\" (UID: \"bbdccd97-7806-4669-a9cb-6aef70599c54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gtk6f"
Nov 01 03:06:32.307899 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:32.307598    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-tls\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:32.307899 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:32.307627    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-service-ca-bundle podName:be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:40.307607387 +0000 UTC m=+46.418859691 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-service-ca-bundle") pod "router-default-ddbff58bc-5dlt4" (UID: "be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88") : configmap references non-existent config key: service-ca.crt
Nov 01 03:06:32.307899 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:32.307672    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-tls\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:32.307899 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:32.307752    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-metrics-certs\") pod \"router-default-ddbff58bc-5dlt4\" (UID: \"be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88\") " pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:32.307899 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:32.307677    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 03:06:32.307899 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:32.307801    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f44999c56-64k9p: secret "image-registry-tls" not found
Nov 01 03:06:32.307899 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:32.307839    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-tls podName:9acc0734-3562-41fc-a05a-8c32cbf36443 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:40.307826479 +0000 UTC m=+46.419078786 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-tls") pod "image-registry-6f44999c56-64k9p" (UID: "9acc0734-3562-41fc-a05a-8c32cbf36443") : secret "image-registry-tls" not found
Nov 01 03:06:32.307899 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:32.307731    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 03:06:32.307899 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:32.307851    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 03:06:32.307899 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:32.307786    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 03:06:32.307899 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:32.307884    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58bc694f-l7vm9: secret "image-registry-tls" not found
Nov 01 03:06:32.307899 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:32.307874    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbdccd97-7806-4669-a9cb-6aef70599c54-samples-operator-tls podName:bbdccd97-7806-4669-a9cb-6aef70599c54 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:40.307865879 +0000 UTC m=+46.419118181 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bbdccd97-7806-4669-a9cb-6aef70599c54-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-gtk6f" (UID: "bbdccd97-7806-4669-a9cb-6aef70599c54") : secret "samples-operator-tls" not found
Nov 01 03:06:32.307899 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:32.307916    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-metrics-certs podName:be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:40.30790702 +0000 UTC m=+46.419159310 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-metrics-certs") pod "router-default-ddbff58bc-5dlt4" (UID: "be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88") : secret "router-metrics-certs-default" not found
Nov 01 03:06:32.308624 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:32.307928    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-tls podName:d28ff750-94d5-4eff-a8ef-58215bfb99e3 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:40.307920945 +0000 UTC m=+46.419173242 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-tls") pod "image-registry-58bc694f-l7vm9" (UID: "d28ff750-94d5-4eff-a8ef-58215bfb99e3") : secret "image-registry-tls" not found
Nov 01 03:06:32.408431 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:32.408404    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19871ddf-e177-4442-b592-8f1c7df24aaf-metrics-tls\") pod \"dns-default-wps29\" (UID: \"19871ddf-e177-4442-b592-8f1c7df24aaf\") " pod="openshift-dns/dns-default-wps29"
Nov 01 03:06:32.408572 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:32.408539    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 03:06:32.408636 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:32.408623    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19871ddf-e177-4442-b592-8f1c7df24aaf-metrics-tls podName:19871ddf-e177-4442-b592-8f1c7df24aaf nodeName:}" failed. No retries permitted until 2026-11-01 03:06:40.408605342 +0000 UTC m=+46.519857626 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19871ddf-e177-4442-b592-8f1c7df24aaf-metrics-tls") pod "dns-default-wps29" (UID: "19871ddf-e177-4442-b592-8f1c7df24aaf") : secret "dns-default-metrics-tls" not found
Nov 01 03:06:32.704664 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:32.704589    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jh955_f29ea61e-2a78-4c11-9566-f52497e135ff/console-operator/1.log"
Nov 01 03:06:32.705192 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:32.705174    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jh955_f29ea61e-2a78-4c11-9566-f52497e135ff/console-operator/0.log"
Nov 01 03:06:32.705254 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:32.705204    2578 generic.go:356] "Generic (PLEG): container finished" podID="f29ea61e-2a78-4c11-9566-f52497e135ff" containerID="3d57ddeb233cc6ef9b56e40c0268f52a146af9adc2a471d9dd2158d7b677263c" exitCode=255
Nov 01 03:06:32.705310 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:32.705268    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jh955" event={"ID":"f29ea61e-2a78-4c11-9566-f52497e135ff","Type":"ContainerDied","Data":"3d57ddeb233cc6ef9b56e40c0268f52a146af9adc2a471d9dd2158d7b677263c"}
Nov 01 03:06:32.705362 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:32.705323    2578 scope.go:122] "RemoveContainer" containerID="7f9a8382d9252f54f838d79cc0027d36b865b97b31dc521e10bec53313737ed5"
Nov 01 03:06:32.705714 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:32.705698    2578 scope.go:122] "RemoveContainer" containerID="3d57ddeb233cc6ef9b56e40c0268f52a146af9adc2a471d9dd2158d7b677263c"
Nov 01 03:06:32.705913 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:32.705893    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jh955_openshift-console-operator(f29ea61e-2a78-4c11-9566-f52497e135ff)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jh955" podUID="f29ea61e-2a78-4c11-9566-f52497e135ff"
Nov 01 03:06:32.708168 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:32.708139    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zdrpv" event={"ID":"a22c3f93-43f6-4908-9549-b966fb3c04e5","Type":"ContainerStarted","Data":"038fe8389ebca4fbc22a2ea914c200b843fe5985893590d29a87d8e6ce4fc8bf"}
Nov 01 03:06:32.709825 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:32.709804    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fsr8j" event={"ID":"2ab1baec-cb71-4942-a246-7920cd44526c","Type":"ContainerStarted","Data":"7a94bd9226c8684775d47339ba0a6966fcee6355d9bb1c9e3d056ff1b1372865"}
Nov 01 03:06:32.712186 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:32.712158    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-62mxc" event={"ID":"e09a58a7-120e-4ec5-9aba-f15863267643","Type":"ContainerStarted","Data":"8573c111562e6ab414d98357c15c0de3e6eedaaf78bab0fdd12adc976abec07e"}
Nov 01 03:06:32.712298 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:32.712265    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-62mxc"
Nov 01 03:06:32.714152 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:32.714128    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbzc6" event={"ID":"45c60c0c-dcff-421d-a9f2-4dc53e7637c9","Type":"ContainerStarted","Data":"5d41367b4bede9a07ce8382f8a3e5be6d36791a840f9cac09cde85411eec36cd"}
Nov 01 03:06:32.715878 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:32.715852    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chqll" event={"ID":"14e2ac70-be2e-4e08-b093-e8a93d6f985a","Type":"ContainerStarted","Data":"59c7db500b6891ccf55e74cdb53331819b25a34a9739617c3dd553b787bcd588"}
Nov 01 03:06:32.950146 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:32.950096    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbzc6" podStartSLOduration=344.529588717 podStartE2EDuration="5m50.950082516s" totalImagesPullingTime="6.420493799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:00:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:06:32.948474429 +0000 UTC m=+39.059726732" watchObservedRunningTime="2026-11-01 03:06:32.950082516 +0000 UTC m=+39.061334821"
Nov 01 03:06:33.013516 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:33.013478    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2vfgr\" (UID: \"caebbba6-44f3-40c1-95ce-b8a5a3f2667b\") " pod="openshift-insights/insights-runtime-extractor-2vfgr"
Nov 01 03:06:33.013690 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:33.013528    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2vfgr\" (UID: \"caebbba6-44f3-40c1-95ce-b8a5a3f2667b\") " pod="openshift-insights/insights-runtime-extractor-2vfgr"
Nov 01 03:06:33.013690 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:33.013650    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 03:06:33.013690 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:33.013669    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 03:06:33.013839 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:33.013703    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-kube-rbac-proxy-cm podName:caebbba6-44f3-40c1-95ce-b8a5a3f2667b nodeName:}" failed. No retries permitted until 2026-11-01 03:06:35.013688731 +0000 UTC m=+41.124941032 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2vfgr" (UID: "caebbba6-44f3-40c1-95ce-b8a5a3f2667b") : configmap "kube-rbac-proxy" not found
Nov 01 03:06:33.013839 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:33.013731    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-insights-runtime-extractor-tls podName:caebbba6-44f3-40c1-95ce-b8a5a3f2667b nodeName:}" failed. No retries permitted until 2026-11-01 03:06:35.013713527 +0000 UTC m=+41.124965826 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2vfgr" (UID: "caebbba6-44f3-40c1-95ce-b8a5a3f2667b") : secret "insights-runtime-extractor-tls" not found
Nov 01 03:06:33.071106 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:33.069519    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fsr8j" podStartSLOduration=345.344756364 podStartE2EDuration="5m52.069503389s" totalImagesPullingTime="6.724747025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:00:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:06:33.068549288 +0000 UTC m=+39.179801597" watchObservedRunningTime="2026-11-01 03:06:33.069503389 +0000 UTC m=+39.180755694"
Nov 01 03:06:33.188713 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:33.188653    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-62mxc" podStartSLOduration=33.945850889 podStartE2EDuration="38.188633767s" totalImagesPullingTime="4.242782878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:05:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:06:33.188571661 +0000 UTC m=+39.299823969" watchObservedRunningTime="2026-11-01 03:06:33.188633767 +0000 UTC m=+39.299886076"
Nov 01 03:06:33.321840 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:33.321717    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chqll" podStartSLOduration=344.931627486 podStartE2EDuration="5m51.321699014s" totalImagesPullingTime="6.390071528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:00:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:06:33.320319215 +0000 UTC m=+39.431571524" watchObservedRunningTime="2026-11-01 03:06:33.321699014 +0000 UTC m=+39.432951320"
Nov 01 03:06:33.384698 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:33.384661    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:33.454077 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:33.454022    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zdrpv" podStartSLOduration=3.117653277 podStartE2EDuration="9.454006636s" totalImagesPullingTime="6.336353359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:06:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:06:33.45268944 +0000 UTC m=+39.563941758" watchObservedRunningTime="2026-11-01 03:06:33.454006636 +0000 UTC m=+39.565258941"
Nov 01 03:06:33.612804 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:33.612728    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:33.629144 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:33.629112    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-cbbs4"]
Nov 01 03:06:33.661316 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:33.661288    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cbbs4"
Nov 01 03:06:33.702080 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:33.702009    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-cbbs4"]
Nov 01 03:06:33.719598 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:33.719572    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 01 03:06:33.720050 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:33.719647    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-j68cz\""
Nov 01 03:06:33.720050 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:33.719690    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 01 03:06:33.721232 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:33.721212    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jh955_f29ea61e-2a78-4c11-9566-f52497e135ff/console-operator/1.log"
Nov 01 03:06:33.721628 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:33.721609    2578 scope.go:122] "RemoveContainer" containerID="3d57ddeb233cc6ef9b56e40c0268f52a146af9adc2a471d9dd2158d7b677263c"
Nov 01 03:06:33.721846 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:33.721817    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jh955_openshift-console-operator(f29ea61e-2a78-4c11-9566-f52497e135ff)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jh955" podUID="f29ea61e-2a78-4c11-9566-f52497e135ff"
Nov 01 03:06:33.787895 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:33.787861    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:33.820998 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:33.820944    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-555tt\" (UniqueName: \"kubernetes.io/projected/f097ea60-e18f-47f0-b6a0-24e2b3a201d0-kube-api-access-555tt\") pod \"migrator-66df5d496b-cbbs4\" (UID: \"f097ea60-e18f-47f0-b6a0-24e2b3a201d0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cbbs4"
Nov 01 03:06:33.922213 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:33.922134    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-555tt\" (UniqueName: \"kubernetes.io/projected/f097ea60-e18f-47f0-b6a0-24e2b3a201d0-kube-api-access-555tt\") pod \"migrator-66df5d496b-cbbs4\" (UID: \"f097ea60-e18f-47f0-b6a0-24e2b3a201d0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cbbs4"
Nov 01 03:06:33.961397 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:33.961369    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:33.983818 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:33.983787    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-555tt\" (UniqueName: \"kubernetes.io/projected/f097ea60-e18f-47f0-b6a0-24e2b3a201d0-kube-api-access-555tt\") pod \"migrator-66df5d496b-cbbs4\" (UID: \"f097ea60-e18f-47f0-b6a0-24e2b3a201d0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cbbs4"
Nov 01 03:06:34.283337 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:34.283265    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cbbs4"
Nov 01 03:06:34.676149 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:34.676126    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-cbbs4"]
Nov 01 03:06:34.693426 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:34.693399    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-wjvnt"]
Nov 01 03:06:34.709894 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:34.709877    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-wjvnt"
Nov 01 03:06:34.712005 ip-10-0-141-122 kubenswrapper[2578]: W1101 03:06:34.711974    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf097ea60_e18f_47f0_b6a0_24e2b3a201d0.slice/crio-de497939fff5cdee5e1dad744e089b36b512add5de3ab717d0729b0f9f64627e WatchSource:0}: Error finding container de497939fff5cdee5e1dad744e089b36b512add5de3ab717d0729b0f9f64627e: Status 404 returned error can't find the container with id de497939fff5cdee5e1dad744e089b36b512add5de3ab717d0729b0f9f64627e
Nov 01 03:06:34.725382 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:34.725354    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cbbs4" event={"ID":"f097ea60-e18f-47f0-b6a0-24e2b3a201d0","Type":"ContainerStarted","Data":"de497939fff5cdee5e1dad744e089b36b512add5de3ab717d0729b0f9f64627e"}
Nov 01 03:06:34.789084 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:34.789061    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-wjvnt"]
Nov 01 03:06:34.817923 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:34.817900    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 01 03:06:34.818163 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:34.818144    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 01 03:06:34.819389 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:34.819374    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 01 03:06:34.819475 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:34.819408    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 01 03:06:34.828910 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:34.828891    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-hdt7b\""
Nov 01 03:06:34.830249 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:34.830226    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bhzm\" (UniqueName: \"kubernetes.io/projected/627e3af2-4a0d-4f19-91bc-760af67e01ad-kube-api-access-8bhzm\") pod \"service-ca-5fffc97bdf-wjvnt\" (UID: \"627e3af2-4a0d-4f19-91bc-760af67e01ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wjvnt"
Nov 01 03:06:34.830366 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:34.830264    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/627e3af2-4a0d-4f19-91bc-760af67e01ad-signing-key\") pod \"service-ca-5fffc97bdf-wjvnt\" (UID: \"627e3af2-4a0d-4f19-91bc-760af67e01ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wjvnt"
Nov 01 03:06:34.830498 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:34.830477    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/627e3af2-4a0d-4f19-91bc-760af67e01ad-signing-cabundle\") pod \"service-ca-5fffc97bdf-wjvnt\" (UID: \"627e3af2-4a0d-4f19-91bc-760af67e01ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wjvnt"
Nov 01 03:06:34.895511 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:34.895489    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:34.931821 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:34.931760    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/627e3af2-4a0d-4f19-91bc-760af67e01ad-signing-cabundle\") pod \"service-ca-5fffc97bdf-wjvnt\" (UID: \"627e3af2-4a0d-4f19-91bc-760af67e01ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wjvnt"
Nov 01 03:06:34.931932 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:34.931870    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bhzm\" (UniqueName: \"kubernetes.io/projected/627e3af2-4a0d-4f19-91bc-760af67e01ad-kube-api-access-8bhzm\") pod \"service-ca-5fffc97bdf-wjvnt\" (UID: \"627e3af2-4a0d-4f19-91bc-760af67e01ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wjvnt"
Nov 01 03:06:34.931932 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:34.931903    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/627e3af2-4a0d-4f19-91bc-760af67e01ad-signing-key\") pod \"service-ca-5fffc97bdf-wjvnt\" (UID: \"627e3af2-4a0d-4f19-91bc-760af67e01ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wjvnt"
Nov 01 03:06:34.932441 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:34.932423    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/627e3af2-4a0d-4f19-91bc-760af67e01ad-signing-cabundle\") pod \"service-ca-5fffc97bdf-wjvnt\" (UID: \"627e3af2-4a0d-4f19-91bc-760af67e01ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wjvnt"
Nov 01 03:06:34.934220 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:34.934204    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/627e3af2-4a0d-4f19-91bc-760af67e01ad-signing-key\") pod \"service-ca-5fffc97bdf-wjvnt\" (UID: \"627e3af2-4a0d-4f19-91bc-760af67e01ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wjvnt"
Nov 01 03:06:34.994106 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:34.994084    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bhzm\" (UniqueName: \"kubernetes.io/projected/627e3af2-4a0d-4f19-91bc-760af67e01ad-kube-api-access-8bhzm\") pod \"service-ca-5fffc97bdf-wjvnt\" (UID: \"627e3af2-4a0d-4f19-91bc-760af67e01ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wjvnt"
Nov 01 03:06:35.032954 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:35.032926    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2vfgr\" (UID: \"caebbba6-44f3-40c1-95ce-b8a5a3f2667b\") " pod="openshift-insights/insights-runtime-extractor-2vfgr"
Nov 01 03:06:35.033051 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:35.033032    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 03:06:35.033102 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:35.033054    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2vfgr\" (UID: \"caebbba6-44f3-40c1-95ce-b8a5a3f2667b\") " pod="openshift-insights/insights-runtime-extractor-2vfgr"
Nov 01 03:06:35.033102 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:35.033084    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-kube-rbac-proxy-cm podName:caebbba6-44f3-40c1-95ce-b8a5a3f2667b nodeName:}" failed. No retries permitted until 2026-11-01 03:06:39.033070763 +0000 UTC m=+45.144323063 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2vfgr" (UID: "caebbba6-44f3-40c1-95ce-b8a5a3f2667b") : configmap "kube-rbac-proxy" not found
Nov 01 03:06:35.033184 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:35.033145    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 03:06:35.033228 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:35.033188    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-insights-runtime-extractor-tls podName:caebbba6-44f3-40c1-95ce-b8a5a3f2667b nodeName:}" failed. No retries permitted until 2026-11-01 03:06:39.033175956 +0000 UTC m=+45.144428256 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2vfgr" (UID: "caebbba6-44f3-40c1-95ce-b8a5a3f2667b") : secret "insights-runtime-extractor-tls" not found
Nov 01 03:06:35.047994 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:35.047973    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-wjvnt"
Nov 01 03:06:35.050687 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:35.050666    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:35.109340 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:35.109317    2578 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-jh955"
Nov 01 03:06:35.109531 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:35.109512    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jh955"
Nov 01 03:06:35.111447 ip-10-0-141-122 kubenswrapper[2578]: I1101 03:06:35.109970    2578 scope.go:122] "RemoveContainer" containerID="3d57ddeb233cc6ef9b56e40c0268f52a146af9adc2a471d9dd2158d7b677263c"
Nov 01 03:06:35.111447 ip-10-0-141-122 kubenswrapper[2578]: E1101 03:06:35.110196    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jh955_openshift-console-operator(f29ea61e-2a78-4c11-9566-f52497e135ff)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jh955" podUID="f29ea61e-2a78-4c11-9566-f52497e135ff"
Nov 01 03:06:35.400425 ip-10-0-141-122 systemd[1]: Stopping Kubernetes Kubelet...
Nov 01 03:06:35.413108 ip-10-0-141-122 systemd[1]: kubelet.service: Deactivated successfully.
Nov 01 03:06:35.413268 ip-10-0-141-122 systemd[1]: Stopped Kubernetes Kubelet.
Nov 01 03:06:35.413428 ip-10-0-141-122 systemd[1]: kubelet.service: Consumed 2.962s CPU time.
Nov 01 03:06:35.430235 ip-10-0-141-122 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 03:06:35.527619 ip-10-0-141-122 kubenswrapper[4930]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 03:06:35.527619 ip-10-0-141-122 kubenswrapper[4930]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 03:06:35.527619 ip-10-0-141-122 kubenswrapper[4930]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 03:06:35.528056 ip-10-0-141-122 kubenswrapper[4930]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 03:06:35.530487 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530413    4930 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:06:35.530487 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530489    4930 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:06:35.530596 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530493    4930 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:06:35.530596 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530496    4930 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:06:35.530596 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530499    4930 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:06:35.530596 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530502    4930 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:06:35.530596 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530505    4930 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:06:35.530596 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530507    4930 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:06:35.530596 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530510    4930 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:06:35.530596 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530513    4930 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:06:35.530596 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530516    4930 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:06:35.530596 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530518    4930 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:06:35.530596 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530521    4930 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:06:35.530596 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530524    4930 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:06:35.530596 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530527    4930 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:06:35.530596 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530530    4930 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:06:35.530596 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530533    4930 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:06:35.530596 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530535    4930 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:06:35.530596 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530539    4930 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:06:35.530596 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530543    4930 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:06:35.530596 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530546    4930 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:06:35.531085 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530549    4930 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:06:35.531085 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530552    4930 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:06:35.531085 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530567    4930 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:06:35.531085 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530570    4930 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:06:35.531085 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530573    4930 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:06:35.531085 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530583    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:06:35.531085 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530587    4930 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:06:35.531085 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530590    4930 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:06:35.531085 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530592    4930 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:06:35.531085 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530595    4930 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:06:35.531085 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530598    4930 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:06:35.531085 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530604    4930 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:06:35.531085 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530607    4930 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:06:35.531085 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530610    4930 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:06:35.531085 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530613    4930 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:06:35.531085 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530616    4930 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:06:35.531085 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530618    4930 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:06:35.531085 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530621    4930 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:06:35.531085 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530624    4930 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:06:35.531085 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530627    4930 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:06:35.531604 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530630    4930 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:06:35.531604 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530632    4930 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:06:35.531604 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530635    4930 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:06:35.531604 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530637    4930 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:06:35.531604 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530640    4930 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:06:35.531604 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530643    4930 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:06:35.531604 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530645    4930 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:06:35.531604 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530649    4930 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:06:35.531604 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530653    4930 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:06:35.531604 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530656    4930 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:06:35.531604 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530659    4930 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:06:35.531604 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530662    4930 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:06:35.531604 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530665    4930 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:06:35.531604 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530668    4930 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:06:35.531604 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530671    4930 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:06:35.531604 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530674    4930 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:06:35.531604 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530676    4930 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:06:35.531604 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530680    4930 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:06:35.531604 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530682    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:06:35.532121 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530685    4930 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:06:35.532121 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530688    4930 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:06:35.532121 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530690    4930 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:06:35.532121 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530693    4930 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:06:35.532121 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530696    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:06:35.532121 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530699    4930 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:06:35.532121 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530701    4930 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:06:35.532121 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530704    4930 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:06:35.532121 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530707    4930 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:06:35.532121 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530709    4930 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:06:35.532121 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530712    4930 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:06:35.532121 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530715    4930 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:06:35.532121 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530718    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:06:35.532121 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530721    4930 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:06:35.532121 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530723    4930 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:06:35.532121 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530726    4930 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:06:35.532121 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530729    4930 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:06:35.532121 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530732    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:06:35.532121 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530734    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:06:35.532662 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530737    4930 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:06:35.532662 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530739    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:06:35.532662 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530744    4930 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:06:35.532662 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530746    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:06:35.532662 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530749    4930 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:06:35.532662 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530752    4930 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:06:35.532662 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530755    4930 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:06:35.532662 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530758    4930 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:06:35.532662 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530760    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:06:35.532662 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530763    4930 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:06:35.532662 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530766    4930 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:06:35.532662 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530769    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:06:35.532662 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530772    4930 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:06:35.532662 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530774    4930 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:06:35.532662 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530777    4930 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:06:35.532662 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530779    4930 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:06:35.532662 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530782    4930 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:06:35.532662 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530784    4930 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:06:35.532662 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530787    4930 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:06:35.532662 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530790    4930 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:06:35.533170 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530792    4930 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:06:35.533170 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530795    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:06:35.533170 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530798    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:06:35.533170 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530801    4930 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:06:35.533170 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530803    4930 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:06:35.533170 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530806    4930 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:06:35.533170 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.530809    4930 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:06:35.533170 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531585    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:06:35.533170 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531592    4930 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:06:35.533170 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531596    4930 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:06:35.533170 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531600    4930 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:06:35.533170 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531603    4930 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:06:35.533170 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531605    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:06:35.533170 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531608    4930 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:06:35.533170 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531611    4930 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:06:35.533170 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531615    4930 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:06:35.533170 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531617    4930 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:06:35.533170 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531620    4930 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:06:35.533170 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531623    4930 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:06:35.533170 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531625    4930 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:06:35.533719 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531628    4930 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:06:35.533719 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531632    4930 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:06:35.533719 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531636    4930 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:06:35.533719 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531642    4930 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:06:35.533719 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531645    4930 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:06:35.533719 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531648    4930 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:06:35.533719 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531650    4930 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:06:35.533719 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531653    4930 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:06:35.533719 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531656    4930 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:06:35.533719 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531659    4930 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:06:35.533719 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531662    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:06:35.533719 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531665    4930 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:06:35.533719 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531667    4930 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:06:35.533719 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531670    4930 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:06:35.533719 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531673    4930 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:06:35.533719 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531676    4930 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:06:35.533719 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531678    4930 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:06:35.533719 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531681    4930 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:06:35.533719 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531683    4930 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:06:35.534196 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531686    4930 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:06:35.534196 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531689    4930 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:06:35.534196 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531692    4930 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:06:35.534196 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531694    4930 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:06:35.534196 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531697    4930 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:06:35.534196 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531700    4930 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:06:35.534196 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531702    4930 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:06:35.534196 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531705    4930 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:06:35.534196 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531708    4930 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:06:35.534196 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531711    4930 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:06:35.534196 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531714    4930 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:06:35.534196 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531717    4930 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:06:35.534196 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531719    4930 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:06:35.534196 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531722    4930 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:06:35.534196 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531725    4930 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:06:35.534196 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531727    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:06:35.534196 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531730    4930 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:06:35.534196 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531733    4930 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:06:35.534196 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531735    4930 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:06:35.534686 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531738    4930 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:06:35.534686 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531741    4930 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:06:35.534686 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531744    4930 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:06:35.534686 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531747    4930 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:06:35.534686 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531750    4930 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:06:35.534686 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531752    4930 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:06:35.534686 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531755    4930 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:06:35.534686 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531758    4930 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:06:35.534686 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531760    4930 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:06:35.534686 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531763    4930 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:06:35.534686 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531766    4930 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:06:35.534686 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531768    4930 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:06:35.534686 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531771    4930 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:06:35.534686 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531775    4930 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:06:35.534686 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531779    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:06:35.534686 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531782    4930 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:06:35.534686 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531785    4930 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:06:35.534686 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531787    4930 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:06:35.534686 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531790    4930 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:06:35.535182 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531792    4930 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:06:35.535182 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531795    4930 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:06:35.535182 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531797    4930 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:06:35.535182 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531800    4930 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:06:35.535182 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531803    4930 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:06:35.535182 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531806    4930 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:06:35.535182 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531808    4930 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:06:35.535182 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531811    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:06:35.535182 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531813    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:06:35.535182 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531816    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:06:35.535182 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531819    4930 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:06:35.535182 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531822    4930 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:06:35.535182 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531824    4930 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:06:35.535182 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531827    4930 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:06:35.535182 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531829    4930 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:06:35.535182 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531832    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:06:35.535182 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531834    4930 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:06:35.535182 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531837    4930 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:06:35.535182 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531839    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:06:35.535182 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531842    4930 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:06:35.535700 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531845    4930 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:06:35.535700 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531852    4930 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:06:35.535700 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531854    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:06:35.535700 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531857    4930 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:06:35.535700 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531859    4930 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:06:35.535700 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531862    4930 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:06:35.535700 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531865    4930 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:06:35.535700 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531867    4930 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:06:35.535700 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531870    4930 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:06:35.535700 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531873    4930 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:06:35.535700 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531875    4930 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:06:35.535700 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531878    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:06:35.535700 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531881    4930 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:06:35.535700 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531883    4930 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:06:35.535700 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531886    4930 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:06:35.535700 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.531890    4930 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:06:35.535700 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532367    4930 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:06:35.535700 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532373    4930 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:06:35.535700 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532376    4930 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:06:35.536193 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532379    4930 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:06:35.536193 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532382    4930 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:06:35.536193 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532384    4930 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:06:35.536193 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532388    4930 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:06:35.536193 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532390    4930 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:06:35.536193 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532393    4930 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:06:35.536193 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532396    4930 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:06:35.536193 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532398    4930 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:06:35.536193 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532401    4930 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:06:35.536193 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532403    4930 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:06:35.536193 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532406    4930 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:06:35.536193 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532409    4930 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:06:35.536193 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532411    4930 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:06:35.536193 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532414    4930 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:06:35.536193 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532417    4930 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:06:35.536193 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532420    4930 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:06:35.536193 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532423    4930 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:06:35.536193 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532425    4930 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:06:35.536193 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532428    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:06:35.536193 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532432    4930 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:06:35.536704 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532435    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:06:35.536704 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532438    4930 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:06:35.536704 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532442    4930 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:06:35.536704 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532444    4930 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:06:35.536704 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532447    4930 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:06:35.536704 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532450    4930 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:06:35.536704 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532452    4930 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:06:35.536704 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532455    4930 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:06:35.536704 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532457    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:06:35.536704 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532460    4930 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:06:35.536704 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532463    4930 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:06:35.536704 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532465    4930 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:06:35.536704 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532468    4930 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:06:35.536704 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532471    4930 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:06:35.536704 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532473    4930 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:06:35.536704 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532476    4930 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:06:35.536704 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532479    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:06:35.536704 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532482    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:06:35.536704 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532484    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:06:35.537225 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532487    4930 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:06:35.537225 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532490    4930 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:06:35.537225 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532492    4930 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:06:35.537225 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532495    4930 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:06:35.537225 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532498    4930 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:06:35.537225 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532500    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:06:35.537225 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532503    4930 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:06:35.537225 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532505    4930 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:06:35.537225 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532508    4930 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:06:35.537225 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532510    4930 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:06:35.537225 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532513    4930 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:06:35.537225 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532515    4930 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:06:35.537225 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532518    4930 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:06:35.537225 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532520    4930 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:06:35.537225 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532523    4930 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:06:35.537225 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532526    4930 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:06:35.537225 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532529    4930 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:06:35.537225 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532531    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:06:35.537225 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532534    4930 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:06:35.537740 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532537    4930 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:06:35.537740 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532540    4930 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:06:35.537740 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532542    4930 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:06:35.537740 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532545    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:06:35.537740 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532547    4930 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:06:35.537740 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532550    4930 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:06:35.537740 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532566    4930 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:06:35.537740 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532569    4930 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:06:35.537740 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532572    4930 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:06:35.537740 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532574    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:06:35.537740 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532577    4930 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:06:35.537740 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532580    4930 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:06:35.537740 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532582    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:06:35.537740 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532587    4930 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:06:35.537740 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532589    4930 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:06:35.537740 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532592    4930 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:06:35.537740 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532594    4930 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:06:35.537740 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532597    4930 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:06:35.537740 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532600    4930 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:06:35.537740 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532602    4930 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:06:35.538275 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532605    4930 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:06:35.538275 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532607    4930 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:06:35.538275 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532610    4930 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:06:35.538275 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532613    4930 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:06:35.538275 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532615    4930 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:06:35.538275 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532618    4930 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:06:35.538275 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532620    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:06:35.538275 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532623    4930 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:06:35.538275 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532630    4930 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:06:35.538275 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532633    4930 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:06:35.538275 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532635    4930 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:06:35.538275 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532638    4930 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:06:35.538275 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532641    4930 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:06:35.538275 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532643    4930 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:06:35.538275 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532646    4930 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:06:35.538275 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532649    4930 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:06:35.538275 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532651    4930 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:06:35.538275 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532654    4930 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:06:35.538275 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532657    4930 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:06:35.538275 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532659    4930 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:06:35.538779 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532663    4930 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:06:35.538779 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532668    4930 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:06:35.538779 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532672    4930 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:06:35.538779 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532676    4930 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:06:35.538779 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.532679    4930 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:06:35.538779 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532772    4930 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 03:06:35.538779 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532788    4930 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 03:06:35.538779 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532797    4930 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 03:06:35.538779 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532804    4930 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 03:06:35.538779 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532810    4930 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 03:06:35.538779 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532815    4930 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 03:06:35.538779 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532821    4930 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 03:06:35.538779 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532830    4930 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 03:06:35.538779 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532833    4930 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 03:06:35.538779 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532837    4930 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 03:06:35.538779 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532840    4930 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 03:06:35.538779 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532843    4930 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 03:06:35.538779 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532847    4930 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 03:06:35.538779 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532850    4930 flags.go:64] FLAG: --cgroup-root=""
Nov 01 03:06:35.538779 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532853    4930 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 03:06:35.538779 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532858    4930 flags.go:64] FLAG: --client-ca-file=""
Nov 01 03:06:35.538779 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532863    4930 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 03:06:35.539307 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532866    4930 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 03:06:35.539307 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532871    4930 flags.go:64] FLAG: --cluster-domain=""
Nov 01 03:06:35.539307 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532874    4930 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 03:06:35.539307 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532877    4930 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 03:06:35.539307 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532880    4930 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 03:06:35.539307 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532883    4930 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 03:06:35.539307 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532887    4930 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 03:06:35.539307 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532890    4930 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 03:06:35.539307 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532894    4930 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 03:06:35.539307 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532903    4930 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 03:06:35.539307 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532906    4930 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 03:06:35.539307 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532909    4930 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 03:06:35.539307 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532912    4930 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 03:06:35.539307 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532916    4930 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 03:06:35.539307 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532919    4930 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 03:06:35.539307 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532923    4930 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 03:06:35.539307 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532926    4930 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 03:06:35.539307 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532929    4930 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 03:06:35.539307 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532932    4930 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 03:06:35.539307 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532935    4930 flags.go:64] FLAG: --enable-server="true"
Nov 01 03:06:35.539307 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532938    4930 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 03:06:35.539307 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532942    4930 flags.go:64] FLAG: --event-burst="100"
Nov 01 03:06:35.539307 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532945    4930 flags.go:64] FLAG: --event-qps="50"
Nov 01 03:06:35.539307 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532948    4930 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 03:06:35.539943 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532951    4930 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 03:06:35.539943 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532954    4930 flags.go:64] FLAG: --eviction-hard=""
Nov 01 03:06:35.539943 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532958    4930 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 03:06:35.539943 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532961    4930 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 03:06:35.539943 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532965    4930 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 03:06:35.539943 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532968    4930 flags.go:64] FLAG: --eviction-soft=""
Nov 01 03:06:35.539943 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532973    4930 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 03:06:35.539943 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532977    4930 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 03:06:35.539943 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532980    4930 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 03:06:35.539943 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532983    4930 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 03:06:35.539943 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532986    4930 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 03:06:35.539943 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532989    4930 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 03:06:35.539943 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532992    4930 flags.go:64] FLAG: --feature-gates=""
Nov 01 03:06:35.539943 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.532996    4930 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 03:06:35.539943 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533000    4930 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 03:06:35.539943 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533003    4930 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 03:06:35.539943 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533006    4930 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 03:06:35.539943 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533009    4930 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 03:06:35.539943 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533020    4930 flags.go:64] FLAG: --help="false"
Nov 01 03:06:35.539943 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533024    4930 flags.go:64] FLAG: --hostname-override="ip-10-0-141-122.us-west-1.compute.internal"
Nov 01 03:06:35.539943 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533027    4930 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 03:06:35.539943 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533030    4930 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 03:06:35.539943 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533033    4930 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 03:06:35.540589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533036    4930 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 03:06:35.540589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533040    4930 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 03:06:35.540589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533043    4930 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 03:06:35.540589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533045    4930 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 03:06:35.540589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533048    4930 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 03:06:35.540589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533051    4930 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 03:06:35.540589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533054    4930 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 03:06:35.540589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533057    4930 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 03:06:35.540589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533060    4930 flags.go:64] FLAG: --kube-reserved=""
Nov 01 03:06:35.540589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533063    4930 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 03:06:35.540589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533066    4930 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 03:06:35.540589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533069    4930 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 03:06:35.540589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533072    4930 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 03:06:35.540589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533074    4930 flags.go:64] FLAG: --lock-file=""
Nov 01 03:06:35.540589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533077    4930 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 03:06:35.540589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533082    4930 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 03:06:35.540589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533086    4930 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 03:06:35.540589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533092    4930 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 03:06:35.540589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533095    4930 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 03:06:35.540589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533098    4930 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 03:06:35.540589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533101    4930 flags.go:64] FLAG: --logging-format="text"
Nov 01 03:06:35.540589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533104    4930 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 03:06:35.540589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533107    4930 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 03:06:35.540589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533110    4930 flags.go:64] FLAG: --manifest-url=""
Nov 01 03:06:35.541422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533113    4930 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 03:06:35.541422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533118    4930 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 03:06:35.541422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533121    4930 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 03:06:35.541422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533124    4930 flags.go:64] FLAG: --max-pods="110"
Nov 01 03:06:35.541422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533133    4930 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 03:06:35.541422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533136    4930 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 03:06:35.541422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533139    4930 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 03:06:35.541422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533142    4930 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 03:06:35.541422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533145    4930 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 03:06:35.541422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533148    4930 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 03:06:35.541422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533151    4930 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 03:06:35.541422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533158    4930 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 03:06:35.541422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533162    4930 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 03:06:35.541422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533164    4930 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 03:06:35.541422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533168    4930 flags.go:64] FLAG: --pod-cidr=""
Nov 01 03:06:35.541422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533171    4930 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 03:06:35.541422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533174    4930 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 03:06:35.541422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533177    4930 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 03:06:35.541422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533180    4930 flags.go:64] FLAG: --port="10250"
Nov 01 03:06:35.541422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533183    4930 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 03:06:35.541422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533186    4930 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0e2dc9ee5fdad2e52"
Nov 01 03:06:35.541422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533190    4930 flags.go:64] FLAG: --qos-reserved=""
Nov 01 03:06:35.541422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533193    4930 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 03:06:35.541422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533197    4930 flags.go:64] FLAG: --register-node="true"
Nov 01 03:06:35.541422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533201    4930 flags.go:64] FLAG: --register-with-taints=""
Nov 01 03:06:35.541422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533205    4930 flags.go:64] FLAG: --registry-burst="10"
Nov 01 03:06:35.542105 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533208    4930 flags.go:64] FLAG: --registry-qps="5"
Nov 01 03:06:35.542105 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533211    4930 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 03:06:35.542105 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533214    4930 flags.go:64] FLAG: --reserved-memory=""
Nov 01 03:06:35.542105 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533218    4930 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 03:06:35.542105 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533221    4930 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 03:06:35.542105 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533224    4930 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 03:06:35.542105 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533227    4930 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 03:06:35.542105 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533230    4930 flags.go:64] FLAG: --runonce="false"
Nov 01 03:06:35.542105 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533243    4930 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 03:06:35.542105 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533249    4930 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 03:06:35.542105 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533254    4930 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 03:06:35.542105 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533271    4930 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 03:06:35.542105 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533275    4930 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 03:06:35.542105 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533278    4930 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 03:06:35.542105 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533281    4930 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 03:06:35.542105 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533284    4930 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 03:06:35.542105 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533287    4930 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 03:06:35.542105 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533290    4930 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 03:06:35.542105 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533293    4930 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 03:06:35.542105 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533296    4930 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 03:06:35.542105 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533299    4930 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 03:06:35.542105 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533302    4930 flags.go:64] FLAG: --system-cgroups=""
Nov 01 03:06:35.542105 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533305    4930 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 03:06:35.542105 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533310    4930 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 03:06:35.542105 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533313    4930 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 03:06:35.542934 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533316    4930 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 03:06:35.542934 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533320    4930 flags.go:64] FLAG: --tls-min-version=""
Nov 01 03:06:35.542934 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533323    4930 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 03:06:35.542934 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533326    4930 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 03:06:35.542934 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533330    4930 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 03:06:35.542934 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533334    4930 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 03:06:35.542934 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533337    4930 flags.go:64] FLAG: --v="2"
Nov 01 03:06:35.542934 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533342    4930 flags.go:64] FLAG: --version="false"
Nov 01 03:06:35.542934 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533346    4930 flags.go:64] FLAG: --vmodule=""
Nov 01 03:06:35.542934 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533350    4930 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 03:06:35.542934 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533353    4930 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 03:06:35.542934 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533482    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:06:35.542934 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533486    4930 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:06:35.542934 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533489    4930 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:06:35.542934 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533492    4930 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:06:35.542934 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533495    4930 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:06:35.542934 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533498    4930 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:06:35.542934 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533501    4930 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:06:35.542934 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533504    4930 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:06:35.542934 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533508    4930 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:06:35.542934 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533511    4930 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:06:35.542934 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533513    4930 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:06:35.543722 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533516    4930 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:06:35.543722 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533519    4930 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:06:35.543722 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533522    4930 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:06:35.543722 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533524    4930 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:06:35.543722 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533527    4930 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:06:35.543722 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533530    4930 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:06:35.543722 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533532    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:06:35.543722 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533535    4930 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:06:35.543722 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533538    4930 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:06:35.543722 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533541    4930 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:06:35.543722 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533543    4930 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:06:35.543722 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533546    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:06:35.543722 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533549    4930 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:06:35.543722 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533552    4930 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:06:35.543722 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533567    4930 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:06:35.543722 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533571    4930 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:06:35.543722 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533573    4930 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:06:35.543722 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533576    4930 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:06:35.543722 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533579    4930 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:06:35.543722 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533582    4930 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:06:35.544500 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533585    4930 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:06:35.544500 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533587    4930 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:06:35.544500 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533590    4930 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:06:35.544500 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533592    4930 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:06:35.544500 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533595    4930 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:06:35.544500 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533598    4930 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:06:35.544500 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533601    4930 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:06:35.544500 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533603    4930 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:06:35.544500 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533606    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:06:35.544500 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533608    4930 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:06:35.544500 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533611    4930 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:06:35.544500 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533615    4930 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:06:35.544500 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533620    4930 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:06:35.544500 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533623    4930 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:06:35.544500 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533625    4930 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:06:35.544500 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533629    4930 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:06:35.544500 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533631    4930 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:06:35.544500 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533634    4930 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:06:35.544500 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533636    4930 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:06:35.545125 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533639    4930 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:06:35.545125 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533642    4930 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:06:35.545125 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533645    4930 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:06:35.545125 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533647    4930 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:06:35.545125 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533650    4930 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:06:35.545125 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533652    4930 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:06:35.545125 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533655    4930 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:06:35.545125 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533659    4930 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:06:35.545125 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533663    4930 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:06:35.545125 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533666    4930 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:06:35.545125 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533668    4930 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:06:35.545125 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533671    4930 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:06:35.545125 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533673    4930 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:06:35.545125 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533676    4930 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:06:35.545125 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533679    4930 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:06:35.545125 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533682    4930 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:06:35.545125 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533684    4930 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:06:35.545125 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533687    4930 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:06:35.545125 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533690    4930 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:06:35.545664 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533692    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:06:35.545664 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533695    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:06:35.545664 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533697    4930 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:06:35.545664 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533700    4930 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:06:35.545664 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533703    4930 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:06:35.545664 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533706    4930 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:06:35.545664 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533709    4930 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:06:35.545664 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533711    4930 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:06:35.545664 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533714    4930 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:06:35.545664 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533716    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:06:35.545664 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533719    4930 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:06:35.545664 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533722    4930 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:06:35.545664 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533725    4930 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:06:35.545664 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533728    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:06:35.545664 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533731    4930 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:06:35.545664 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533734    4930 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:06:35.545664 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533736    4930 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:06:35.545664 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533739    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:06:35.545664 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533741    4930 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:06:35.545664 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533744    4930 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:06:35.546188 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533748    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:06:35.546188 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533751    4930 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:06:35.546188 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533754    4930 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:06:35.546188 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533757    4930 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:06:35.546188 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533760    4930 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:06:35.546188 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533762    4930 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:06:35.546188 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533765    4930 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:06:35.546188 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533768    4930 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:06:35.546188 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533770    4930 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:06:35.546188 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533773    4930 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:06:35.546188 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533776    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:06:35.546188 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533778    4930 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:06:35.546188 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533781    4930 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:06:35.546188 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533783    4930 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:06:35.546188 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533787    4930 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:06:35.546188 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533792    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:06:35.546188 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533795    4930 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:06:35.546753 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.533820    4930 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 03:06:35.546753 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533858    4930 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:06:35.546753 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533863    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:06:35.546753 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533866    4930 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:06:35.546753 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533869    4930 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:06:35.546753 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533872    4930 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:06:35.546753 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533874    4930 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:06:35.546753 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533877    4930 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:06:35.546753 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533879    4930 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:06:35.546753 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533882    4930 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:06:35.546753 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533885    4930 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:06:35.546753 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533887    4930 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:06:35.546753 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533890    4930 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:06:35.546753 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533892    4930 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:06:35.546753 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533895    4930 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:06:35.546753 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533899    4930 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:06:35.547245 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533902    4930 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:06:35.547245 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533905    4930 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:06:35.547245 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533909    4930 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:06:35.547245 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533911    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:06:35.547245 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533914    4930 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:06:35.547245 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533916    4930 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:06:35.547245 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533920    4930 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:06:35.547245 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533924    4930 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:06:35.547245 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533927    4930 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:06:35.547245 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533930    4930 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:06:35.547245 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533933    4930 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:06:35.547245 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533935    4930 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:06:35.547245 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533938    4930 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:06:35.547245 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533941    4930 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:06:35.547245 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533943    4930 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:06:35.547245 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533946    4930 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:06:35.547245 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533949    4930 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:06:35.547245 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533951    4930 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:06:35.547245 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533955    4930 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:06:35.547806 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533957    4930 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:06:35.547806 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533960    4930 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:06:35.547806 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533962    4930 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:06:35.547806 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533965    4930 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:06:35.547806 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533967    4930 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:06:35.547806 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533970    4930 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:06:35.547806 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533973    4930 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:06:35.547806 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533975    4930 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:06:35.547806 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533978    4930 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:06:35.547806 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533981    4930 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:06:35.547806 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533984    4930 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:06:35.547806 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533987    4930 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:06:35.547806 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533991    4930 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:06:35.547806 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533994    4930 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:06:35.547806 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.533999    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:06:35.547806 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534001    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:06:35.547806 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534004    4930 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:06:35.547806 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534007    4930 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:06:35.547806 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534009    4930 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:06:35.548273 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534012    4930 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:06:35.548273 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534015    4930 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:06:35.548273 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534018    4930 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:06:35.548273 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534020    4930 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:06:35.548273 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534023    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:06:35.548273 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534026    4930 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:06:35.548273 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534028    4930 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:06:35.548273 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534031    4930 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:06:35.548273 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534033    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:06:35.548273 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534036    4930 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:06:35.548273 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534039    4930 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:06:35.548273 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534042    4930 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:06:35.548273 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534045    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:06:35.548273 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534047    4930 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:06:35.548273 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534050    4930 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:06:35.548273 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534053    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:06:35.548273 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534055    4930 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:06:35.548273 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534058    4930 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:06:35.548273 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534060    4930 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:06:35.548273 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534063    4930 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:06:35.548768 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534065    4930 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:06:35.548768 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534068    4930 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:06:35.548768 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534070    4930 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:06:35.548768 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534073    4930 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:06:35.548768 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534076    4930 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:06:35.548768 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534080    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:06:35.548768 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534082    4930 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:06:35.548768 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534086    4930 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:06:35.548768 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534089    4930 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:06:35.548768 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534093    4930 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:06:35.548768 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534097    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:06:35.548768 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534100    4930 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:06:35.548768 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534103    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:06:35.548768 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534105    4930 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:06:35.548768 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534108    4930 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:06:35.548768 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534111    4930 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:06:35.548768 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534113    4930 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:06:35.548768 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534116    4930 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:06:35.548768 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534118    4930 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:06:35.549216 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534121    4930 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:06:35.549216 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534124    4930 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:06:35.549216 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534126    4930 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:06:35.549216 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534129    4930 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:06:35.549216 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534131    4930 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:06:35.549216 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534134    4930 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:06:35.549216 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534137    4930 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:06:35.549216 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534139    4930 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:06:35.549216 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534142    4930 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:06:35.549216 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534148    4930 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:06:35.549216 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534151    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:06:35.549216 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534154    4930 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:06:35.549216 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534156    4930 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:06:35.549216 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.534159    4930 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:06:35.549216 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.537818    4930 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 03:06:35.549216 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.537835    4930 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 03:06:35.549216 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.537852    4930 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 03:06:35.549216 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.537858    4930 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 03:06:35.549216 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.537916    4930 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:06:35.549216 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.537921    4930 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:06:35.549859 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.537925    4930 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:06:35.549859 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.537928    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:06:35.549859 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.537931    4930 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:06:35.549859 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.537934    4930 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:06:35.549859 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.537936    4930 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:06:35.549859 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.537939    4930 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:06:35.549859 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.537942    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:06:35.549859 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.537944    4930 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:06:35.549859 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.537947    4930 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:06:35.549859 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.537950    4930 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:06:35.549859 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.537953    4930 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:06:35.549859 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.537956    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:06:35.549859 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.537959    4930 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:06:35.549859 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.537961    4930 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:06:35.549859 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.537964    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:06:35.549859 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.537967    4930 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:06:35.549859 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.537969    4930 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:06:35.549859 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.537972    4930 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:06:35.549859 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.537976    4930 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:06:35.550317 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.537978    4930 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:06:35.550317 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.537981    4930 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:06:35.550317 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.537983    4930 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:06:35.550317 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.537986    4930 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:06:35.550317 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.537989    4930 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:06:35.550317 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.537992    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:06:35.550317 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.537995    4930 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:06:35.550317 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.537998    4930 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:06:35.550317 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538002    4930 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:06:35.550317 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538004    4930 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:06:35.550317 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538007    4930 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:06:35.550317 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538010    4930 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:06:35.550317 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538013    4930 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:06:35.550317 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538016    4930 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:06:35.550317 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538018    4930 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:06:35.550317 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538021    4930 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:06:35.550317 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538024    4930 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:06:35.550317 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538026    4930 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:06:35.550317 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538030    4930 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:06:35.550317 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538034    4930 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:06:35.550840 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538037    4930 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:06:35.550840 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538040    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:06:35.550840 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538044    4930 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:06:35.550840 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538046    4930 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:06:35.550840 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538049    4930 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:06:35.550840 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538052    4930 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:06:35.550840 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538054    4930 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:06:35.550840 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538057    4930 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:06:35.550840 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538060    4930 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:06:35.550840 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538063    4930 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:06:35.550840 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538065    4930 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:06:35.550840 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538068    4930 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:06:35.550840 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538071    4930 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:06:35.550840 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538073    4930 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:06:35.550840 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538076    4930 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:06:35.550840 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538079    4930 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:06:35.550840 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538082    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:06:35.550840 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538085    4930 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:06:35.550840 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538087    4930 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:06:35.550840 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538090    4930 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:06:35.551327 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538093    4930 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:06:35.551327 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538096    4930 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:06:35.551327 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538099    4930 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:06:35.551327 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538102    4930 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:06:35.551327 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538105    4930 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:06:35.551327 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538108    4930 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:06:35.551327 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538110    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:06:35.551327 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538113    4930 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:06:35.551327 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538116    4930 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:06:35.551327 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538119    4930 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:06:35.551327 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538122    4930 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:06:35.551327 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538124    4930 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:06:35.551327 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538127    4930 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:06:35.551327 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538129    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:06:35.551327 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538132    4930 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:06:35.551327 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538135    4930 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:06:35.551327 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538138    4930 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:06:35.551327 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538140    4930 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:06:35.551327 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538143    4930 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:06:35.551327 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538146    4930 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:06:35.551866 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538149    4930 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:06:35.551866 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538151    4930 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:06:35.551866 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538154    4930 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:06:35.551866 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538157    4930 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:06:35.551866 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538160    4930 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:06:35.551866 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538163    4930 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:06:35.551866 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538166    4930 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:06:35.551866 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538169    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:06:35.551866 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538172    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:06:35.551866 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538176    4930 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:06:35.551866 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538179    4930 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:06:35.551866 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538183    4930 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:06:35.551866 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538186    4930 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:06:35.551866 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538188    4930 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:06:35.551866 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538192    4930 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:06:35.551866 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538194    4930 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:06:35.551866 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538197    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:06:35.551866 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538200    4930 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:06:35.551866 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538202    4930 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:06:35.552321 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538205    4930 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:06:35.552321 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538208    4930 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:06:35.552321 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538210    4930 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:06:35.552321 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538213    4930 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:06:35.552321 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538215    4930 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:06:35.552321 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538218    4930 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:06:35.552321 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.538245    4930 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 03:06:35.552321 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538385    4930 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:06:35.552321 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538389    4930 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:06:35.552321 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538392    4930 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:06:35.552321 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538395    4930 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:06:35.552321 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538398    4930 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:06:35.552321 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538401    4930 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:06:35.552321 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538403    4930 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:06:35.552321 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538406    4930 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:06:35.552321 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538409    4930 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:06:35.552732 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538411    4930 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:06:35.552732 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538414    4930 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:06:35.552732 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538418    4930 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:06:35.552732 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538421    4930 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:06:35.552732 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538423    4930 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:06:35.552732 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538426    4930 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:06:35.552732 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538429    4930 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:06:35.552732 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538431    4930 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:06:35.552732 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538434    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:06:35.552732 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538436    4930 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:06:35.552732 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538439    4930 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:06:35.552732 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538442    4930 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:06:35.552732 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538449    4930 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:06:35.552732 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538452    4930 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:06:35.552732 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538455    4930 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:06:35.552732 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538458    4930 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:06:35.552732 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538460    4930 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:06:35.552732 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538463    4930 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:06:35.552732 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538466    4930 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:06:35.553188 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538468    4930 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:06:35.553188 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538471    4930 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:06:35.553188 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538474    4930 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:06:35.553188 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538476    4930 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:06:35.553188 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538479    4930 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:06:35.553188 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538481    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:06:35.553188 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538484    4930 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:06:35.553188 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538487    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:06:35.553188 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538490    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:06:35.553188 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538492    4930 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:06:35.553188 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538495    4930 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:06:35.553188 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538497    4930 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:06:35.553188 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538500    4930 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:06:35.553188 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538503    4930 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:06:35.553188 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538505    4930 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:06:35.553188 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538508    4930 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:06:35.553188 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538511    4930 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:06:35.553188 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538513    4930 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:06:35.553188 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538516    4930 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:06:35.553777 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538520    4930 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:06:35.553777 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538524    4930 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:06:35.553777 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538527    4930 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:06:35.553777 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538531    4930 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:06:35.553777 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538534    4930 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:06:35.553777 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538537    4930 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:06:35.553777 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538539    4930 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:06:35.553777 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538543    4930 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:06:35.553777 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538545    4930 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:06:35.553777 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538548    4930 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:06:35.553777 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538551    4930 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:06:35.553777 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538568    4930 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:06:35.553777 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538571    4930 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:06:35.553777 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538574    4930 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:06:35.553777 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538583    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:06:35.553777 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538586    4930 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:06:35.553777 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538589    4930 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:06:35.553777 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538592    4930 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:06:35.553777 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538594    4930 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:06:35.553777 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538597    4930 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:06:35.554310 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538600    4930 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:06:35.554310 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538603    4930 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:06:35.554310 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538606    4930 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:06:35.554310 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538608    4930 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:06:35.554310 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538611    4930 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:06:35.554310 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538614    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:06:35.554310 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538616    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:06:35.554310 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538619    4930 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:06:35.554310 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538624    4930 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:06:35.554310 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538628    4930 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:06:35.554310 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538631    4930 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:06:35.554310 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538633    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:06:35.554310 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538636    4930 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:06:35.554310 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538639    4930 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:06:35.554310 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538641    4930 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:06:35.554310 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538644    4930 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:06:35.554310 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538647    4930 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:06:35.554310 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538650    4930 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:06:35.554310 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538652    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:06:35.554797 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538655    4930 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:06:35.554797 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538658    4930 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:06:35.554797 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538661    4930 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:06:35.554797 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538664    4930 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:06:35.554797 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538666    4930 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:06:35.554797 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538669    4930 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:06:35.554797 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538671    4930 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:06:35.554797 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538674    4930 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:06:35.554797 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538677    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:06:35.554797 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538679    4930 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:06:35.554797 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538682    4930 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:06:35.554797 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538684    4930 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:06:35.554797 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538687    4930 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:06:35.554797 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538689    4930 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:06:35.554797 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538692    4930 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:06:35.554797 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538694    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:06:35.554797 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538697    4930 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:06:35.554797 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538699    4930 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:06:35.554797 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538703    4930 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:06:35.554797 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538705    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:06:35.555299 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.538729    4930 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 03:06:35.555299 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538781    4930 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 03:06:35.555299 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538785    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 03:06:35.555299 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538789    4930 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 03:06:35.555299 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538792    4930 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 03:06:35.555299 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538794    4930 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 03:06:35.555299 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538797    4930 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 03:06:35.555299 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538800    4930 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 03:06:35.555299 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538803    4930 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 03:06:35.555299 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538805    4930 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 03:06:35.555299 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538808    4930 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 03:06:35.555299 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538811    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 03:06:35.555299 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538814    4930 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 03:06:35.555299 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538816    4930 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 03:06:35.555299 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538819    4930 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 03:06:35.555299 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538822    4930 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 03:06:35.555701 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538825    4930 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 03:06:35.555701 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538827    4930 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 03:06:35.555701 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538830    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 03:06:35.555701 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538832    4930 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 03:06:35.555701 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538835    4930 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 03:06:35.555701 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538838    4930 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 03:06:35.555701 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538840    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 03:06:35.555701 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538843    4930 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 03:06:35.555701 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538846    4930 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 03:06:35.555701 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538848    4930 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 03:06:35.555701 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538851    4930 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 03:06:35.555701 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538854    4930 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 03:06:35.555701 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538856    4930 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 03:06:35.555701 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538859    4930 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 03:06:35.555701 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538862    4930 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 03:06:35.555701 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538864    4930 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 03:06:35.555701 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538867    4930 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 03:06:35.555701 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538869    4930 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 03:06:35.555701 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538872    4930 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 03:06:35.555701 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538875    4930 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 03:06:35.556178 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538877    4930 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 03:06:35.556178 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538880    4930 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 03:06:35.556178 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538882    4930 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 03:06:35.556178 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538885    4930 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 03:06:35.556178 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538887    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 03:06:35.556178 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538890    4930 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 03:06:35.556178 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538893    4930 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 03:06:35.556178 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538895    4930 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 03:06:35.556178 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538898    4930 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 03:06:35.556178 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538901    4930 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 03:06:35.556178 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538903    4930 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 03:06:35.556178 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538908    4930 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 03:06:35.556178 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538910    4930 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 03:06:35.556178 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538913    4930 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 03:06:35.556178 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538916    4930 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 03:06:35.556178 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538918    4930 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 03:06:35.556178 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538921    4930 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 03:06:35.556178 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538924    4930 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 03:06:35.556178 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538926    4930 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 03:06:35.556686 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538929    4930 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 03:06:35.556686 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538931    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 03:06:35.556686 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538934    4930 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 03:06:35.556686 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538937    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 03:06:35.556686 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538940    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 03:06:35.556686 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538942    4930 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 03:06:35.556686 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538945    4930 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 03:06:35.556686 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538947    4930 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 03:06:35.556686 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538950    4930 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 03:06:35.556686 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538952    4930 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 03:06:35.556686 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538955    4930 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 03:06:35.556686 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538958    4930 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 03:06:35.556686 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538960    4930 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 03:06:35.556686 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538963    4930 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 03:06:35.556686 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538965    4930 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 03:06:35.556686 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538972    4930 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 03:06:35.556686 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538976    4930 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 03:06:35.556686 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538979    4930 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 03:06:35.556686 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538982    4930 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 03:06:35.557149 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538986    4930 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 03:06:35.557149 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538989    4930 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 03:06:35.557149 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538992    4930 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 03:06:35.557149 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538994    4930 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 03:06:35.557149 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538997    4930 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 03:06:35.557149 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.538999    4930 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 03:06:35.557149 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.539002    4930 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 03:06:35.557149 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.539005    4930 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 03:06:35.557149 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.539008    4930 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 03:06:35.557149 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.539011    4930 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 03:06:35.557149 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.539013    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 03:06:35.557149 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.539016    4930 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 03:06:35.557149 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.539018    4930 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 03:06:35.557149 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.539021    4930 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 03:06:35.557149 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.539024    4930 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 03:06:35.557149 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.539026    4930 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 03:06:35.557149 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.539029    4930 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 03:06:35.557149 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.539031    4930 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 03:06:35.557149 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.539034    4930 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 03:06:35.557149 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.539037    4930 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 03:06:35.557683 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.539041    4930 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 03:06:35.557683 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.539043    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 03:06:35.557683 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.539046    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 03:06:35.557683 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.539049    4930 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 03:06:35.557683 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.539053    4930 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 03:06:35.557683 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.539057    4930 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 03:06:35.557683 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.539059    4930 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 03:06:35.557683 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.539062    4930 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 03:06:35.557683 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.539067    4930 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 03:06:35.557683 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.539070    4930 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 03:06:35.557683 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.539072    4930 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 03:06:35.557683 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.539075    4930 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 03:06:35.557683 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:35.539077    4930 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 03:06:35.557683 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.539129    4930 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 03:06:35.557683 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.540081    4930 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 01 03:06:35.557683 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.540132    4930 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 01 03:06:35.557683 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.540309    4930 server.go:1014] "Starting client certificate rotation"
Nov 01 03:06:35.557683 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.540461    4930 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 03:06:35.558120 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.540507    4930 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 03:00:54 +0000 UTC" deadline="2028-04-01 14:03:25.748232506 +0000 UTC"
Nov 01 03:06:35.558120 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.540567    4930 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12418h56m50.20768243s"
Nov 01 03:06:35.558120 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.540929    4930 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 03:06:35.558120 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.542319    4930 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 03:06:35.558120 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.544694    4930 log.go:25] "Validated CRI v1 runtime API"
Nov 01 03:06:35.558120 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.546374    4930 log.go:25] "Validated CRI v1 image API"
Nov 01 03:06:35.558120 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.547191    4930 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 03:06:35.558120 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.548972    4930 fs.go:135] Filesystem UUIDs: map[3c16cd3a-e64a-47c3-8190-467f00fdb4dc:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ac5ba056-168c-4877-a711-d4cf6e474e80:/dev/nvme0n1p4]
Nov 01 03:06:35.558380 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.548991    4930 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0b7cef567f173ef3b6d0baafd5015f8074a46e4fb248df070231b3edd177ed2f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0b7cef567f173ef3b6d0baafd5015f8074a46e4fb248df070231b3edd177ed2f/userdata/shm major:0 minor:289 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/12af2981349a6df9f1a2182daa931d656f31626994e1b49fb30a715a93b95b48/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/12af2981349a6df9f1a2182daa931d656f31626994e1b49fb30a715a93b95b48/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/18b2e76dfc01da8daf9dbf0daef2c0616a2343e95f4faea38329ce5563c0fbe4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/18b2e76dfc01da8daf9dbf0daef2c0616a2343e95f4faea38329ce5563c0fbe4/userdata/shm major:0 minor:259 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3697c9672a50903bf381fe097bc80913031101f10a9832dd057ba5be41978ee4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3697c9672a50903bf381fe097bc80913031101f10a9832dd057ba5be41978ee4/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4377b50074c12c555f7dd80219a8b159edab09ba6fb89b227a4e218d4492f097/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4377b50074c12c555f7dd80219a8b159edab09ba6fb89b227a4e218d4492f097/userdata/shm major:0 minor:346 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/49fa91383c974839e7c75e8668112795f422c06c53aafa284bd3c532c9c7be31/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/49fa91383c974839e7c75e8668112795f422c06c53aafa284bd3c532c9c7be31/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4bbef179152e161a50c2a59d9194272f6473ebc2b3c2ece917e5743eb50a63f5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4bbef179152e161a50c2a59d9194272f6473ebc2b3c2ece917e5743eb50a63f5/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/54484f3a947b1f4d94c67d7fa4f7025617444bc1dec9c1eca62fe9bf48d2808b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/54484f3a947b1f4d94c67d7fa4f7025617444bc1dec9c1eca62fe9bf48d2808b/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5ba36b4263d1487516ad584e54be8f8ad2bc6b75bb923831ab769b92c1116d47/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5ba36b4263d1487516ad584e54be8f8ad2bc6b75bb923831ab769b92c1116d47/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/63ac8c7636952aa5d2fb015f80cd4690f2cf189a60bb36eb8136f47cbf331941/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/63ac8c7636952aa5d2fb015f80cd4690f2cf189a60bb36eb8136f47cbf331941/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6ae3ed4f2fb47fac082d2e201eb9d84583e7293300d9cf621e865522115ad012/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6ae3ed4f2fb47fac082d2e201eb9d84583e7293300d9cf621e865522115ad012/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7a916b464e271033e5650cc96ca39397503853853c9742d5600a5f956a7d7d47/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7a916b464e271033e5650cc96ca39397503853853c9742d5600a5f956a7d7d47/userdata/shm major:0 minor:204 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7ff97e9cf55bf6cc1b4f494df68854b3713a287490b20ae897e75f2a656355c9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7ff97e9cf55bf6cc1b4f494df68854b3713a287490b20ae897e75f2a656355c9/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/88ff02f45ff7b504af9f584fe1c578af34b1f29ec41a475a5e75cfe4eb18ed50/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/88ff02f45ff7b504af9f584fe1c578af34b1f29ec41a475a5e75cfe4eb18ed50/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a6ab6d91e787fd5d75e21c3a846dd81f99aeeae8c8cdd9c980045c29db5e19d1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a6ab6d91e787fd5d75e21c3a846dd81f99aeeae8c8cdd9c980045c29db5e19d1/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a77fcbed07b128d61f26fdab0b3b9290df54cd8244c91c2a2c52d9c5cd27331c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a77fcbed07b128d61f26fdab0b3b9290df54cd8244c91c2a2c52d9c5cd27331c/userdata/shm major:0 minor:217 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a8e3518e155c194ecccfadb3976945cfcf143446b46944d5ec821842d71d5342/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a8e3518e155c194ecccfadb3976945cfcf143446b46944d5ec821842d71d5342/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ad50c82576c60f43adea717755051e72585691739af4421c5b2c785460d1309e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ad50c82576c60f43adea717755051e72585691739af4421c5b2c785460d1309e/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b2223a9b09494c4f811b61916b2a8f35e16d4f4c5cde2281b3aa270664b436fa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b2223a9b09494c4f811b61916b2a8f35e16d4f4c5cde2281b3aa270664b436fa/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c82f8c77dbea3089c4c6a24196b9841b6d15dcd34e64e9f297e5a0ba52f242f7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c82f8c77dbea3089c4c6a24196b9841b6d15dcd34e64e9f297e5a0ba52f242f7/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cae338536d142cad1185c0ffac4012f88a6180d3352778fbaea998cf592a298e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cae338536d142cad1185c0ffac4012f88a6180d3352778fbaea998cf592a298e/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/de497939fff5cdee5e1dad744e089b36b512add5de3ab717d0729b0f9f64627e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/de497939fff5cdee5e1dad744e089b36b512add5de3ab717d0729b0f9f64627e/userdata/shm major:0 minor:340 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f37006ff7bdb9b2342fb996deec818e8a3e5f535a2cbb2a0fbcd4cd18fd1771c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f37006ff7bdb9b2342fb996deec818e8a3e5f535a2cbb2a0fbcd4cd18fd1771c/userdata/shm major:0 minor:212 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fde6d237ec81277da04848aed98d89c282c896aae61e77ba5ff427fe0e4e99c7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fde6d237ec81277da04848aed98d89c282c896aae61e77ba5ff427fe0e4e99c7/userdata/shm major:0 minor:63 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a66c33f-29bc-4920-9d7a-0c45215514a0/volumes/kubernetes.io~projected/kube-api-access-xnmwf:{mountpoint:/var/lib/kubelet/pods/0a66c33f-29bc-4920-9d7a-0c45215514a0/volumes/kubernetes.io~projected/kube-api-access-xnmwf major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14e2ac70-be2e-4e08-b093-e8a93d6f985a/volumes/kubernetes.io~projected/kube-api-access-p5z8d:{mountpoint:/var/lib/kubelet/pods/14e2ac70-be2e-4e08-b093-e8a93d6f985a/volumes/kubernetes.io~projected/kube-api-access-p5z8d major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14e2ac70-be2e-4e08-b093-e8a93d6f985a/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/14e2ac70-be2e-4e08-b093-e8a93d6f985a/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19871ddf-e177-4442-b592-8f1c7df24aaf/volumes/kubernetes.io~projected/kube-api-access-z96gl:{mountpoint:/var/lib/kubelet/pods/19871ddf-e177-4442-b592-8f1c7df24aaf/volumes/kubernetes.io~projected/kube-api-access-z96gl major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d8dba89-5673-4cd3-a6ac-914765f228d6/volumes/kubernetes.io~projected/kube-api-access-5x5t7:{mountpoint:/var/lib/kubelet/pods/1d8dba89-5673-4cd3-a6ac-914765f228d6/volumes/kubernetes.io~projected/kube-api-access-5x5t7 major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ab1baec-cb71-4942-a246-7920cd44526c/volumes/kubernetes.io~projected/kube-api-access-t9ppn:{mountpoint:/var/lib/kubelet/pods/2ab1baec-cb71-4942-a246-7920cd44526c/volumes/kubernetes.io~projected/kube-api-access-t9ppn major:0 minor:238 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45c60c0c-dcff-421d-a9f2-4dc53e7637c9/volumes/kubernetes.io~projected/kube-api-access-m5xx5:{mountpoint:/var/lib/kubelet/pods/45c60c0c-dcff-421d-a9f2-4dc53e7637c9/volumes/kubernetes.io~projected/kube-api-access-m5xx5 major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45c60c0c-dcff-421d-a9f2-4dc53e7637c9/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/45c60c0c-dcff-421d-a9f2-4dc53e7637c9/volumes/kubernetes.io~secret/serving-cert major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/482aebaa-cb7f-4be2-991a-7fb4cd801c7f/volumes/kubernetes.io~projected/kube-api-access-mm7m6:{mountpoint:/var/lib/kubelet/pods/482aebaa-cb7f-4be2-991a-7fb4cd801c7f/volumes/kubernetes.io~projected/kube-api-access-mm7m6 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/581612ca-5a0b-4426-9d07-ee65f85e0412/volumes/kubernetes.io~projected/kube-api-access-67zjh:{mountpoint:/var/lib/kubelet/pods/581612ca-5a0b-4426-9d07-ee65f85e0412/volumes/kubernetes.io~projected/kube-api-access-67zjh major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/627e3af2-4a0d-4f19-91bc-760af67e01ad/volumes/kubernetes.io~projected/kube-api-access-8bhzm:{mountpoint:/var/lib/kubelet/pods/627e3af2-4a0d-4f19-91bc-760af67e01ad/volumes/kubernetes.io~projected/kube-api-access-8bhzm major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/627e3af2-4a0d-4f19-91bc-760af67e01ad/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/627e3af2-4a0d-4f19-91bc-760af67e01ad/volumes/kubernetes.io~secret/signing-key major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73622c04-1428-4f7a-8faf-8eb17604f681/volumes/kubernetes.io~projected/kube-api-access-hjt9z:{mountpoint:/var/lib/kubelet/pods/73622c04-1428-4f7a-8faf-8eb17604f681/volumes/kubernetes.io~projected/kube-api-access-hjt9z major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8190fea3-4e71-4250-9a0f-3a1d78dd32db/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/8190fea3-4e71-4250-9a0f-3a1d78dd32db/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8190fea3-4e71-4250-9a0f-3a1d78dd32db/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/8190fea3-4e71-4250-9a0f-3a1d78dd32db/volumes/kubernetes.io~empty-dir/tmp major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8190fea3-4e71-4250-9a0f-3a1d78dd32db/volumes/kubernetes.io~projected/kube-api-access-q546p:{mountpoint:/var/lib/kubelet/pods/8190fea3-4e71-4250-9a0f-3a1d78dd32db/volumes/kubernetes.io~projected/kube-api-access-q546p major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/998efdf0-7719-4768-bfd0-3da04d85d108/volumes/kubernetes.io~projected/kube-api-access-qfxfn:{mountpoint:/var/lib/kubelet/pods/998efdf0-7719-4768-bfd0-3da04d85d108/volumes/kubernetes.io~projected/kube-api-access-qfxfn major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9acc0734-3562-41fc-a05a-8c32cbf36443/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/9acc0734-3562-41fc-a05a-8c32cbf36443/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9acc0734-3562-41fc-a05a-8c32cbf36443/volumes/kubernetes.io~projected/kube-api-access-phjdn:{mountpoint:/var/lib/kubelet/pods/9acc0734-3562-41fc-a05a-8c32cbf36443/volumes/kubernetes.io~projected/kube-api-access-phjdn major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9acc0734-3562-41fc-a05a-8c32cbf36443/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/9acc0734-3562-41fc-a05a-8c32cbf36443/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9acc0734-3562-41fc-a05a-8c32cbf36443/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/9acc0734-3562-41fc-a05a-8c32cbf36443/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e212e37-6c34-4b3c-941c-b38621f1f184/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/9e212e37-6c34-4b3c-941c-b38621f1f184/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e212e37-6c34-4b3c-941c-b38621f1f184/volumes/kubernetes.io~projected/kube-api-access-pwpzw:{mountpoint:/var/lib/kubelet/pods/9e212e37-6c34-4b3c-941c-b38621f1f184/volumes/kubernetes.io~projected/kube-api-access-pwpzw major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e212e37-6c34-4b3c-941c-b38621f1f184/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/9e212e37-6c34-4b3c-941c-b38621f1f184/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a22c3f93-43f6-4908-9549-b966fb3c04e5/volumes/kubernetes.io~projected/kube-api-access-dpmsp:{mountpoint:/var/lib/kubelet/pods/a22c3f93-43f6-4908-9549-b966fb3c04e5/volumes/kubernetes.io~projected/kube-api-access-dpmsp major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a22c3f93-43f6-4908-9549-b966fb3c04e5/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/a22c3f93-43f6-4908-9549-b966fb3c04e5/volumes/kubernetes.io~secret/cert major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a972ee28-29fa-4755-adcd-b40d476c22f0/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/a972ee28-29fa-4755-adcd-b40d476c22f0/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbdccd97-7806-4669-a9cb-6aef70599c54/volumes/kubernetes.io~projected/kube-api-access-qlznq:{mountpoint:/var/lib/kubelet/pods/bbdccd97-7806-4669-a9cb-6aef70599c54/volumes/kubernetes.io~projected/kube-api-access-qlznq major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bcbdd4ff-e252-4aee-9418-9b1b05eed574/volumes/kubernetes.io~projected/kube-api-access-z5fh9:{mountpoint:/var/lib/kubelet/pods/bcbdd4ff-e252-4aee-9418-9b1b05eed574/volumes/kubernetes.io~projected/kube-api-access-z5fh9 major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88/volumes/kubernetes.io~projected/kube-api-access-h78fl:{mountpoint:/var/lib/kubelet/pods/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88/volumes/kubernetes.io~projected/kube-api-access-h78fl major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88/volumes/kubernetes.io~secret/default-certificate major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88/volumes/kubernetes.io~secret/stats-auth major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c014e8ac-58ca-4161-a778-ba58b0e84147/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/c014e8ac-58ca-4161-a778-ba58b0e84147/volumes/kubernetes.io~secret/agent-certs major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca0be51d-87a4-462d-9c83-fcfb8bf224e8/volumes/kubernetes.io~projected/kube-api-access-cbfdt:{mountpoint:/var/lib/kubelet/pods/ca0be51d-87a4-462d-9c83-fcfb8bf224e8/volumes/kubernetes.io~projected/kube-api-access-cbfdt major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/caebbba6-44f3-40c1-95ce-b8a5a3f2667b/volumes/kubernetes.io~projected/kube-api-access-wlvp2:{mountpoint:/var/lib/kubelet/pods/caebbba6-44f3-40c1-95ce-b8a5a3f2667b/volumes/kubernetes.io~projected/kube-api-access-wlvp2 major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d28ff750-94d5-4eff-a8ef-58215bfb99e3/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/d28ff750-94d5-4eff-a8ef-58215bfb99e3/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d28ff750-94d5-4eff-a8ef-58215bfb99e3/volumes/kubernetes.io~projected/kube-api-access-dtt7q:{mountpoint:/var/lib/kubelet/pods/d28ff750-94d5-4eff-a8ef-58215bfb99e3/volumes/kubernetes.io~projected/kube-api-access-dtt7q major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d28ff750-94d5-4eff-a8ef-58215bfb99e3/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/d28ff750-94d5-4eff-a8ef-58215bfb99e3/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d28ff750-94d5-4eff-a8ef-58215bfb99e3/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/d28ff750-94d5-4eff-a8ef-58215bfb99e3/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e09a58a7-120e-4ec5-9aba-f15863267643/volumes/kubernetes.io~projected/kube-api-access-7lbzv:{mountpoint:/var/lib/kubelet/pods/e09a58a7-120e-4ec5-9aba-f15863267643/volumes/kubernetes.io~projected/kube-api-access-7lbzv major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e60132c0-948e-4f84-883e-d22564f210e1/volumes/kubernetes.io~projected/kube-api-access-wbhvz:{mountpoint:/var/lib/kubelet/pods/e60132c0-948e-4f84-883e-d22564f210e1/volumes/kubernetes.io~projected/kube-api-access-wbhvz major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eee1338d-e2a0-4a45-8f74-b81eb7d23019/volumes/kubernetes.io~projected/kube-api-access-8b7cm:{mountpoint:/var/lib/kubelet/pods/eee1338d-e2a0-4a45-8f74-b81eb7d23019/volumes/kubernetes.io~projected/kube-api-access-8b7cm major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eee1338d-e2a0-4a45-8f74-b81eb7d23019/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/eee1338d-e2a0-4a45-8f74-b81eb7d23019/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f097ea60-e18f-47f0-b6a0-24e2b3a201d0/volumes/kubernetes.io~projected/kube-api-access-555tt:{mountpoint:/var/lib/kubelet/pods/f097ea60-e18f-47f0-b6a0-24e2b3a201d0/volumes/kubernetes.io~projected/kube-api-access-555tt major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f29ea61e-2a78-4c11-9566-f52497e135ff/volumes/kubernetes.io~projected/kube-api-access-tvjgq:{mountpoint:/var/lib/kubelet/pods/f29ea61e-2a78-4c11-9566-f52497e135ff/volumes/kubernetes.io~projected/kube-api-access-tvjgq major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f29ea61e-2a78-4c11-9566-f52497e135ff/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f29ea61e-2a78-4c11-9566-f52497e135ff/volumes/kubernetes.io~secret/serving-cert major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1/volumes/kubernetes.io~projected/kube-api-access-vkm29:{mountpoint:/var/lib/kubelet/pods/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1/volumes/kubernetes.io~projected/kube-api-access-vkm29 major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fecfe177-e63f-490f-94bc-c307041640d5/volumes/kubernetes.io~projected/kube-api-access-gqgps:{mountpoint:/var/lib/kubelet/pods/fecfe177-e63f-490f-94bc-c307041640d5/volumes/kubernetes.io~projected/kube-api-access-gqgps major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffdeddae-4086-4fb5-b9f8-c8ffcb765925/volumes/kubernetes.io~projected/kube-api-access-fxfvf:{mountpoint:/var/lib/kubelet/pods/ffdeddae-4086-4fb5-b9f8-c8ffcb765925/volumes/kubernetes.io~projected/kube-api-access-fxfvf major:0 minor:201 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/a6c70fc4fa5c7e96481f946ce3fb4db8baa500ab5a19f676684ca6c36c360bc8/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/ab86a31ae6cdbd80a731fab1d3f218bc6e2f41f9cd8e27996049ab44739d358e/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/448bd3e77958564e22b06737eb092dd5b01300af39ac1372ca495d06a12ff5ca/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/20312e214c1e32b87bcee5d3816b91a110c714d28e01230faef0d70cf3163dc4/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/499da9284796dc8e0a10e7475420963d3148395d09c0060694dbaafb8cf384d0/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/29a8ed18adf9fcaef097d6f853d7a9d182666917e514dede27b796a5da5ab427/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/2ed620000588d9b3deb073d54789a18304877d01ec98513a2bbc5681c7e02761/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/55fabeb8854b479fcde97c203619698215b352dad6d229ea8ecb2a8f3d1f7e3d/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/8b9b580f5151efe123a1522d410696ab60e818e009593e46ec8a401fa1989a22/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/651dfcae045a80cc33ade13e682a825e50d551169441aea2f711b1179d8c4772/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/b5817691c5b95acc22969f4618b072feb62b2ca427ed0d5accbb82c15144da7d/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/d0cab2211c9552c5a74e0ae0800a8b91b774d60b6d0c87797cf2ee1d12faddf1/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/6ba4cba8165c5d0d6f1409203e8c438e0b7eab861e3bebba54695cfcc07a2ce8/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/a04ed947dcc2289dde31d8a204d150bad288ecf253af61142ecac4b8f94e1dca/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/6f8fb92ee9194decaf639f316cdb958c5368915e70a9e40c2f5ee84c99f32208/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/52af082ec300e0bec315dba6f4876681b66e9cc797e3e31108cc062b1e097416/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/e49056b1b846c84de004d25c4f85aa7f61e03ab7bb55d0a53bf58c88c51d712c/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/3b9c4a38b1e04f7bf2a2f0e1e2eae08fb75b1edacaf1a7e0bf7f9276024edf68/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/4bc47a9b8b0901e5a1dd90bd4d61ae0c2b034e351a347a729859f5a4d4b408c6/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/c0afbfdafdb520be280984bd7740c4a014b472a663bf4fa718ac1cd926c81854/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/f5a9f99946dc028fbceaf31fbb1803e36d5654249b0edebd5839b4fda3d390bd/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/a93a1b56864799d98cd995bbed74ff11532363e9659082b2cc9ccb4512cddb52/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/44e6bc0f13fca93c3f1a84cb97b1b4c4696b74a348d4776302051b5dd2c42277/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/8bd76b5747ac14996ce4527083188a917299e2c685e3b33dad72e3c2202adef7/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/56bc485d3826ddc0f4592d9f3864f629a29efaf5ad7e41e82f7c588b676a645d/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/1eca5fe3be414f689ebe26114d669e582ddb9cbe17f7e34ed78b191205c290f2/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/33bdcb05bd15e090062998bcc5e81530ac324071f4e76dc093258dc5d65d10d6/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/6a488e5af62ec52c12f25809ca8b2f6f2f1551b0b1ca75bad8210ecb1f149e01/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/a68d8d25b1d7a0882412fe6df9cb64c162d2cac16d1e6988603068f32acfb676/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/615a501491668ed4b7c71b9356a4b4ef5fc69f89fe136f30cf1174c794434829/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/5030ca4fd5c036e8ed3c707b09670f6c9488504c9ffa0c25f4cf8a56dc35b0a4/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/56d77392e332c6fd2bf89ce12540b4a80605a507069d7683c28ee97043ee057c/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/ea5cb600f5973c8c84d9949c980eaf3b123eb28895ee2f1723145027379a8ac4/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/7168b72a04033f25f19050926eccf8c2db576b8f97617feab57132f420b021f2/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/68793f20387757453d25b6ca08275769dd6d2396bf5ee8ba9ea98d79789619a0/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/558d28c32be2efa5fa350d7684a7896e60e9bd7fb0e108d1ed6880e389dccaba/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/75d1331e14423b3a0803ac297fb23d92b5ff59a1b4b123aa0b3d24a4ed0f5196/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/be561a97001b2a1281509268af5713414b42aa87cc35253558d16401060a49e2/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/e6d0b588dad38daeaa35ab36a8cd4db6f034f5cb992a4764febc71cf73160058/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-311:{mountpoint:/var/lib/containers/storage/overlay/d7a0ceae90fdb0473fa2c460939fa339a34f89c0843d98a3d74dd4c71bf34c19/merged major:0 minor:311 fsType:overlay blockSize:0} overlay_0-313:{mountpoint:/var/lib/containers/storage/overlay/89b7a2879573ec7822dcf499ea45d2684a8f9cfb55925edfa180c94b1517482c/merged major:0 minor:313 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/c05ca1f2633f8752061528f7daf73b62c1318f5e599eed9456ce298931f59388/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/a1fb788232134f3f099152fdf39caa3f2bdd5fec27ef94c19760f555c20a2029/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/758d94257edf3a3f89ff054b4c389fb4d6b103496b463e5b1bffb59906141a59/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/e401862993fb6a86ada48ee1d16451cc70eeacc131b491889a40527da33c38aa/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-46:{mountpoint:/var/lib/containers/storage/overlay/b0f4ce8256f2ed8089e40288cfd99352fefccaebf504c14f8186ab3d40e65e32/merged major:0 minor:46 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/3b4c7296423bde138d116cd5df8f6099c1cc72308f4ae03c189246ceea17cef9/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/9d45981044f9d094be0251b2bece248335c463e99ab11123e1c71bfff4285fa4/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/7c62907756157557d437c3e224b1fcb0961a1a97132a2396e53759ad1212a338/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/6176f46b2260d3c75237a6b664aecdd046e951fd043fcf1b75931e524fbc2dee/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/a8aa1811f7fb496ac5bc0d7f1cf827e7d0b78a04389fe0a06100a4f2e89cdc42/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/d9a790f7da0f9e6682553baad995b072b6999f66f05180234b252b27e67bf21f/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/86069c9efff50610874284bc8928749eb62c8decb10e4045f3b914a7521ed5df/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/94537cd9745ff90cdca725d96aa2f553dbf363587808254a67c0ce8f1a9a8024/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/f62ae7602ca02dbead476f606a873fbffd4442bf0aac6d058ef51c600803efe8/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/e6e8a2b212efb842f943f9531c50e0fd8ce5d976c6108e2afcc5ede10bacc161/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 01 03:06:35.558667 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.556987    4930 manager.go:217] Machine: {Timestamp:2026-11-01 03:06:35.556503225 +0000 UTC m=+0.079509804 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099431 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b890a10dee2b78d4631eafe98a290 SystemUUID:ec2b890a-10de-e2b7-8d46-31eafe98a290 BootID:5c802453-d661-409f-b139-7fbd23e8d7cf Filesystems:[{Device:/var/lib/kubelet/pods/ffdeddae-4086-4fb5-b9f8-c8ffcb765925/volumes/kubernetes.io~projected/kube-api-access-fxfvf DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/49fa91383c974839e7c75e8668112795f422c06c53aafa284bd3c532c9c7be31/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bcbdd4ff-e252-4aee-9418-9b1b05eed574/volumes/kubernetes.io~projected/kube-api-access-z5fh9 DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/19871ddf-e177-4442-b592-8f1c7df24aaf/volumes/kubernetes.io~projected/kube-api-access-z96gl DeviceMajor:0 DeviceMinor:237 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-313 DeviceMajor:0 DeviceMinor:313 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/88ff02f45ff7b504af9f584fe1c578af34b1f29ec41a475a5e75cfe4eb18ed50/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a22c3f93-43f6-4908-9549-b966fb3c04e5/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:236 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e09a58a7-120e-4ec5-9aba-f15863267643/volumes/kubernetes.io~projected/kube-api-access-7lbzv DeviceMajor:0 DeviceMinor:287 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-311 DeviceMajor:0 DeviceMinor:311 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4bbef179152e161a50c2a59d9194272f6473ebc2b3c2ece917e5743eb50a63f5/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9acc0734-3562-41fc-a05a-8c32cbf36443/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9e212e37-6c34-4b3c-941c-b38621f1f184/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:47 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8190fea3-4e71-4250-9a0f-3a1d78dd32db/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a972ee28-29fa-4755-adcd-b40d476c22f0/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:286 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/482aebaa-cb7f-4be2-991a-7fb4cd801c7f/volumes/kubernetes.io~projected/kube-api-access-mm7m6 DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/eee1338d-e2a0-4a45-8f74-b81eb7d23019/volumes/kubernetes.io~projected/kube-api-access-8b7cm DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d28ff750-94d5-4eff-a8ef-58215bfb99e3/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:234 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ca0be51d-87a4-462d-9c83-fcfb8bf224e8/volumes/kubernetes.io~projected/kube-api-access-cbfdt DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eee1338d-e2a0-4a45-8f74-b81eb7d23019/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f29ea61e-2a78-4c11-9566-f52497e135ff/volumes/kubernetes.io~projected/kube-api-access-tvjgq DeviceMajor:0 DeviceMinor:231 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/18b2e76dfc01da8daf9dbf0daef2c0616a2343e95f4faea38329ce5563c0fbe4/userdata/shm DeviceMajor:0 DeviceMinor:259 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/de497939fff5cdee5e1dad744e089b36b512add5de3ab717d0729b0f9f64627e/userdata/shm DeviceMajor:0 DeviceMinor:340 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d28ff750-94d5-4eff-a8ef-58215bfb99e3/volumes/kubernetes.io~projected/kube-api-access-dtt7q DeviceMajor:0 DeviceMinor:230 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f097ea60-e18f-47f0-b6a0-24e2b3a201d0/volumes/kubernetes.io~projected/kube-api-access-555tt DeviceMajor:0 DeviceMinor:327 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c82f8c77dbea3089c4c6a24196b9841b6d15dcd34e64e9f297e5a0ba52f242f7/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5ba36b4263d1487516ad584e54be8f8ad2bc6b75bb923831ab769b92c1116d47/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/45c60c0c-dcff-421d-a9f2-4dc53e7637c9/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/d28ff750-94d5-4eff-a8ef-58215bfb99e3/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bbdccd97-7806-4669-a9cb-6aef70599c54/volumes/kubernetes.io~projected/kube-api-access-qlznq DeviceMajor:0 DeviceMinor:233 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/12af2981349a6df9f1a2182daa931d656f31626994e1b49fb30a715a93b95b48/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9e212e37-6c34-4b3c-941c-b38621f1f184/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/14e2ac70-be2e-4e08-b093-e8a93d6f985a/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/14e2ac70-be2e-4e08-b093-e8a93d6f985a/volumes/kubernetes.io~projected/kube-api-access-p5z8d DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f37006ff7bdb9b2342fb996deec818e8a3e5f535a2cbb2a0fbcd4cd18fd1771c/userdata/shm DeviceMajor:0 DeviceMinor:212 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9acc0734-3562-41fc-a05a-8c32cbf36443/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:229 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2ab1baec-cb71-4942-a246-7920cd44526c/volumes/kubernetes.io~projected/kube-api-access-t9ppn DeviceMajor:0 DeviceMinor:238 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/998efdf0-7719-4768-bfd0-3da04d85d108/volumes/kubernetes.io~projected/kube-api-access-qfxfn DeviceMajor:0 DeviceMinor:52 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9e212e37-6c34-4b3c-941c-b38621f1f184/volumes/kubernetes.io~projected/kube-api-access-pwpzw DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a8e3518e155c194ecccfadb3976945cfcf143446b46944d5ec821842d71d5342/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cae338536d142cad1185c0ffac4012f88a6180d3352778fbaea998cf592a298e/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1/volumes/kubernetes.io~projected/kube-api-access-vkm29 DeviceMajor:0 DeviceMinor:54 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7ff97e9cf55bf6cc1b4f494df68854b3713a287490b20ae897e75f2a656355c9/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0a66c33f-29bc-4920-9d7a-0c45215514a0/volumes/kubernetes.io~projected/kube-api-access-xnmwf DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ad50c82576c60f43adea717755051e72585691739af4421c5b2c785460d1309e/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/45c60c0c-dcff-421d-a9f2-4dc53e7637c9/volumes/kubernetes.io~projected/kube-api-access-m5xx5 DeviceMajor:0 DeviceMinor:232 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9acc0734-3562-41fc-a05a-8c32cbf36443/volumes/kubernetes.io~projected/kube-api-access-phjdn DeviceMajor:0 DeviceMinor:235 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7a916b464e271033e5650cc96ca39397503853853c9742d5600a5f956a7d7d47/userdata/shm DeviceMajor:0 DeviceMinor:204 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88/volumes/kubernetes.io~projected/kube-api-access-h78fl DeviceMajor:0 DeviceMinor:228 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a77fcbed07b128d61f26fdab0b3b9290df54cd8244c91c2a2c52d9c5cd27331c/userdata/shm DeviceMajor:0 DeviceMinor:217 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/c014e8ac-58ca-4161-a778-ba58b0e84147/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:53 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/54484f3a947b1f4d94c67d7fa4f7025617444bc1dec9c1eca62fe9bf48d2808b/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6ae3ed4f2fb47fac082d2e201eb9d84583e7293300d9cf621e865522115ad012/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-46 DeviceMajor:0 DeviceMinor:46 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fde6d237ec81277da04848aed98d89c282c896aae61e77ba5ff427fe0e4e99c7/userdata/shm DeviceMajor:0 DeviceMinor:63 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0b7cef567f173ef3b6d0baafd5015f8074a46e4fb248df070231b3edd177ed2f/userdata/shm DeviceMajor:0 DeviceMinor:289 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/627e3af2-4a0d-4f19-91bc-760af67e01ad/volumes/kubernetes.io~projected/kube-api-access-8bhzm DeviceMajor:0 DeviceMinor:345 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4377b50074c12c555f7dd80219a8b159edab09ba6fb89b227a4e218d4492f097/userdata/shm DeviceMajor:0 DeviceMinor:346 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d28ff750-94d5-4eff-a8ef-58215bfb99e3/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a22c3f93-43f6-4908-9549-b966fb3c04e5/volumes/kubernetes.io~projected/kube-api-access-dpmsp DeviceMajor:0 DeviceMinor:239 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/caebbba6-44f3-40c1-95ce-b8a5a3f2667b/volumes/kubernetes.io~projected/kube-api-access-wlvp2 DeviceMajor:0 DeviceMinor:304 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fecfe177-e63f-490f-94bc-c307041640d5/volumes/kubernetes.io~projected/kube-api-access-gqgps DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/627e3af2-4a0d-4f19-91bc-760af67e01ad/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:344 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a6ab6d91e787fd5d75e21c3a846dd81f99aeeae8c8cdd9c980045c29db5e19d1/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/581612ca-5a0b-4426-9d07-ee65f85e0412/volumes/kubernetes.io~projected/kube-api-access-67zjh DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/63ac8c7636952aa5d2fb015f80cd4690f2cf189a60bb36eb8136f47cbf331941/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8190fea3-4e71-4250-9a0f-3a1d78dd32db/volumes/kubernetes.io~projected/kube-api-access-q546p DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/73622c04-1428-4f7a-8faf-8eb17604f681/volumes/kubernetes.io~projected/kube-api-access-hjt9z DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b2223a9b09494c4f811b61916b2a8f35e16d4f4c5cde2281b3aa270664b436fa/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8190fea3-4e71-4250-9a0f-3a1d78dd32db/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:48 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9acc0734-3562-41fc-a05a-8c32cbf36443/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3697c9672a50903bf381fe097bc80913031101f10a9832dd057ba5be41978ee4/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e60132c0-948e-4f84-883e-d22564f210e1/volumes/kubernetes.io~projected/kube-api-access-wbhvz DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f29ea61e-2a78-4c11-9566-f52497e135ff/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1d8dba89-5673-4cd3-a6ac-914765f228d6/volumes/kubernetes.io~projected/kube-api-access-5x5t7 DeviceMajor:0 DeviceMinor:50 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0b7cef567f173ef MacAddress:3a:5e:c0:76:5e:ba Speed:10000 Mtu:8901} {Name:12af2981349a6df MacAddress:7a:7d:5e:ad:e7:81 Speed:10000 Mtu:8901} {Name:18b2e76dfc01da8 MacAddress:0e:84:09:b8:8a:14 Speed:10000 Mtu:8901} {Name:3697c9672a50903 MacAddress:46:b2:a9:1c:ac:9e Speed:10000 Mtu:8901} {Name:4377b50074c12c5 MacAddress:5e:2a:e5:f1:dc:ba Speed:10000 Mtu:8901} {Name:4bbef179152e161 MacAddress:fe:05:1d:2a:bf:75 Speed:10000 Mtu:8901} {Name:63ac8c7636952aa MacAddress:02:38:b5:7c:fc:f8 Speed:10000 Mtu:8901} {Name:6ae3ed4f2fb47fa MacAddress:16:1e:04:6b:fd:ca Speed:10000 Mtu:8901} {Name:7a916b464e27103 MacAddress:96:95:82:ec:a6:d3 Speed:10000 Mtu:8901} {Name:7ff97e9cf55bf6c MacAddress:86:fb:91:f4:b6:3a Speed:10000 Mtu:8901} {Name:a77fcbed07b128d MacAddress:02:4e:a0:fb:00:68 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:7e:47:ef:38:99 Speed:0 Mtu:9001} {Name:br-int MacAddress:86:e0:f5:70:55:b8 Speed:0 Mtu:8901} {Name:cae338536d142ca MacAddress:6e:af:c0:c5:79:70 Speed:10000 Mtu:8901} {Name:de497939fff5cde MacAddress:5e:c5:fe:f6:65:fd Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:7e:47:ef:38:99 Speed:0 Mtu:9001} {Name:f37006ff7bdb9b2 MacAddress:32:1a:ed:4c:db:01 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:4a:93:ba:ca:17:54 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:3a:1b:7c:92:45:95 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 03:06:35.558667 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.557319    4930 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 03:06:35.558667 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.557359    4930 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 03:06:35.558667 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.557650    4930 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 03:06:35.558667 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.557670    4930 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-122.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 03:06:35.558667 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.557791    4930 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 03:06:35.558667 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.557799    4930 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 03:06:35.558667 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.557811    4930 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 03:06:35.558667 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.557830    4930 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 03:06:35.558667 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.557848    4930 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 03:06:35.558667 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.558009    4930 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 03:06:35.558667 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.558115    4930 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 03:06:35.558667 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.558149    4930 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 03:06:35.558667 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.558163    4930 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 03:06:35.558667 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.558173    4930 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 03:06:35.558667 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.558181    4930 kubelet.go:405] "Adding apiserver pod source"
Nov 01 03:06:35.558667 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.558189    4930 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 03:06:35.559500 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.559305    4930 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 03:06:35.559500 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.559316    4930 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 03:06:35.559745 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.559724    4930 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 03:06:35.571982 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.571961    4930 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 03:06:35.571982 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.571977    4930 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 03:06:35.572095 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.572071    4930 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 01 03:06:35.572261 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.572249    4930 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 03:06:35.572294 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.572279    4930 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 03:06:35.572377 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.572368    4930 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 03:06:35.572405 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.572382    4930 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 03:06:35.572405 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.572389    4930 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 03:06:35.572405 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.572398    4930 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 03:06:35.572477 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.572408    4930 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 03:06:35.572477 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.572415    4930 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 03:06:35.572477 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.572420    4930 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 03:06:35.572477 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.572426    4930 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 03:06:35.572477 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.572432    4930 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 03:06:35.572477 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.572438    4930 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 03:06:35.572477 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.572453    4930 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 03:06:35.572477 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.572463    4930 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 03:06:35.572716 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.572490    4930 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 03:06:35.572716 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.572500    4930 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 03:06:35.575979 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.575967    4930 server.go:1263] "Started kubelet"
Nov 01 03:06:35.576065 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.576036    4930 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 03:06:35.576119 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.576061    4930 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 03:06:35.576172 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.576138    4930 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 03:06:35.576569 ip-10-0-141-122 systemd[1]: Started Kubernetes Kubelet.
Nov 01 03:06:35.576700 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.576578    4930 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 03:06:35.577292 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.577242    4930 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 03:06:35.581993 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.581795    4930 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 03:06:35.581993 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.581813    4930 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 03:06:35.581993 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.581850    4930 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 03:00:55 +0000 UTC" deadline="2028-07-30 10:01:52.799088692 +0000 UTC"
Nov 01 03:06:35.581993 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.581887    4930 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15294h55m17.217205311s"
Nov 01 03:06:35.581993 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.581901    4930 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 03:06:35.581993 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.581911    4930 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 03:06:35.581993 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:35.581924    4930 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-122.us-west-1.compute.internal\" not found"
Nov 01 03:06:35.582360 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.582244    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jh955_f29ea61e-2a78-4c11-9566-f52497e135ff/console-operator/1.log"
Nov 01 03:06:35.583512 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.582638    4930 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 03:06:35.583512 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.583318    4930 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 03:06:35.583512 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.583333    4930 factory.go:55] Registering systemd factory
Nov 01 03:06:35.583512 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.583344    4930 factory.go:223] Registration of the systemd container factory successfully
Nov 01 03:06:35.586722 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.583874    4930 factory.go:153] Registering CRI-O factory
Nov 01 03:06:35.586722 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.583901    4930 factory.go:223] Registration of the crio container factory successfully
Nov 01 03:06:35.586722 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.583925    4930 factory.go:103] Registering Raw factory
Nov 01 03:06:35.586722 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.583946    4930 manager.go:1199] Started watching for new ooms in manager
Nov 01 03:06:35.586722 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.584313    4930 manager.go:319] Starting recovery of all containers
Nov 01 03:06:35.587316 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.587154    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e212e37-6c34-4b3c-941c-b38621f1f184" volumeName="kubernetes.io/secret/9e212e37-6c34-4b3c-941c-b38621f1f184-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 01 03:06:35.587602 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.587534    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d28ff750-94d5-4eff-a8ef-58215bfb99e3" volumeName="kubernetes.io/empty-dir/d28ff750-94d5-4eff-a8ef-58215bfb99e3-ca-trust-extracted" seLinuxMountContext=""
Nov 01 03:06:35.587674 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.587594    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d8dba89-5673-4cd3-a6ac-914765f228d6" volumeName="kubernetes.io/configmap/1d8dba89-5673-4cd3-a6ac-914765f228d6-cni-binary-copy" seLinuxMountContext=""
Nov 01 03:06:35.587674 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.587630    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e212e37-6c34-4b3c-941c-b38621f1f184" volumeName="kubernetes.io/configmap/9e212e37-6c34-4b3c-941c-b38621f1f184-env-overrides" seLinuxMountContext=""
Nov 01 03:06:35.587674 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.587644    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e212e37-6c34-4b3c-941c-b38621f1f184" volumeName="kubernetes.io/configmap/9e212e37-6c34-4b3c-941c-b38621f1f184-ovnkube-script-lib" seLinuxMountContext=""
Nov 01 03:06:35.587674 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.587663    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88" volumeName="kubernetes.io/secret/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-stats-auth" seLinuxMountContext=""
Nov 01 03:06:35.587818 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.587676    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eee1338d-e2a0-4a45-8f74-b81eb7d23019" volumeName="kubernetes.io/secret/eee1338d-e2a0-4a45-8f74-b81eb7d23019-serving-cert" seLinuxMountContext=""
Nov 01 03:06:35.588344 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588320    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f29ea61e-2a78-4c11-9566-f52497e135ff" volumeName="kubernetes.io/configmap/f29ea61e-2a78-4c11-9566-f52497e135ff-config" seLinuxMountContext=""
Nov 01 03:06:35.588434 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588353    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f308b2b4-03e5-4d52-97ff-f1eb3ff951b1" volumeName="kubernetes.io/projected/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-kube-api-access-vkm29" seLinuxMountContext=""
Nov 01 03:06:35.588434 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588366    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a66c33f-29bc-4920-9d7a-0c45215514a0" volumeName="kubernetes.io/projected/0a66c33f-29bc-4920-9d7a-0c45215514a0-kube-api-access-xnmwf" seLinuxMountContext=""
Nov 01 03:06:35.588434 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588377    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19871ddf-e177-4442-b592-8f1c7df24aaf" volumeName="kubernetes.io/configmap/19871ddf-e177-4442-b592-8f1c7df24aaf-config-volume" seLinuxMountContext=""
Nov 01 03:06:35.588434 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588389    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ab1baec-cb71-4942-a246-7920cd44526c" volumeName="kubernetes.io/projected/2ab1baec-cb71-4942-a246-7920cd44526c-kube-api-access-t9ppn" seLinuxMountContext=""
Nov 01 03:06:35.588434 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588396    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73622c04-1428-4f7a-8faf-8eb17604f681" volumeName="kubernetes.io/configmap/73622c04-1428-4f7a-8faf-8eb17604f681-serviceca" seLinuxMountContext=""
Nov 01 03:06:35.588434 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588406    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88" volumeName="kubernetes.io/projected/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-kube-api-access-h78fl" seLinuxMountContext=""
Nov 01 03:06:35.588434 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588422    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fecfe177-e63f-490f-94bc-c307041640d5" volumeName="kubernetes.io/configmap/fecfe177-e63f-490f-94bc-c307041640d5-telemetry-config" seLinuxMountContext=""
Nov 01 03:06:35.588434 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588435    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d8dba89-5673-4cd3-a6ac-914765f228d6" volumeName="kubernetes.io/projected/1d8dba89-5673-4cd3-a6ac-914765f228d6-kube-api-access-5x5t7" seLinuxMountContext=""
Nov 01 03:06:35.588930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588450    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45c60c0c-dcff-421d-a9f2-4dc53e7637c9" volumeName="kubernetes.io/projected/45c60c0c-dcff-421d-a9f2-4dc53e7637c9-kube-api-access-m5xx5" seLinuxMountContext=""
Nov 01 03:06:35.588930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588462    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7dbf71ad-8399-4a40-87d7-98746e830fa9" volumeName="kubernetes.io/configmap/7dbf71ad-8399-4a40-87d7-98746e830fa9-nginx-conf" seLinuxMountContext=""
Nov 01 03:06:35.588930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588473    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88" volumeName="kubernetes.io/secret/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-default-certificate" seLinuxMountContext=""
Nov 01 03:06:35.588930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588483    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d28ff750-94d5-4eff-a8ef-58215bfb99e3" volumeName="kubernetes.io/configmap/d28ff750-94d5-4eff-a8ef-58215bfb99e3-trusted-ca" seLinuxMountContext=""
Nov 01 03:06:35.588930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588525    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eee1338d-e2a0-4a45-8f74-b81eb7d23019" volumeName="kubernetes.io/empty-dir/eee1338d-e2a0-4a45-8f74-b81eb7d23019-tmp" seLinuxMountContext=""
Nov 01 03:06:35.588930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588539    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19871ddf-e177-4442-b592-8f1c7df24aaf" volumeName="kubernetes.io/empty-dir/19871ddf-e177-4442-b592-8f1c7df24aaf-tmp-dir" seLinuxMountContext=""
Nov 01 03:06:35.588930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588550    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45c60c0c-dcff-421d-a9f2-4dc53e7637c9" volumeName="kubernetes.io/secret/45c60c0c-dcff-421d-a9f2-4dc53e7637c9-serving-cert" seLinuxMountContext=""
Nov 01 03:06:35.588930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588589    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="482aebaa-cb7f-4be2-991a-7fb4cd801c7f" volumeName="kubernetes.io/empty-dir/482aebaa-cb7f-4be2-991a-7fb4cd801c7f-tmp-dir" seLinuxMountContext=""
Nov 01 03:06:35.588930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588601    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="482aebaa-cb7f-4be2-991a-7fb4cd801c7f" volumeName="kubernetes.io/projected/482aebaa-cb7f-4be2-991a-7fb4cd801c7f-kube-api-access-mm7m6" seLinuxMountContext=""
Nov 01 03:06:35.588930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588613    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="581612ca-5a0b-4426-9d07-ee65f85e0412" volumeName="kubernetes.io/projected/581612ca-5a0b-4426-9d07-ee65f85e0412-kube-api-access-67zjh" seLinuxMountContext=""
Nov 01 03:06:35.588930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588623    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca0be51d-87a4-462d-9c83-fcfb8bf224e8" volumeName="kubernetes.io/projected/ca0be51d-87a4-462d-9c83-fcfb8bf224e8-kube-api-access-cbfdt" seLinuxMountContext=""
Nov 01 03:06:35.588930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588636    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d28ff750-94d5-4eff-a8ef-58215bfb99e3" volumeName="kubernetes.io/configmap/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-certificates" seLinuxMountContext=""
Nov 01 03:06:35.588930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588670    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d8dba89-5673-4cd3-a6ac-914765f228d6" volumeName="kubernetes.io/configmap/1d8dba89-5673-4cd3-a6ac-914765f228d6-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 01 03:06:35.588930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588683    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eee1338d-e2a0-4a45-8f74-b81eb7d23019" volumeName="kubernetes.io/empty-dir/eee1338d-e2a0-4a45-8f74-b81eb7d23019-snapshots" seLinuxMountContext=""
Nov 01 03:06:35.588930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588693    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fecfe177-e63f-490f-94bc-c307041640d5" volumeName="kubernetes.io/projected/fecfe177-e63f-490f-94bc-c307041640d5-kube-api-access-gqgps" seLinuxMountContext=""
Nov 01 03:06:35.588930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588706    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffdeddae-4086-4fb5-b9f8-c8ffcb765925" volumeName="kubernetes.io/projected/ffdeddae-4086-4fb5-b9f8-c8ffcb765925-kube-api-access-fxfvf" seLinuxMountContext=""
Nov 01 03:06:35.588930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588717    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19871ddf-e177-4442-b592-8f1c7df24aaf" volumeName="kubernetes.io/projected/19871ddf-e177-4442-b592-8f1c7df24aaf-kube-api-access-z96gl" seLinuxMountContext=""
Nov 01 03:06:35.588930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588732    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="998efdf0-7719-4768-bfd0-3da04d85d108" volumeName="kubernetes.io/projected/998efdf0-7719-4768-bfd0-3da04d85d108-kube-api-access-qfxfn" seLinuxMountContext=""
Nov 01 03:06:35.588930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588744    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9acc0734-3562-41fc-a05a-8c32cbf36443" volumeName="kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-bound-sa-token" seLinuxMountContext=""
Nov 01 03:06:35.588930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588754    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbdccd97-7806-4669-a9cb-6aef70599c54" volumeName="kubernetes.io/configmap/bbdccd97-7806-4669-a9cb-6aef70599c54-samples-operator-config" seLinuxMountContext=""
Nov 01 03:06:35.588930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588766    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c014e8ac-58ca-4161-a778-ba58b0e84147" volumeName="kubernetes.io/configmap/c014e8ac-58ca-4161-a778-ba58b0e84147-konnectivity-ca" seLinuxMountContext=""
Nov 01 03:06:35.588930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588777    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca0be51d-87a4-462d-9c83-fcfb8bf224e8" volumeName="kubernetes.io/configmap/ca0be51d-87a4-462d-9c83-fcfb8bf224e8-iptables-alerter-script" seLinuxMountContext=""
Nov 01 03:06:35.588930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588789    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d28ff750-94d5-4eff-a8ef-58215bfb99e3" volumeName="kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-kube-api-access-dtt7q" seLinuxMountContext=""
Nov 01 03:06:35.588930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588800    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d28ff750-94d5-4eff-a8ef-58215bfb99e3" volumeName="kubernetes.io/secret/d28ff750-94d5-4eff-a8ef-58215bfb99e3-installation-pull-secrets" seLinuxMountContext=""
Nov 01 03:06:35.588930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588812    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="627e3af2-4a0d-4f19-91bc-760af67e01ad" volumeName="kubernetes.io/projected/627e3af2-4a0d-4f19-91bc-760af67e01ad-kube-api-access-8bhzm" seLinuxMountContext=""
Nov 01 03:06:35.588930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588823    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8190fea3-4e71-4250-9a0f-3a1d78dd32db" volumeName="kubernetes.io/empty-dir/8190fea3-4e71-4250-9a0f-3a1d78dd32db-etc-tuned" seLinuxMountContext=""
Nov 01 03:06:35.588930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588835    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9acc0734-3562-41fc-a05a-8c32cbf36443" volumeName="kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-kube-api-access-phjdn" seLinuxMountContext=""
Nov 01 03:06:35.588930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588846    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e212e37-6c34-4b3c-941c-b38621f1f184" volumeName="kubernetes.io/projected/9e212e37-6c34-4b3c-941c-b38621f1f184-kube-api-access-pwpzw" seLinuxMountContext=""
Nov 01 03:06:35.588930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588856    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="caebbba6-44f3-40c1-95ce-b8a5a3f2667b" volumeName="kubernetes.io/projected/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-kube-api-access-wlvp2" seLinuxMountContext=""
Nov 01 03:06:35.588930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588870    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eee1338d-e2a0-4a45-8f74-b81eb7d23019" volumeName="kubernetes.io/configmap/eee1338d-e2a0-4a45-8f74-b81eb7d23019-trusted-ca-bundle" seLinuxMountContext=""
Nov 01 03:06:35.588930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588882    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eee1338d-e2a0-4a45-8f74-b81eb7d23019" volumeName="kubernetes.io/configmap/eee1338d-e2a0-4a45-8f74-b81eb7d23019-service-ca-bundle" seLinuxMountContext=""
Nov 01 03:06:35.588930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588893    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eee1338d-e2a0-4a45-8f74-b81eb7d23019" volumeName="kubernetes.io/projected/eee1338d-e2a0-4a45-8f74-b81eb7d23019-kube-api-access-8b7cm" seLinuxMountContext=""
Nov 01 03:06:35.588930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588905    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a22c3f93-43f6-4908-9549-b966fb3c04e5" volumeName="kubernetes.io/projected/a22c3f93-43f6-4908-9549-b966fb3c04e5-kube-api-access-dpmsp" seLinuxMountContext=""
Nov 01 03:06:35.588930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588917    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f29ea61e-2a78-4c11-9566-f52497e135ff" volumeName="kubernetes.io/projected/f29ea61e-2a78-4c11-9566-f52497e135ff-kube-api-access-tvjgq" seLinuxMountContext=""
Nov 01 03:06:35.588930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588930    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f308b2b4-03e5-4d52-97ff-f1eb3ff951b1" volumeName="kubernetes.io/configmap/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-cni-binary-copy" seLinuxMountContext=""
Nov 01 03:06:35.588930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588941    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8190fea3-4e71-4250-9a0f-3a1d78dd32db" volumeName="kubernetes.io/empty-dir/8190fea3-4e71-4250-9a0f-3a1d78dd32db-tmp" seLinuxMountContext=""
Nov 01 03:06:35.588930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588953    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9acc0734-3562-41fc-a05a-8c32cbf36443" volumeName="kubernetes.io/secret/9acc0734-3562-41fc-a05a-8c32cbf36443-installation-pull-secrets" seLinuxMountContext=""
Nov 01 03:06:35.590652 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588965    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c014e8ac-58ca-4161-a778-ba58b0e84147" volumeName="kubernetes.io/secret/c014e8ac-58ca-4161-a778-ba58b0e84147-agent-certs" seLinuxMountContext=""
Nov 01 03:06:35.590652 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588975    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e60132c0-948e-4f84-883e-d22564f210e1" volumeName="kubernetes.io/projected/e60132c0-948e-4f84-883e-d22564f210e1-kube-api-access-wbhvz" seLinuxMountContext=""
Nov 01 03:06:35.590652 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.588986    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f308b2b4-03e5-4d52-97ff-f1eb3ff951b1" volumeName="kubernetes.io/configmap/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-multus-daemon-config" seLinuxMountContext=""
Nov 01 03:06:35.590652 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.589008    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9acc0734-3562-41fc-a05a-8c32cbf36443" volumeName="kubernetes.io/configmap/9acc0734-3562-41fc-a05a-8c32cbf36443-trusted-ca" seLinuxMountContext=""
Nov 01 03:06:35.590652 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.589019    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9acc0734-3562-41fc-a05a-8c32cbf36443" volumeName="kubernetes.io/empty-dir/9acc0734-3562-41fc-a05a-8c32cbf36443-ca-trust-extracted" seLinuxMountContext=""
Nov 01 03:06:35.590652 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.589030    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e212e37-6c34-4b3c-941c-b38621f1f184" volumeName="kubernetes.io/configmap/9e212e37-6c34-4b3c-941c-b38621f1f184-ovnkube-config" seLinuxMountContext=""
Nov 01 03:06:35.590652 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.589040    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a972ee28-29fa-4755-adcd-b40d476c22f0" volumeName="kubernetes.io/secret/a972ee28-29fa-4755-adcd-b40d476c22f0-original-pull-secret" seLinuxMountContext=""
Nov 01 03:06:35.590652 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.589051    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14e2ac70-be2e-4e08-b093-e8a93d6f985a" volumeName="kubernetes.io/configmap/14e2ac70-be2e-4e08-b093-e8a93d6f985a-config" seLinuxMountContext=""
Nov 01 03:06:35.590652 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.589062    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="627e3af2-4a0d-4f19-91bc-760af67e01ad" volumeName="kubernetes.io/configmap/627e3af2-4a0d-4f19-91bc-760af67e01ad-signing-cabundle" seLinuxMountContext=""
Nov 01 03:06:35.590652 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.589073    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e09a58a7-120e-4ec5-9aba-f15863267643" volumeName="kubernetes.io/projected/e09a58a7-120e-4ec5-9aba-f15863267643-kube-api-access-7lbzv" seLinuxMountContext=""
Nov 01 03:06:35.590652 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.589085    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f29ea61e-2a78-4c11-9566-f52497e135ff" volumeName="kubernetes.io/secret/f29ea61e-2a78-4c11-9566-f52497e135ff-serving-cert" seLinuxMountContext=""
Nov 01 03:06:35.590652 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.589097    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14e2ac70-be2e-4e08-b093-e8a93d6f985a" volumeName="kubernetes.io/projected/14e2ac70-be2e-4e08-b093-e8a93d6f985a-kube-api-access-p5z8d" seLinuxMountContext=""
Nov 01 03:06:35.590652 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.589109    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="627e3af2-4a0d-4f19-91bc-760af67e01ad" volumeName="kubernetes.io/secret/627e3af2-4a0d-4f19-91bc-760af67e01ad-signing-key" seLinuxMountContext=""
Nov 01 03:06:35.590652 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.589120    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8190fea3-4e71-4250-9a0f-3a1d78dd32db" volumeName="kubernetes.io/projected/8190fea3-4e71-4250-9a0f-3a1d78dd32db-kube-api-access-q546p" seLinuxMountContext=""
Nov 01 03:06:35.590652 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.589131    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a22c3f93-43f6-4908-9549-b966fb3c04e5" volumeName="kubernetes.io/secret/a22c3f93-43f6-4908-9549-b966fb3c04e5-cert" seLinuxMountContext=""
Nov 01 03:06:35.590652 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.589140    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d28ff750-94d5-4eff-a8ef-58215bfb99e3" volumeName="kubernetes.io/secret/d28ff750-94d5-4eff-a8ef-58215bfb99e3-image-registry-private-configuration" seLinuxMountContext=""
Nov 01 03:06:35.590652 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.589152    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f097ea60-e18f-47f0-b6a0-24e2b3a201d0" volumeName="kubernetes.io/projected/f097ea60-e18f-47f0-b6a0-24e2b3a201d0-kube-api-access-555tt" seLinuxMountContext=""
Nov 01 03:06:35.590652 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.589162    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f29ea61e-2a78-4c11-9566-f52497e135ff" volumeName="kubernetes.io/configmap/f29ea61e-2a78-4c11-9566-f52497e135ff-trusted-ca" seLinuxMountContext=""
Nov 01 03:06:35.590652 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.589174    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14e2ac70-be2e-4e08-b093-e8a93d6f985a" volumeName="kubernetes.io/secret/14e2ac70-be2e-4e08-b093-e8a93d6f985a-serving-cert" seLinuxMountContext=""
Nov 01 03:06:35.590652 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.589184    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d8dba89-5673-4cd3-a6ac-914765f228d6" volumeName="kubernetes.io/configmap/1d8dba89-5673-4cd3-a6ac-914765f228d6-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 01 03:06:35.590652 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.589195    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45c60c0c-dcff-421d-a9f2-4dc53e7637c9" volumeName="kubernetes.io/configmap/45c60c0c-dcff-421d-a9f2-4dc53e7637c9-config" seLinuxMountContext=""
Nov 01 03:06:35.590652 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.589205    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9acc0734-3562-41fc-a05a-8c32cbf36443" volumeName="kubernetes.io/configmap/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-certificates" seLinuxMountContext=""
Nov 01 03:06:35.590652 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.589217    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9acc0734-3562-41fc-a05a-8c32cbf36443" volumeName="kubernetes.io/secret/9acc0734-3562-41fc-a05a-8c32cbf36443-image-registry-private-configuration" seLinuxMountContext=""
Nov 01 03:06:35.590652 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.589228    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbdccd97-7806-4669-a9cb-6aef70599c54" volumeName="kubernetes.io/projected/bbdccd97-7806-4669-a9cb-6aef70599c54-kube-api-access-qlznq" seLinuxMountContext=""
Nov 01 03:06:35.590652 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.589251    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="caebbba6-44f3-40c1-95ce-b8a5a3f2667b" volumeName="kubernetes.io/empty-dir/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-data-volume" seLinuxMountContext=""
Nov 01 03:06:35.590652 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.589262    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73622c04-1428-4f7a-8faf-8eb17604f681" volumeName="kubernetes.io/projected/73622c04-1428-4f7a-8faf-8eb17604f681-kube-api-access-hjt9z" seLinuxMountContext=""
Nov 01 03:06:35.590652 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.589273    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcbdd4ff-e252-4aee-9418-9b1b05eed574" volumeName="kubernetes.io/projected/bcbdd4ff-e252-4aee-9418-9b1b05eed574-kube-api-access-z5fh9" seLinuxMountContext=""
Nov 01 03:06:35.590652 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.589287    4930 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d28ff750-94d5-4eff-a8ef-58215bfb99e3" volumeName="kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-bound-sa-token" seLinuxMountContext=""
Nov 01 03:06:35.590652 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.589298    4930 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 03:06:35.590652 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.589319    4930 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 03:06:35.597732 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.597714    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jh955_f29ea61e-2a78-4c11-9566-f52497e135ff/console-operator/1.log"
Nov 01 03:06:35.612790 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.610677    4930 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 03:06:35.612790 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.612172    4930 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 03:06:35.612790 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.612194    4930 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 03:06:35.612790 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.612230    4930 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 03:06:35.612790 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:35.612270    4930 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 01 03:06:35.654466 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.654430    4930 generic.go:356] "Generic (PLEG): container finished" podID="1d8dba89-5673-4cd3-a6ac-914765f228d6" containerID="d25851b3aff82b1af421546f23051319145235925704d99408a0fd13c6d4349a" exitCode=0
Nov 01 03:06:35.654466 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.654462    4930 generic.go:356] "Generic (PLEG): container finished" podID="1d8dba89-5673-4cd3-a6ac-914765f228d6" containerID="945ab6571529e03188314b47ba85dc7c4207fb40514db7d5b7d7674bddf16841" exitCode=0
Nov 01 03:06:35.654646 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.654471    4930 generic.go:356] "Generic (PLEG): container finished" podID="1d8dba89-5673-4cd3-a6ac-914765f228d6" containerID="bc59703f6577f9703c11bd045736d591a4fff02da699cf4adea2a08b3130dc85" exitCode=0
Nov 01 03:06:35.654646 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.654480    4930 generic.go:356] "Generic (PLEG): container finished" podID="1d8dba89-5673-4cd3-a6ac-914765f228d6" containerID="423f3551e2412f97df3d5cc444e868096bf40e1be43578ff42f38b3d359477dc" exitCode=0
Nov 01 03:06:35.654646 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.654488    4930 generic.go:356] "Generic (PLEG): container finished" podID="1d8dba89-5673-4cd3-a6ac-914765f228d6" containerID="f2b517a8c456aa1b611d12ea901c5cdef22c2bed4f1e306b8e36ce6a1a6c1936" exitCode=0
Nov 01 03:06:35.654646 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.654496    4930 generic.go:356] "Generic (PLEG): container finished" podID="1d8dba89-5673-4cd3-a6ac-914765f228d6" containerID="4c9b0734d980feadfe29210ae9fea30b82dc90202bdee5a0602b62d54ada4365" exitCode=0
Nov 01 03:06:35.658674 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.658649    4930 generic.go:356] "Generic (PLEG): container finished" podID="bbcea9cdf04d326735591dc246925152" containerID="754ac17bd698683907c39572acbf0deeb9f733504c9436719964fb0dcba06f13" exitCode=0
Nov 01 03:06:35.660317 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.660299    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jh955_f29ea61e-2a78-4c11-9566-f52497e135ff/console-operator/1.log"
Nov 01 03:06:35.660387 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.660324    4930 generic.go:356] "Generic (PLEG): container finished" podID="f29ea61e-2a78-4c11-9566-f52497e135ff" containerID="3d57ddeb233cc6ef9b56e40c0268f52a146af9adc2a471d9dd2158d7b677263c" exitCode=255
Nov 01 03:06:35.675316 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.675300    4930 manager.go:324] Recovery completed
Nov 01 03:06:35.681991 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:35.681974    4930 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-122.us-west-1.compute.internal\" not found"
Nov 01 03:06:35.691322 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.691304    4930 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 03:06:35.692283 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.692266    4930 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 03:06:35.692345 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.692295    4930 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 03:06:35.692345 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.692305    4930 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 03:06:35.695042 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.695023    4930 cpu_manager.go:226] "Starting" policy="none"
Nov 01 03:06:35.695042 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.695039    4930 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 03:06:35.695162 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.695059    4930 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 03:06:35.695297 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.695280    4930 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 01 03:06:35.695343 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.695299    4930 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 01 03:06:35.695343 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.695321    4930 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 01 03:06:35.695343 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.695329    4930 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 01 03:06:35.695343 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.695337    4930 policy_none.go:50] "Start"
Nov 01 03:06:35.695628 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.695357    4930 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 03:06:35.695628 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.695414    4930 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 03:06:35.695628 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.695463    4930 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 03:06:35.695835 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.695717    4930 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 01 03:06:35.695835 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.695729    4930 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 01 03:06:35.695835 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.695744    4930 policy_none.go:44] "Start"
Nov 01 03:06:35.696505 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.696482    4930 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 03:06:35.696599 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.696518    4930 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 03:06:35.696599 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.696534    4930 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 03:06:35.701695 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.701671    4930 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 03:06:35.704656 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.704642    4930 manager.go:341] "Starting Device Plugin manager"
Nov 01 03:06:35.704724 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:35.704667    4930 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 03:06:35.704724 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.704676    4930 server.go:87] "Starting device plugin registration server"
Nov 01 03:06:35.704965 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.704950    4930 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 03:06:35.705011 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.704971    4930 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 03:06:35.705049 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.705038    4930 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 03:06:35.705172 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.705134    4930 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 03:06:35.705279 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.705180    4930 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 03:06:35.705279 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.705187    4930 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 03:06:35.705380 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.705236    4930 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T03:06:35.705165629Z","UUID":"1bbb1981-2933-43db-8503-fd02ccdb7254","Handler":null,"Name":"","Endpoint":""}
Nov 01 03:06:35.706532 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.706513    4930 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 03:06:35.706667 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.706539    4930 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 03:06:35.706667 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:35.706638    4930 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 03:06:35.706746 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:35.706677    4930 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-122.us-west-1.compute.internal\" not found"
Nov 01 03:06:35.712417 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.712385    4930 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-122.us-west-1.compute.internal"]
Nov 01 03:06:35.712511 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.712492    4930 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 03:06:35.713186 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.713171    4930 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 03:06:35.713265 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.713194    4930 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 03:06:35.713265 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.713206    4930 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 03:06:35.713527 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.713513    4930 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 03:06:35.713680 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.713663    4930 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 03:06:35.714227 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.714208    4930 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 03:06:35.714321 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.714235    4930 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 03:06:35.714321 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.714242    4930 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 03:06:35.714321 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.714269    4930 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 03:06:35.714321 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.714279    4930 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 03:06:35.714321 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.714250    4930 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 03:06:35.714543 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:35.714438    4930 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-122.us-west-1.compute.internal\" not found" node="ip-10-0-141-122.us-west-1.compute.internal"
Nov 01 03:06:35.714656 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.714625    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-122.us-west-1.compute.internal" event={"ID":"5391522643ff9514bed3da110ddd8b71","Type":"ContainerStarted","Data":"c68ca34bd87df0550778d481600d8de4ff2ef0b14f4bc9ff408644b5babc1647"}
Nov 01 03:06:35.714706 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.714666    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-122.us-west-1.compute.internal" event={"ID":"5391522643ff9514bed3da110ddd8b71","Type":"ContainerStarted","Data":"a6ab6d91e787fd5d75e21c3a846dd81f99aeeae8c8cdd9c980045c29db5e19d1"}
Nov 01 03:06:35.714742 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.714713    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal" event={"ID":"bbcea9cdf04d326735591dc246925152","Type":"ContainerStarted","Data":"635ad1341241bcdf3d52441407e1e9b11354792e7a1f5a3d402b4a5f631c6561"}
Nov 01 03:06:35.714742 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.714731    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal" event={"ID":"bbcea9cdf04d326735591dc246925152","Type":"ContainerDied","Data":"754ac17bd698683907c39572acbf0deeb9f733504c9436719964fb0dcba06f13"}
Nov 01 03:06:35.714742 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.714668    4930 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 03:06:35.714869 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.714744    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal" event={"ID":"bbcea9cdf04d326735591dc246925152","Type":"ContainerStarted","Data":"b2223a9b09494c4f811b61916b2a8f35e16d4f4c5cde2281b3aa270664b436fa"}
Nov 01 03:06:35.715414 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.715394    4930 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 03:06:35.715498 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.715422    4930 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 03:06:35.715498 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.715435    4930 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 03:06:35.715596 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:35.715587    4930 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-122.us-west-1.compute.internal\" not found" node="ip-10-0-141-122.us-west-1.compute.internal"
Nov 01 03:06:35.774112 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.774063    4930 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 03:06:35.774360 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.774345    4930 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 03:06:35.774459 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.774448    4930 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 01 03:06:35.806054 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.806037    4930 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 03:06:35.806824 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.806808    4930 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 01 03:06:35.806885 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.806836    4930 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 03:06:35.806885 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.806845    4930 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 01 03:06:35.806885 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.806881    4930 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-122.us-west-1.compute.internal"
Nov 01 03:06:35.823104 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.823086    4930 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 03:06:35.825257 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.825243    4930 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 03:06:35.874787 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.874762    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bbcea9cdf04d326735591dc246925152-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal\" (UID: \"bbcea9cdf04d326735591dc246925152\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal"
Nov 01 03:06:35.874787 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.874787    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbcea9cdf04d326735591dc246925152-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal\" (UID: \"bbcea9cdf04d326735591dc246925152\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal"
Nov 01 03:06:35.874907 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.874805    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5391522643ff9514bed3da110ddd8b71-config\") pod \"kube-apiserver-proxy-ip-10-0-141-122.us-west-1.compute.internal\" (UID: \"5391522643ff9514bed3da110ddd8b71\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-122.us-west-1.compute.internal"
Nov 01 03:06:35.882902 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.882883    4930 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-122.us-west-1.compute.internal"
Nov 01 03:06:35.920735 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.920715    4930 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-141-122.us-west-1.compute.internal"
Nov 01 03:06:35.920808 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.920768    4930 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-122.us-west-1.compute.internal"
Nov 01 03:06:35.921227 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.921213    4930 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeNotReady"
Nov 01 03:06:35.921276 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.921241    4930 setters.go:546] "Node became not ready" node="ip-10-0-141-122.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-01T03:06:35Z","lastTransitionTime":"2026-11-01T03:06:35Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Nov 01 03:06:35.939894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.939874    4930 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 03:06:35.939958 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:35.939909    4930 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-122.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-122.us-west-1.compute.internal"
Nov 01 03:06:35.939958 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.939921    4930 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal"
Nov 01 03:06:35.975002 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.974979    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bbcea9cdf04d326735591dc246925152-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal\" (UID: \"bbcea9cdf04d326735591dc246925152\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal"
Nov 01 03:06:35.975095 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.975010    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbcea9cdf04d326735591dc246925152-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal\" (UID: \"bbcea9cdf04d326735591dc246925152\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal"
Nov 01 03:06:35.975095 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.975037    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5391522643ff9514bed3da110ddd8b71-config\") pod \"kube-apiserver-proxy-ip-10-0-141-122.us-west-1.compute.internal\" (UID: \"5391522643ff9514bed3da110ddd8b71\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-122.us-west-1.compute.internal"
Nov 01 03:06:35.975181 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.975092    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5391522643ff9514bed3da110ddd8b71-config\") pod \"kube-apiserver-proxy-ip-10-0-141-122.us-west-1.compute.internal\" (UID: \"5391522643ff9514bed3da110ddd8b71\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-122.us-west-1.compute.internal"
Nov 01 03:06:35.975181 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.975096    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bbcea9cdf04d326735591dc246925152-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal\" (UID: \"bbcea9cdf04d326735591dc246925152\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal"
Nov 01 03:06:35.975181 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.975136    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bbcea9cdf04d326735591dc246925152-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal\" (UID: \"bbcea9cdf04d326735591dc246925152\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal"
Nov 01 03:06:35.979475 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.979461    4930 nodeinfomanager.go:471] existing CSINode "ip-10-0-141-122.us-west-1.compute.internal" is owned by different node (oldNodeID="1b1aaa24-fbda-40d3-97ee-421a6dad1466", newNodeID=""), cleaning up...
Nov 01 03:06:35.999981 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:35.999958    4930 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 03:06:36.000057 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:35.999993    4930 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal"
Nov 01 03:06:36.291291 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.291267    4930 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-122.us-west-1.compute.internal" event="NodeReady"
Nov 01 03:06:36.291492 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.291354    4930 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 01 03:06:36.558615 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.558519    4930 apiserver.go:52] "Watching apiserver"
Nov 01 03:06:36.663374 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.663347    4930 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 03:06:36.664329 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.664298    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2vfgr","openshift-image-registry/node-ca-stpgr","openshift-network-diagnostics/network-check-target-62mxc","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gtk6f","openshift-console-operator/console-operator-7cd56b8dbc-jh955","kube-system/kube-apiserver-proxy-ip-10-0-141-122.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk","openshift-cluster-node-tuning-operator/tuned-l892q","openshift-service-ca/service-ca-5fffc97bdf-wjvnt","openshift-monitoring/cluster-monitoring-operator-884576c57-xbdmd","openshift-ingress-canary/ingress-canary-zdrpv","openshift-network-diagnostics/network-check-source-6f78d59f99-fsr8j","openshift-kube-storage-version-migrator/migrator-66df5d496b-cbbs4","openshift-multus/multus-p5ght","openshift-network-console/networking-console-plugin-86c478bb88-t6tw7","kube-system/kas-connection-checker-685895c7c9-fznsl","openshift-insights/insights-operator-7c4986d449-48ssl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-9nhh6","openshift-multus/network-metrics-daemon-mzkbj","openshift-ovn-kubernetes/ovnkube-node-h5ksq","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chqll","openshift-dns/node-resolver-77nf7","openshift-image-registry/image-registry-6f44999c56-64k9p","openshift-image-registry/image-registry-58bc694f-l7vm9","kube-system/kas-connection-checker-685895c7c9-j6kdp","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-62ws6","kube-system/global-pull-secret-syncer-4bvb6","kube-system/kas-connection-checker-685895c7c9-8plk2","openshift-ingress/router-default-ddbff58bc-5dlt4","openshift-network-operator/iptables-alerter-ksj2n","openshift-dns/dns-default-wps29","openshift-service-ca-operator/service-ca-operator-544d749c8c-cbzc6","kube-system/konnectivity-agent-62ccn"]
Nov 01 03:06:36.665105 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.665070    4930 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-122.us-west-1.compute.internal"
Nov 01 03:06:36.665279 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.665263    4930 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal"
Nov 01 03:06:36.665706 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.665694    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xbdmd"
Nov 01 03:06:36.668027 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.668006    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gtk6f"
Nov 01 03:06:36.668434 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.668417    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:36.668846 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.668827    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:36.669189 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.669174    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-t6tw7"
Nov 01 03:06:36.674376 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.674356    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzkbj"
Nov 01 03:06:36.678238 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.678218    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:36.678833 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.678818    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wps29"
Nov 01 03:06:36.679964 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.679950    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2vfgr"
Nov 01 03:06:36.681018 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.681000    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-wjvnt"
Nov 01 03:06:36.684959 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.684943    4930 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 03:06:36.719010 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.718988    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 01 03:06:36.719104 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.719046    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 01 03:06:36.719104 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.719058    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 01 03:06:36.719104 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.719061    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 01 03:06:36.719357 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.719305    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:36.719462 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.719358    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 01 03:06:36.719462 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.719407    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 01 03:06:36.719462 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.719422    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 01 03:06:36.719462 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.719407    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 01 03:06:36.719462 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.719453    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 01 03:06:36.719731 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.719503    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 01 03:06:36.719731 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.719570    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-tdml4\""
Nov 01 03:06:36.719832 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.719761    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 01 03:06:36.719832 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.719781    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-95fqh\""
Nov 01 03:06:36.719930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.719887    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-n9bgf\""
Nov 01 03:06:36.720104 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.720089    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 01 03:06:36.720836 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.720821    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 01 03:06:36.731374 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.731355    4930 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 03:06:36.731457 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:36.731394    4930 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-122.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-122.us-west-1.compute.internal"
Nov 01 03:06:36.731962 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.731948    4930 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 03:06:36.731998 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:36.731983    4930 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-122.us-west-1.compute.internal"
Nov 01 03:06:36.778711 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.778689    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqgps\" (UniqueName: \"kubernetes.io/projected/fecfe177-e63f-490f-94bc-c307041640d5-kube-api-access-gqgps\") pod \"cluster-monitoring-operator-884576c57-xbdmd\" (UID: \"fecfe177-e63f-490f-94bc-c307041640d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xbdmd"
Nov 01 03:06:36.778789 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.778717    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca0be51d-87a4-462d-9c83-fcfb8bf224e8-host-slash\") pod \"iptables-alerter-ksj2n\" (UID: \"ca0be51d-87a4-462d-9c83-fcfb8bf224e8\") " pod="openshift-network-operator/iptables-alerter-ksj2n"
Nov 01 03:06:36.778789 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.778736    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-bound-sa-token\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:36.778789 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.778753    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxfvf\" (UniqueName: \"kubernetes.io/projected/ffdeddae-4086-4fb5-b9f8-c8ffcb765925-kube-api-access-fxfvf\") pod \"kas-connection-checker-685895c7c9-j6kdp\" (UID: \"ffdeddae-4086-4fb5-b9f8-c8ffcb765925\") " pod="kube-system/kas-connection-checker-685895c7c9-j6kdp"
Nov 01 03:06:36.778927 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.778798    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-host-var-lib-cni-multus\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.778927 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.778829    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkm29\" (UniqueName: \"kubernetes.io/projected/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-kube-api-access-vkm29\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.778927 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.778847    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/19871ddf-e177-4442-b592-8f1c7df24aaf-config-volume\") pod \"dns-default-wps29\" (UID: \"19871ddf-e177-4442-b592-8f1c7df24aaf\") " pod="openshift-dns/dns-default-wps29"
Nov 01 03:06:36.778927 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.778863    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpmsp\" (UniqueName: \"kubernetes.io/projected/a22c3f93-43f6-4908-9549-b966fb3c04e5-kube-api-access-dpmsp\") pod \"ingress-canary-zdrpv\" (UID: \"a22c3f93-43f6-4908-9549-b966fb3c04e5\") " pod="openshift-ingress-canary/ingress-canary-zdrpv"
Nov 01 03:06:36.778927 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.778892    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-service-ca-bundle\") pod \"router-default-ddbff58bc-5dlt4\" (UID: \"be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88\") " pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:36.778927 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.778913    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1d8dba89-5673-4cd3-a6ac-914765f228d6-cnibin\") pod \"multus-additional-cni-plugins-9nhh6\" (UID: \"1d8dba89-5673-4cd3-a6ac-914765f228d6\") " pod="openshift-multus/multus-additional-cni-plugins-9nhh6"
Nov 01 03:06:36.779118 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.778979    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-systemd-units\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.779118 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.779004    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-log-socket\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.779118 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.779019    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwpzw\" (UniqueName: \"kubernetes.io/projected/9e212e37-6c34-4b3c-941c-b38621f1f184-kube-api-access-pwpzw\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.779118 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.779034    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z96gl\" (UniqueName: \"kubernetes.io/projected/19871ddf-e177-4442-b592-8f1c7df24aaf-kube-api-access-z96gl\") pod \"dns-default-wps29\" (UID: \"19871ddf-e177-4442-b592-8f1c7df24aaf\") " pod="openshift-dns/dns-default-wps29"
Nov 01 03:06:36.779118 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.779071    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eee1338d-e2a0-4a45-8f74-b81eb7d23019-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-48ssl\" (UID: \"eee1338d-e2a0-4a45-8f74-b81eb7d23019\") " pod="openshift-insights/insights-operator-7c4986d449-48ssl"
Nov 01 03:06:36.779118 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.779107    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjt9z\" (UniqueName: \"kubernetes.io/projected/73622c04-1428-4f7a-8faf-8eb17604f681-kube-api-access-hjt9z\") pod \"node-ca-stpgr\" (UID: \"73622c04-1428-4f7a-8faf-8eb17604f681\") " pod="openshift-image-registry/node-ca-stpgr"
Nov 01 03:06:36.779349 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.779133    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/482aebaa-cb7f-4be2-991a-7fb4cd801c7f-tmp-dir\") pod \"node-resolver-77nf7\" (UID: \"482aebaa-cb7f-4be2-991a-7fb4cd801c7f\") " pod="openshift-dns/node-resolver-77nf7"
Nov 01 03:06:36.779349 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.779162    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/998efdf0-7719-4768-bfd0-3da04d85d108-socket-dir\") pod \"aws-ebs-csi-driver-node-rjtwk\" (UID: \"998efdf0-7719-4768-bfd0-3da04d85d108\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk"
Nov 01 03:06:36.779349 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.779187    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-var-lib-openvswitch\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.779349 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.779267    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/482aebaa-cb7f-4be2-991a-7fb4cd801c7f-tmp-dir\") pod \"node-resolver-77nf7\" (UID: \"482aebaa-cb7f-4be2-991a-7fb4cd801c7f\") " pod="openshift-dns/node-resolver-77nf7"
Nov 01 03:06:36.779349 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.779294    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-host-var-lib-kubelet\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.779676 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.779639    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-host-run-multus-certs\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.779762 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.779705    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-etc-kubernetes\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:06:36.779812 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.779765    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-run\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:06:36.779858 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.779829    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a972ee28-29fa-4755-adcd-b40d476c22f0-kubelet-config\") pod \"global-pull-secret-syncer-4bvb6\" (UID: \"a972ee28-29fa-4755-adcd-b40d476c22f0\") " pod="kube-system/global-pull-secret-syncer-4bvb6"
Nov 01 03:06:36.779904 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.779876    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-host-cni-bin\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.779965 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.779933    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ca0be51d-87a4-462d-9c83-fcfb8bf224e8-iptables-alerter-script\") pod \"iptables-alerter-ksj2n\" (UID: \"ca0be51d-87a4-462d-9c83-fcfb8bf224e8\") " pod="openshift-network-operator/iptables-alerter-ksj2n"
Nov 01 03:06:36.780078 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.780045    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-sys\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:06:36.780138 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.780084    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtt7q\" (UniqueName: \"kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-kube-api-access-dtt7q\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:36.780138 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.780120    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phjdn\" (UniqueName: \"kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-kube-api-access-phjdn\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:36.780232 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.780153    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7dbf71ad-8399-4a40-87d7-98746e830fa9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-t6tw7\" (UID: \"7dbf71ad-8399-4a40-87d7-98746e830fa9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-t6tw7"
Nov 01 03:06:36.780232 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.780187    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-tls\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:36.780232 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.780215    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9e212e37-6c34-4b3c-941c-b38621f1f184-ovnkube-script-lib\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.780384 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.780251    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-multus-socket-dir-parent\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.780384 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.780288    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-host-run-netns\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.780384 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.780320    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-host-var-lib-cni-bin\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.780384 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.780363    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eee1338d-e2a0-4a45-8f74-b81eb7d23019-tmp\") pod \"insights-operator-7c4986d449-48ssl\" (UID: \"eee1338d-e2a0-4a45-8f74-b81eb7d23019\") " pod="openshift-insights/insights-operator-7c4986d449-48ssl"
Nov 01 03:06:36.780597 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.780391    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eee1338d-e2a0-4a45-8f74-b81eb7d23019-serving-cert\") pod \"insights-operator-7c4986d449-48ssl\" (UID: \"eee1338d-e2a0-4a45-8f74-b81eb7d23019\") " pod="openshift-insights/insights-operator-7c4986d449-48ssl"
Nov 01 03:06:36.780668 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.780650    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eee1338d-e2a0-4a45-8f74-b81eb7d23019-tmp\") pod \"insights-operator-7c4986d449-48ssl\" (UID: \"eee1338d-e2a0-4a45-8f74-b81eb7d23019\") " pod="openshift-insights/insights-operator-7c4986d449-48ssl"
Nov 01 03:06:36.780803 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.780786    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q546p\" (UniqueName: \"kubernetes.io/projected/8190fea3-4e71-4250-9a0f-3a1d78dd32db-kube-api-access-q546p\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:06:36.780896 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.780883    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f29ea61e-2a78-4c11-9566-f52497e135ff-serving-cert\") pod \"console-operator-7cd56b8dbc-jh955\" (UID: \"f29ea61e-2a78-4c11-9566-f52497e135ff\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jh955"
Nov 01 03:06:36.781004 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.780990    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9acc0734-3562-41fc-a05a-8c32cbf36443-installation-pull-secrets\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:36.781138 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.781124    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5xx5\" (UniqueName: \"kubernetes.io/projected/45c60c0c-dcff-421d-a9f2-4dc53e7637c9-kube-api-access-m5xx5\") pod \"service-ca-operator-544d749c8c-cbzc6\" (UID: \"45c60c0c-dcff-421d-a9f2-4dc53e7637c9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbzc6"
Nov 01 03:06:36.781229 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.781216    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9acc0734-3562-41fc-a05a-8c32cbf36443-ca-trust-extracted\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:36.781312 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.781298    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5z8d\" (UniqueName: \"kubernetes.io/projected/14e2ac70-be2e-4e08-b093-e8a93d6f985a-kube-api-access-p5z8d\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-chqll\" (UID: \"14e2ac70-be2e-4e08-b093-e8a93d6f985a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chqll"
Nov 01 03:06:36.781396 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.781384    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d28ff750-94d5-4eff-a8ef-58215bfb99e3-ca-trust-extracted\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:36.781486 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.781475    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c014e8ac-58ca-4161-a778-ba58b0e84147-agent-certs\") pod \"konnectivity-agent-62ccn\" (UID: \"c014e8ac-58ca-4161-a778-ba58b0e84147\") " pod="kube-system/konnectivity-agent-62ccn"
Nov 01 03:06:36.781594 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.781578    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/bbdccd97-7806-4669-a9cb-6aef70599c54-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-gtk6f\" (UID: \"bbdccd97-7806-4669-a9cb-6aef70599c54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gtk6f"
Nov 01 03:06:36.781731 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.781717    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-node-log\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.781827 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.781814    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/998efdf0-7719-4768-bfd0-3da04d85d108-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rjtwk\" (UID: \"998efdf0-7719-4768-bfd0-3da04d85d108\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk"
Nov 01 03:06:36.781920 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.781906    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvjgq\" (UniqueName: \"kubernetes.io/projected/f29ea61e-2a78-4c11-9566-f52497e135ff-kube-api-access-tvjgq\") pod \"console-operator-7cd56b8dbc-jh955\" (UID: \"f29ea61e-2a78-4c11-9566-f52497e135ff\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jh955"
Nov 01 03:06:36.782016 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.782004    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-host-kubelet\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.782103 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.782090    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19871ddf-e177-4442-b592-8f1c7df24aaf-metrics-tls\") pod \"dns-default-wps29\" (UID: \"19871ddf-e177-4442-b592-8f1c7df24aaf\") " pod="openshift-dns/dns-default-wps29"
Nov 01 03:06:36.782210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.782197    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d28ff750-94d5-4eff-a8ef-58215bfb99e3-image-registry-private-configuration\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:36.782296 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.782283    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a972ee28-29fa-4755-adcd-b40d476c22f0-original-pull-secret\") pod \"global-pull-secret-syncer-4bvb6\" (UID: \"a972ee28-29fa-4755-adcd-b40d476c22f0\") " pod="kube-system/global-pull-secret-syncer-4bvb6"
Nov 01 03:06:36.782830 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.782813    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/73622c04-1428-4f7a-8faf-8eb17604f681-host\") pod \"node-ca-stpgr\" (UID: \"73622c04-1428-4f7a-8faf-8eb17604f681\") " pod="openshift-image-registry/node-ca-stpgr"
Nov 01 03:06:36.782915 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.782894    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f29ea61e-2a78-4c11-9566-f52497e135ff-serving-cert\") pod \"console-operator-7cd56b8dbc-jh955\" (UID: \"f29ea61e-2a78-4c11-9566-f52497e135ff\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jh955"
Nov 01 03:06:36.782999 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.782986    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-run-ovn\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.783101 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.783088    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1d8dba89-5673-4cd3-a6ac-914765f228d6-cni-binary-copy\") pod \"multus-additional-cni-plugins-9nhh6\" (UID: \"1d8dba89-5673-4cd3-a6ac-914765f228d6\") " pod="openshift-multus/multus-additional-cni-plugins-9nhh6"
Nov 01 03:06:36.783192 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.783178    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlznq\" (UniqueName: \"kubernetes.io/projected/bbdccd97-7806-4669-a9cb-6aef70599c54-kube-api-access-qlznq\") pod \"cluster-samples-operator-6ddb6c8b94-gtk6f\" (UID: \"bbdccd97-7806-4669-a9cb-6aef70599c54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gtk6f"
Nov 01 03:06:36.783289 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.783276    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lbzv\" (UniqueName: \"kubernetes.io/projected/e09a58a7-120e-4ec5-9aba-f15863267643-kube-api-access-7lbzv\") pod \"network-check-target-62mxc\" (UID: \"e09a58a7-120e-4ec5-9aba-f15863267643\") " pod="openshift-network-diagnostics/network-check-target-62mxc"
Nov 01 03:06:36.783386 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.783373    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-stats-auth\") pod \"router-default-ddbff58bc-5dlt4\" (UID: \"be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88\") " pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:36.783685 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.783662    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-lib-modules\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:06:36.783768 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.783706    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-host-slash\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.783768 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.783753    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9e212e37-6c34-4b3c-941c-b38621f1f184-env-overrides\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.783866 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.783782    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-cnibin\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.784067 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.784045    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9acc0734-3562-41fc-a05a-8c32cbf36443-ca-trust-extracted\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:36.784138 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.784117    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d28ff750-94d5-4eff-a8ef-58215bfb99e3-ca-trust-extracted\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:36.784393 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.784328    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-host-run-k8s-cni-cncf-io\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.784505 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.784425    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-multus-daemon-config\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.784592 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.784574    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-certificates\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:36.784650 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.784622    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbhvz\" (UniqueName: \"kubernetes.io/projected/e60132c0-948e-4f84-883e-d22564f210e1-kube-api-access-wbhvz\") pod \"kas-connection-checker-685895c7c9-fznsl\" (UID: \"e60132c0-948e-4f84-883e-d22564f210e1\") " pod="kube-system/kas-connection-checker-685895c7c9-fznsl"
Nov 01 03:06:36.784700 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.784660    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/eee1338d-e2a0-4a45-8f74-b81eb7d23019-snapshots\") pod \"insights-operator-7c4986d449-48ssl\" (UID: \"eee1338d-e2a0-4a45-8f74-b81eb7d23019\") " pod="openshift-insights/insights-operator-7c4986d449-48ssl"
Nov 01 03:06:36.784751 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.784724    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.784751 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.784742    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/eee1338d-e2a0-4a45-8f74-b81eb7d23019-snapshots\") pod \"insights-operator-7c4986d449-48ssl\" (UID: \"eee1338d-e2a0-4a45-8f74-b81eb7d23019\") " pod="openshift-insights/insights-operator-7c4986d449-48ssl"
Nov 01 03:06:36.784856 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.784755    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-multus-conf-dir\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.784856 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.784843    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a22c3f93-43f6-4908-9549-b966fb3c04e5-cert\") pod \"ingress-canary-zdrpv\" (UID: \"a22c3f93-43f6-4908-9549-b966fb3c04e5\") " pod="openshift-ingress-canary/ingress-canary-zdrpv"
Nov 01 03:06:36.784947 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.784876    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-host\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:06:36.784947 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.784911    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-tls\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:36.785008 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.784953    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-etc-modprobe-d\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:06:36.785008 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.784987    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2vfgr\" (UID: \"caebbba6-44f3-40c1-95ce-b8a5a3f2667b\") " pod="openshift-insights/insights-runtime-extractor-2vfgr"
Nov 01 03:06:36.785066 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.785039    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5fh9\" (UniqueName: \"kubernetes.io/projected/bcbdd4ff-e252-4aee-9418-9b1b05eed574-kube-api-access-z5fh9\") pod \"volume-data-source-validator-58c9f5f46d-62ws6\" (UID: \"bcbdd4ff-e252-4aee-9418-9b1b05eed574\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-62ws6"
Nov 01 03:06:36.785095 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.785077    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/998efdf0-7719-4768-bfd0-3da04d85d108-registration-dir\") pod \"aws-ebs-csi-driver-node-rjtwk\" (UID: \"998efdf0-7719-4768-bfd0-3da04d85d108\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk"
Nov 01 03:06:36.785124 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.785098    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fecfe177-e63f-490f-94bc-c307041640d5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xbdmd\" (UID: \"fecfe177-e63f-490f-94bc-c307041640d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xbdmd"
Nov 01 03:06:36.785156 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.785132    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-multus-cni-dir\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.785283 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.785151    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b7cm\" (UniqueName: \"kubernetes.io/projected/eee1338d-e2a0-4a45-8f74-b81eb7d23019-kube-api-access-8b7cm\") pod \"insights-operator-7c4986d449-48ssl\" (UID: \"eee1338d-e2a0-4a45-8f74-b81eb7d23019\") " pod="openshift-insights/insights-operator-7c4986d449-48ssl"
Nov 01 03:06:36.785643 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.785623    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7dbf71ad-8399-4a40-87d7-98746e830fa9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-t6tw7\" (UID: \"7dbf71ad-8399-4a40-87d7-98746e830fa9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-t6tw7"
Nov 01 03:06:36.785723 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.785677    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-etc-sysconfig\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:06:36.785723 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.785714    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-etc-sysctl-d\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:06:36.785826 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.785773    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-metrics-certs\") pod \"router-default-ddbff58bc-5dlt4\" (UID: \"be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88\") " pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:36.785826 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.785809    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8190fea3-4e71-4250-9a0f-3a1d78dd32db-etc-tuned\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:06:36.785913 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.785826    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/998efdf0-7719-4768-bfd0-3da04d85d108-sys-fs\") pod \"aws-ebs-csi-driver-node-rjtwk\" (UID: \"998efdf0-7719-4768-bfd0-3da04d85d108\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk"
Nov 01 03:06:36.785913 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.785887    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1d8dba89-5673-4cd3-a6ac-914765f228d6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9nhh6\" (UID: \"1d8dba89-5673-4cd3-a6ac-914765f228d6\") " pod="openshift-multus/multus-additional-cni-plugins-9nhh6"
Nov 01 03:06:36.786001 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.785942    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8190fea3-4e71-4250-9a0f-3a1d78dd32db-etc-tuned\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:06:36.786001 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.785979    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fecfe177-e63f-490f-94bc-c307041640d5-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-xbdmd\" (UID: \"fecfe177-e63f-490f-94bc-c307041640d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xbdmd"
Nov 01 03:06:36.786090 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.786017    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9acc0734-3562-41fc-a05a-8c32cbf36443-image-registry-private-configuration\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:36.786090 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.786052    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/14e2ac70-be2e-4e08-b093-e8a93d6f985a-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-chqll\" (UID: \"14e2ac70-be2e-4e08-b093-e8a93d6f985a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chqll"
Nov 01 03:06:36.786242 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.786129    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnmwf\" (UniqueName: \"kubernetes.io/projected/0a66c33f-29bc-4920-9d7a-0c45215514a0-kube-api-access-xnmwf\") pod \"network-metrics-daemon-mzkbj\" (UID: \"0a66c33f-29bc-4920-9d7a-0c45215514a0\") " pod="openshift-multus/network-metrics-daemon-mzkbj"
Nov 01 03:06:36.786242 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.786164    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bhzm\" (UniqueName: \"kubernetes.io/projected/627e3af2-4a0d-4f19-91bc-760af67e01ad-kube-api-access-8bhzm\") pod \"service-ca-5fffc97bdf-wjvnt\" (UID: \"627e3af2-4a0d-4f19-91bc-760af67e01ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wjvnt"
Nov 01 03:06:36.786242 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.786198    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-etc-openvswitch\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.786242 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.786227    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c014e8ac-58ca-4161-a778-ba58b0e84147-konnectivity-ca\") pod \"konnectivity-agent-62ccn\" (UID: \"c014e8ac-58ca-4161-a778-ba58b0e84147\") " pod="kube-system/konnectivity-agent-62ccn"
Nov 01 03:06:36.786242 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.786226    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fecfe177-e63f-490f-94bc-c307041640d5-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-xbdmd\" (UID: \"fecfe177-e63f-490f-94bc-c307041640d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xbdmd"
Nov 01 03:06:36.786443 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.786274    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d28ff750-94d5-4eff-a8ef-58215bfb99e3-trusted-ca\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:36.786443 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.786301    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-var-lib-kubelet\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:06:36.786443 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.786344    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bbdccd97-7806-4669-a9cb-6aef70599c54-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gtk6f\" (UID: \"bbdccd97-7806-4669-a9cb-6aef70599c54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gtk6f"
Nov 01 03:06:36.786443 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.786379    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-host-run-netns\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.786443 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.786412    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/627e3af2-4a0d-4f19-91bc-760af67e01ad-signing-cabundle\") pod \"service-ca-5fffc97bdf-wjvnt\" (UID: \"627e3af2-4a0d-4f19-91bc-760af67e01ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wjvnt"
Nov 01 03:06:36.786662 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.786477    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlvp2\" (UniqueName: \"kubernetes.io/projected/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-kube-api-access-wlvp2\") pod \"insights-runtime-extractor-2vfgr\" (UID: \"caebbba6-44f3-40c1-95ce-b8a5a3f2667b\") " pod="openshift-insights/insights-runtime-extractor-2vfgr"
Nov 01 03:06:36.786662 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.786497    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-etc-kubernetes\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.786662 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.786515    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbfdt\" (UniqueName: \"kubernetes.io/projected/ca0be51d-87a4-462d-9c83-fcfb8bf224e8-kube-api-access-cbfdt\") pod \"iptables-alerter-ksj2n\" (UID: \"ca0be51d-87a4-462d-9c83-fcfb8bf224e8\") " pod="openshift-network-operator/iptables-alerter-ksj2n"
Nov 01 03:06:36.786662 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.786537    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19871ddf-e177-4442-b592-8f1c7df24aaf-tmp-dir\") pod \"dns-default-wps29\" (UID: \"19871ddf-e177-4442-b592-8f1c7df24aaf\") " pod="openshift-dns/dns-default-wps29"
Nov 01 03:06:36.786662 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.786575    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-bound-sa-token\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:36.786662 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.786601    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-default-certificate\") pod \"router-default-ddbff58bc-5dlt4\" (UID: \"be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88\") " pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:36.786662 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.786633    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19871ddf-e177-4442-b592-8f1c7df24aaf-tmp-dir\") pod \"dns-default-wps29\" (UID: \"19871ddf-e177-4442-b592-8f1c7df24aaf\") " pod="openshift-dns/dns-default-wps29"
Nov 01 03:06:36.786918 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.786698    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a66c33f-29bc-4920-9d7a-0c45215514a0-metrics-certs\") pod \"network-metrics-daemon-mzkbj\" (UID: \"0a66c33f-29bc-4920-9d7a-0c45215514a0\") " pod="openshift-multus/network-metrics-daemon-mzkbj"
Nov 01 03:06:36.786918 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.786723    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-host-run-ovn-kubernetes\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.786918 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.786745    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/73622c04-1428-4f7a-8faf-8eb17604f681-serviceca\") pod \"node-ca-stpgr\" (UID: \"73622c04-1428-4f7a-8faf-8eb17604f681\") " pod="openshift-image-registry/node-ca-stpgr"
Nov 01 03:06:36.786918 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.786762    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1d8dba89-5673-4cd3-a6ac-914765f228d6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9nhh6\" (UID: \"1d8dba89-5673-4cd3-a6ac-914765f228d6\") " pod="openshift-multus/multus-additional-cni-plugins-9nhh6"
Nov 01 03:06:36.786918 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.786780    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67zjh\" (UniqueName: \"kubernetes.io/projected/581612ca-5a0b-4426-9d07-ee65f85e0412-kube-api-access-67zjh\") pod \"kas-connection-checker-685895c7c9-8plk2\" (UID: \"581612ca-5a0b-4426-9d07-ee65f85e0412\") " pod="kube-system/kas-connection-checker-685895c7c9-8plk2"
Nov 01 03:06:36.786918 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.786806    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/998efdf0-7719-4768-bfd0-3da04d85d108-etc-selinux\") pod \"aws-ebs-csi-driver-node-rjtwk\" (UID: \"998efdf0-7719-4768-bfd0-3da04d85d108\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk"
Nov 01 03:06:36.786918 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.786828    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a972ee28-29fa-4755-adcd-b40d476c22f0-dbus\") pod \"global-pull-secret-syncer-4bvb6\" (UID: \"a972ee28-29fa-4755-adcd-b40d476c22f0\") " pod="kube-system/global-pull-secret-syncer-4bvb6"
Nov 01 03:06:36.786918 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.786843    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-crio-socket\") pod \"insights-runtime-extractor-2vfgr\" (UID: \"caebbba6-44f3-40c1-95ce-b8a5a3f2667b\") " pod="openshift-insights/insights-runtime-extractor-2vfgr"
Nov 01 03:06:36.786918 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.786860    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d28ff750-94d5-4eff-a8ef-58215bfb99e3-installation-pull-secrets\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:36.786918 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.786901    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-hostroot\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.787179 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.786949    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-etc-systemd\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:06:36.787179 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.786984    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-555tt\" (UniqueName: \"kubernetes.io/projected/f097ea60-e18f-47f0-b6a0-24e2b3a201d0-kube-api-access-555tt\") pod \"migrator-66df5d496b-cbbs4\" (UID: \"f097ea60-e18f-47f0-b6a0-24e2b3a201d0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cbbs4"
Nov 01 03:06:36.787179 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.787009    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d8dba89-5673-4cd3-a6ac-914765f228d6-system-cni-dir\") pod \"multus-additional-cni-plugins-9nhh6\" (UID: \"1d8dba89-5673-4cd3-a6ac-914765f228d6\") " pod="openshift-multus/multus-additional-cni-plugins-9nhh6"
Nov 01 03:06:36.787179 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.787031    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9e212e37-6c34-4b3c-941c-b38621f1f184-ovnkube-config\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.787179 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.787112    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9acc0734-3562-41fc-a05a-8c32cbf36443-trusted-ca\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:36.787179 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.787133    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-certificates\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:36.787335 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.787184    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/998efdf0-7719-4768-bfd0-3da04d85d108-device-dir\") pod \"aws-ebs-csi-driver-node-rjtwk\" (UID: \"998efdf0-7719-4768-bfd0-3da04d85d108\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk"
Nov 01 03:06:36.787335 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.787214    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-host-cni-netd\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.787335 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.787240    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-system-cni-dir\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.787335 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.787269    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eee1338d-e2a0-4a45-8f74-b81eb7d23019-service-ca-bundle\") pod \"insights-operator-7c4986d449-48ssl\" (UID: \"eee1338d-e2a0-4a45-8f74-b81eb7d23019\") " pod="openshift-insights/insights-operator-7c4986d449-48ssl"
Nov 01 03:06:36.787335 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.787294    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-run-systemd\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.787498 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.787340    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/482aebaa-cb7f-4be2-991a-7fb4cd801c7f-hosts-file\") pod \"node-resolver-77nf7\" (UID: \"482aebaa-cb7f-4be2-991a-7fb4cd801c7f\") " pod="openshift-dns/node-resolver-77nf7"
Nov 01 03:06:36.787498 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.787376    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-cni-binary-copy\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.787498 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.787408    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9ppn\" (UniqueName: \"kubernetes.io/projected/2ab1baec-cb71-4942-a246-7920cd44526c-kube-api-access-t9ppn\") pod \"network-check-source-6f78d59f99-fsr8j\" (UID: \"2ab1baec-cb71-4942-a246-7920cd44526c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fsr8j"
Nov 01 03:06:36.787498 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.787427    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-run-openvswitch\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.787498 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.787464    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/45c60c0c-dcff-421d-a9f2-4dc53e7637c9-config\") pod \"service-ca-operator-544d749c8c-cbzc6\" (UID: \"45c60c0c-dcff-421d-a9f2-4dc53e7637c9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbzc6"
Nov 01 03:06:36.787498 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.787491    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-etc-sysctl-conf\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:06:36.787690 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.787530    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8190fea3-4e71-4250-9a0f-3a1d78dd32db-tmp\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:06:36.787690 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.787548    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f29ea61e-2a78-4c11-9566-f52497e135ff-trusted-ca\") pod \"console-operator-7cd56b8dbc-jh955\" (UID: \"f29ea61e-2a78-4c11-9566-f52497e135ff\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jh955"
Nov 01 03:06:36.787690 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.787582    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/627e3af2-4a0d-4f19-91bc-760af67e01ad-signing-key\") pod \"service-ca-5fffc97bdf-wjvnt\" (UID: \"627e3af2-4a0d-4f19-91bc-760af67e01ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wjvnt"
Nov 01 03:06:36.787690 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.787599    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9e212e37-6c34-4b3c-941c-b38621f1f184-ovn-node-metrics-cert\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.787690 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.787622    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8190fea3-4e71-4250-9a0f-3a1d78dd32db-tmp\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:06:36.787690 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.787628    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-data-volume\") pod \"insights-runtime-extractor-2vfgr\" (UID: \"caebbba6-44f3-40c1-95ce-b8a5a3f2667b\") " pod="openshift-insights/insights-runtime-extractor-2vfgr"
Nov 01 03:06:36.787690 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.787667    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2vfgr\" (UID: \"caebbba6-44f3-40c1-95ce-b8a5a3f2667b\") " pod="openshift-insights/insights-runtime-extractor-2vfgr"
Nov 01 03:06:36.787690 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.787683    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-data-volume\") pod \"insights-runtime-extractor-2vfgr\" (UID: \"caebbba6-44f3-40c1-95ce-b8a5a3f2667b\") " pod="openshift-insights/insights-runtime-extractor-2vfgr"
Nov 01 03:06:36.787931 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.787716    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/14e2ac70-be2e-4e08-b093-e8a93d6f985a-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-chqll\" (UID: \"14e2ac70-be2e-4e08-b093-e8a93d6f985a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chqll"
Nov 01 03:06:36.787931 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.787781    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1d8dba89-5673-4cd3-a6ac-914765f228d6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9nhh6\" (UID: \"1d8dba89-5673-4cd3-a6ac-914765f228d6\") " pod="openshift-multus/multus-additional-cni-plugins-9nhh6"
Nov 01 03:06:36.787931 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.787803    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/45c60c0c-dcff-421d-a9f2-4dc53e7637c9-serving-cert\") pod \"service-ca-operator-544d749c8c-cbzc6\" (UID: \"45c60c0c-dcff-421d-a9f2-4dc53e7637c9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbzc6"
Nov 01 03:06:36.787931 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.787839    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm7m6\" (UniqueName: \"kubernetes.io/projected/482aebaa-cb7f-4be2-991a-7fb4cd801c7f-kube-api-access-mm7m6\") pod \"node-resolver-77nf7\" (UID: \"482aebaa-cb7f-4be2-991a-7fb4cd801c7f\") " pod="openshift-dns/node-resolver-77nf7"
Nov 01 03:06:36.787931 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.787868    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f29ea61e-2a78-4c11-9566-f52497e135ff-trusted-ca\") pod \"console-operator-7cd56b8dbc-jh955\" (UID: \"f29ea61e-2a78-4c11-9566-f52497e135ff\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jh955"
Nov 01 03:06:36.787931 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.787902    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfxfn\" (UniqueName: \"kubernetes.io/projected/998efdf0-7719-4768-bfd0-3da04d85d108-kube-api-access-qfxfn\") pod \"aws-ebs-csi-driver-node-rjtwk\" (UID: \"998efdf0-7719-4768-bfd0-3da04d85d108\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk"
Nov 01 03:06:36.788089 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.787935    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h78fl\" (UniqueName: \"kubernetes.io/projected/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-kube-api-access-h78fl\") pod \"router-default-ddbff58bc-5dlt4\" (UID: \"be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88\") " pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:36.788089 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.787957    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f29ea61e-2a78-4c11-9566-f52497e135ff-config\") pod \"console-operator-7cd56b8dbc-jh955\" (UID: \"f29ea61e-2a78-4c11-9566-f52497e135ff\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jh955"
Nov 01 03:06:36.788089 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.787986    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x5t7\" (UniqueName: \"kubernetes.io/projected/1d8dba89-5673-4cd3-a6ac-914765f228d6-kube-api-access-5x5t7\") pod \"multus-additional-cni-plugins-9nhh6\" (UID: \"1d8dba89-5673-4cd3-a6ac-914765f228d6\") " pod="openshift-multus/multus-additional-cni-plugins-9nhh6"
Nov 01 03:06:36.788200 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.788184    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f29ea61e-2a78-4c11-9566-f52497e135ff-config\") pod \"console-operator-7cd56b8dbc-jh955\" (UID: \"f29ea61e-2a78-4c11-9566-f52497e135ff\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jh955"
Nov 01 03:06:36.818911 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.818857    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 01 03:06:36.819117 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.819098    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 03:06:36.819173 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.819123    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 01 03:06:36.819230 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.819217    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 03:06:36.819261 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.819220    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 03:06:36.819261 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.819235    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 01 03:06:36.819261 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.819242    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 03:06:36.819261 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.819258    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 03:06:36.819261 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.819234    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 03:06:36.819476 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.819411    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 03:06:36.819476 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.819469    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 03:06:36.819585 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.819530    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 03:06:36.819641 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.819599    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 03:06:36.819641 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.819610    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 03:06:36.819739 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.819683    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 01 03:06:36.819739 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.819692    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 03:06:36.819739 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.819705    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 03:06:36.819739 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.819719    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 03:06:36.819911 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.819696    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 03:06:36.819911 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.819751    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 03:06:36.819911 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.819770    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 03:06:36.819911 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.819782    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 03:06:36.819911 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.819794    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 03:06:36.819911 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.819839    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 03:06:36.819911 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.819876    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 01 03:06:36.819911 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.819885    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 01 03:06:36.819911 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.819903    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 03:06:36.819911 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.819909    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 03:06:36.820200 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.819929    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 01 03:06:36.820200 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.819981    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/19871ddf-e177-4442-b592-8f1c7df24aaf-config-volume\") pod \"dns-default-wps29\" (UID: \"19871ddf-e177-4442-b592-8f1c7df24aaf\") " pod="openshift-dns/dns-default-wps29"
Nov 01 03:06:36.820200 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.820150    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 01 03:06:36.820200 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.820177    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 03:06:36.820399 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.820224    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 03:06:36.820399 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.820226    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 03:06:36.820399 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.820272    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 03:06:36.820399 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.820279    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 03:06:36.820399 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.820302    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ca0be51d-87a4-462d-9c83-fcfb8bf224e8-iptables-alerter-script\") pod \"iptables-alerter-ksj2n\" (UID: \"ca0be51d-87a4-462d-9c83-fcfb8bf224e8\") " pod="openshift-network-operator/iptables-alerter-ksj2n"
Nov 01 03:06:36.820638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.820447    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 01 03:06:36.820638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.820458    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 01 03:06:36.820638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.820473    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 03:06:36.820638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.820497    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 01 03:06:36.820638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.820497    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 01 03:06:36.820638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.820517    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 03:06:36.820638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.820497    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 03:06:36.820638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.820544    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 01 03:06:36.820638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.820576    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 03:06:36.820638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.820583    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 01 03:06:36.820638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.820596    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 01 03:06:36.820638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.820610    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 01 03:06:36.820638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.820600    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 01 03:06:36.820638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.820614    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 01 03:06:36.821068 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.820703    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 01 03:06:36.821208 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.821190    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7dbf71ad-8399-4a40-87d7-98746e830fa9-nginx-conf\") pod \"networking-console-plugin-86c478bb88-t6tw7\" (UID: \"7dbf71ad-8399-4a40-87d7-98746e830fa9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-t6tw7"
Nov 01 03:06:36.821722 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.821680    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 01 03:06:36.821722 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.821693    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 03:06:36.821814 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.821724    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 03:06:36.821814 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.821767    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 01 03:06:36.823243 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.823226    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 03:06:36.823350 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.823338    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-j68cz\""
Nov 01 03:06:36.823526 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.823512    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wgrq6\""
Nov 01 03:06:36.823739 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.823723    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1d8dba89-5673-4cd3-a6ac-914765f228d6-cni-binary-copy\") pod \"multus-additional-cni-plugins-9nhh6\" (UID: \"1d8dba89-5673-4cd3-a6ac-914765f228d6\") " pod="openshift-multus/multus-additional-cni-plugins-9nhh6"
Nov 01 03:06:36.823873 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.823857    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 03:06:36.823930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.823872    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 01 03:06:36.823930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.823886    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 01 03:06:36.823930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.823891    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 01 03:06:36.823930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.823889    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 01 03:06:36.823930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.823911    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 01 03:06:36.824115 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.823934    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 01 03:06:36.824115 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.824069    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 03:06:36.824115 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.824078    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 01 03:06:36.824115 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.824083    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 01 03:06:36.824115 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.824101    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 01 03:06:36.824115 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.824101    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 01 03:06:36.824345 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.824188    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 03:06:36.824345 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.824197    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-pn9xg\""
Nov 01 03:06:36.824345 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.824232    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-vzcml\""
Nov 01 03:06:36.824345 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.824233    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d28ff750-94d5-4eff-a8ef-58215bfb99e3-image-registry-private-configuration\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:36.824345 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.824233    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 03:06:36.824345 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.824262    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 01 03:06:36.824503 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.824371    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9e212e37-6c34-4b3c-941c-b38621f1f184-env-overrides\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.824503 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.824495    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c014e8ac-58ca-4161-a778-ba58b0e84147-agent-certs\") pod \"konnectivity-agent-62ccn\" (UID: \"c014e8ac-58ca-4161-a778-ba58b0e84147\") " pod="kube-system/konnectivity-agent-62ccn"
Nov 01 03:06:36.824619 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.824502    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/bbdccd97-7806-4669-a9cb-6aef70599c54-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-gtk6f\" (UID: \"bbdccd97-7806-4669-a9cb-6aef70599c54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gtk6f"
Nov 01 03:06:36.824619 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.824494    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-stats-auth\") pod \"router-default-ddbff58bc-5dlt4\" (UID: \"be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88\") " pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:36.824707 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.824693    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-multus-daemon-config\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.824799 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.824788    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-certificates\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:36.825476 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.825464    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a22c3f93-43f6-4908-9549-b966fb3c04e5-cert\") pod \"ingress-canary-zdrpv\" (UID: \"a22c3f93-43f6-4908-9549-b966fb3c04e5\") " pod="openshift-ingress-canary/ingress-canary-zdrpv"
Nov 01 03:06:36.826590 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.826566    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c014e8ac-58ca-4161-a778-ba58b0e84147-konnectivity-ca\") pod \"konnectivity-agent-62ccn\" (UID: \"c014e8ac-58ca-4161-a778-ba58b0e84147\") " pod="kube-system/konnectivity-agent-62ccn"
Nov 01 03:06:36.826672 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.826628    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/14e2ac70-be2e-4e08-b093-e8a93d6f985a-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-chqll\" (UID: \"14e2ac70-be2e-4e08-b093-e8a93d6f985a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chqll"
Nov 01 03:06:36.826672 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.826641    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9acc0734-3562-41fc-a05a-8c32cbf36443-image-registry-private-configuration\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:36.826672 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.826660    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/627e3af2-4a0d-4f19-91bc-760af67e01ad-signing-cabundle\") pod \"service-ca-5fffc97bdf-wjvnt\" (UID: \"627e3af2-4a0d-4f19-91bc-760af67e01ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wjvnt"
Nov 01 03:06:36.826785 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.826641    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1d8dba89-5673-4cd3-a6ac-914765f228d6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9nhh6\" (UID: \"1d8dba89-5673-4cd3-a6ac-914765f228d6\") " pod="openshift-multus/multus-additional-cni-plugins-9nhh6"
Nov 01 03:06:36.826785 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.826747    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d28ff750-94d5-4eff-a8ef-58215bfb99e3-trusted-ca\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:36.826844 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.826783    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-default-certificate\") pod \"router-default-ddbff58bc-5dlt4\" (UID: \"be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88\") " pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:36.827974 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.827801    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-certificates\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:36.827974 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.827821    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9e212e37-6c34-4b3c-941c-b38621f1f184-ovn-node-metrics-cert\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.827974 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.827828    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/45c60c0c-dcff-421d-a9f2-4dc53e7637c9-config\") pod \"service-ca-operator-544d749c8c-cbzc6\" (UID: \"45c60c0c-dcff-421d-a9f2-4dc53e7637c9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbzc6"
Nov 01 03:06:36.827974 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.827824    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/627e3af2-4a0d-4f19-91bc-760af67e01ad-signing-key\") pod \"service-ca-5fffc97bdf-wjvnt\" (UID: \"627e3af2-4a0d-4f19-91bc-760af67e01ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wjvnt"
Nov 01 03:06:36.827974 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.827858    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-cni-binary-copy\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.827974 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.827917    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9e212e37-6c34-4b3c-941c-b38621f1f184-ovnkube-config\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.827974 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.827933    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/73622c04-1428-4f7a-8faf-8eb17604f681-serviceca\") pod \"node-ca-stpgr\" (UID: \"73622c04-1428-4f7a-8faf-8eb17604f681\") " pod="openshift-image-registry/node-ca-stpgr"
Nov 01 03:06:36.827974 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.827931    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eee1338d-e2a0-4a45-8f74-b81eb7d23019-service-ca-bundle\") pod \"insights-operator-7c4986d449-48ssl\" (UID: \"eee1338d-e2a0-4a45-8f74-b81eb7d23019\") " pod="openshift-insights/insights-operator-7c4986d449-48ssl"
Nov 01 03:06:36.827974 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.827966    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1d8dba89-5673-4cd3-a6ac-914765f228d6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9nhh6\" (UID: \"1d8dba89-5673-4cd3-a6ac-914765f228d6\") " pod="openshift-multus/multus-additional-cni-plugins-9nhh6"
Nov 01 03:06:36.828314 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.827991    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/14e2ac70-be2e-4e08-b093-e8a93d6f985a-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-chqll\" (UID: \"14e2ac70-be2e-4e08-b093-e8a93d6f985a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chqll"
Nov 01 03:06:36.828314 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.828024    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/45c60c0c-dcff-421d-a9f2-4dc53e7637c9-serving-cert\") pod \"service-ca-operator-544d749c8c-cbzc6\" (UID: \"45c60c0c-dcff-421d-a9f2-4dc53e7637c9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbzc6"
Nov 01 03:06:36.828314 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.828026    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9acc0734-3562-41fc-a05a-8c32cbf36443-trusted-ca\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:36.830788 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.830767    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eee1338d-e2a0-4a45-8f74-b81eb7d23019-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-48ssl\" (UID: \"eee1338d-e2a0-4a45-8f74-b81eb7d23019\") " pod="openshift-insights/insights-operator-7c4986d449-48ssl"
Nov 01 03:06:36.830880 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.830867    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eee1338d-e2a0-4a45-8f74-b81eb7d23019-serving-cert\") pod \"insights-operator-7c4986d449-48ssl\" (UID: \"eee1338d-e2a0-4a45-8f74-b81eb7d23019\") " pod="openshift-insights/insights-operator-7c4986d449-48ssl"
Nov 01 03:06:36.830975 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.830960    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9e212e37-6c34-4b3c-941c-b38621f1f184-ovnkube-script-lib\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.868193 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.868169    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mt5gq\""
Nov 01 03:06:36.868193 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.868181    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-62njk\""
Nov 01 03:06:36.868460 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.868442    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-hdt7b\""
Nov 01 03:06:36.868496 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.868442    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-fdfjz\""
Nov 01 03:06:36.869000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.868985    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-dsb6b\""
Nov 01 03:06:36.869064 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.869031    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-tmf7b\""
Nov 01 03:06:36.869118 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.869062    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-bt25w\""
Nov 01 03:06:36.869165 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.869123    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-p789z\""
Nov 01 03:06:36.869165 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.869128    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-77n5m\""
Nov 01 03:06:36.869165 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.869134    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hrmfk\""
Nov 01 03:06:36.869165 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.869154    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 03:06:36.869327 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.869182    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4hc7l\""
Nov 01 03:06:36.869327 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.869207    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6592m\""
Nov 01 03:06:36.869327 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.869199    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 01 03:06:36.869327 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.869226    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8s4ww\""
Nov 01 03:06:36.869327 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.869256    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-cdgq2\""
Nov 01 03:06:36.869327 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.869285    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-vtrwc\""
Nov 01 03:06:36.869327 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.869304    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kkkrb\""
Nov 01 03:06:36.869606 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.869380    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z9gxp\""
Nov 01 03:06:36.869606 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.869390    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ksxvj\""
Nov 01 03:06:36.869606 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.869442    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-kbjpv\""
Nov 01 03:06:36.869606 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.869464    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dg4zg\""
Nov 01 03:06:36.869606 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.869476    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-74fv2\""
Nov 01 03:06:36.869781 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.869708    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 03:06:36.872000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.870387    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkm29\" (UniqueName: \"kubernetes.io/projected/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-kube-api-access-vkm29\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.872000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.870424    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwpzw\" (UniqueName: \"kubernetes.io/projected/9e212e37-6c34-4b3c-941c-b38621f1f184-kube-api-access-pwpzw\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.872000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.870389    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-bound-sa-token\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:36.872000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.870581    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvjgq\" (UniqueName: \"kubernetes.io/projected/f29ea61e-2a78-4c11-9566-f52497e135ff-kube-api-access-tvjgq\") pod \"console-operator-7cd56b8dbc-jh955\" (UID: \"f29ea61e-2a78-4c11-9566-f52497e135ff\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jh955"
Nov 01 03:06:36.872000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.870587    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqgps\" (UniqueName: \"kubernetes.io/projected/fecfe177-e63f-490f-94bc-c307041640d5-kube-api-access-gqgps\") pod \"cluster-monitoring-operator-884576c57-xbdmd\" (UID: \"fecfe177-e63f-490f-94bc-c307041640d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xbdmd"
Nov 01 03:06:36.872000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.870632    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lbzv\" (UniqueName: \"kubernetes.io/projected/e09a58a7-120e-4ec5-9aba-f15863267643-kube-api-access-7lbzv\") pod \"network-check-target-62mxc\" (UID: \"e09a58a7-120e-4ec5-9aba-f15863267643\") " pod="openshift-network-diagnostics/network-check-target-62mxc"
Nov 01 03:06:36.872000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.870647    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phjdn\" (UniqueName: \"kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-kube-api-access-phjdn\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:36.872000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.870666    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5z8d\" (UniqueName: \"kubernetes.io/projected/14e2ac70-be2e-4e08-b093-e8a93d6f985a-kube-api-access-p5z8d\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-chqll\" (UID: \"14e2ac70-be2e-4e08-b093-e8a93d6f985a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chqll"
Nov 01 03:06:36.872000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.871025    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbhvz\" (UniqueName: \"kubernetes.io/projected/e60132c0-948e-4f84-883e-d22564f210e1-kube-api-access-wbhvz\") pod \"kas-connection-checker-685895c7c9-fznsl\" (UID: \"e60132c0-948e-4f84-883e-d22564f210e1\") " pod="kube-system/kas-connection-checker-685895c7c9-fznsl"
Nov 01 03:06:36.872000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.871122    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpmsp\" (UniqueName: \"kubernetes.io/projected/a22c3f93-43f6-4908-9549-b966fb3c04e5-kube-api-access-dpmsp\") pod \"ingress-canary-zdrpv\" (UID: \"a22c3f93-43f6-4908-9549-b966fb3c04e5\") " pod="openshift-ingress-canary/ingress-canary-zdrpv"
Nov 01 03:06:36.872000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.871134    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnmwf\" (UniqueName: \"kubernetes.io/projected/0a66c33f-29bc-4920-9d7a-0c45215514a0-kube-api-access-xnmwf\") pod \"network-metrics-daemon-mzkbj\" (UID: \"0a66c33f-29bc-4920-9d7a-0c45215514a0\") " pod="openshift-multus/network-metrics-daemon-mzkbj"
Nov 01 03:06:36.872000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.871151    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlvp2\" (UniqueName: \"kubernetes.io/projected/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-kube-api-access-wlvp2\") pod \"insights-runtime-extractor-2vfgr\" (UID: \"caebbba6-44f3-40c1-95ce-b8a5a3f2667b\") " pod="openshift-insights/insights-runtime-extractor-2vfgr"
Nov 01 03:06:36.872000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.871157    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q546p\" (UniqueName: \"kubernetes.io/projected/8190fea3-4e71-4250-9a0f-3a1d78dd32db-kube-api-access-q546p\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:06:36.872000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.871191    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlznq\" (UniqueName: \"kubernetes.io/projected/bbdccd97-7806-4669-a9cb-6aef70599c54-kube-api-access-qlznq\") pod \"cluster-samples-operator-6ddb6c8b94-gtk6f\" (UID: \"bbdccd97-7806-4669-a9cb-6aef70599c54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gtk6f"
Nov 01 03:06:36.872000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.871233    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm7m6\" (UniqueName: \"kubernetes.io/projected/482aebaa-cb7f-4be2-991a-7fb4cd801c7f-kube-api-access-mm7m6\") pod \"node-resolver-77nf7\" (UID: \"482aebaa-cb7f-4be2-991a-7fb4cd801c7f\") " pod="openshift-dns/node-resolver-77nf7"
Nov 01 03:06:36.872000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.871460    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-bound-sa-token\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:36.872000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.871640    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-555tt\" (UniqueName: \"kubernetes.io/projected/f097ea60-e18f-47f0-b6a0-24e2b3a201d0-kube-api-access-555tt\") pod \"migrator-66df5d496b-cbbs4\" (UID: \"f097ea60-e18f-47f0-b6a0-24e2b3a201d0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cbbs4"
Nov 01 03:06:36.872000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.871699    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5xx5\" (UniqueName: \"kubernetes.io/projected/45c60c0c-dcff-421d-a9f2-4dc53e7637c9-kube-api-access-m5xx5\") pod \"service-ca-operator-544d749c8c-cbzc6\" (UID: \"45c60c0c-dcff-421d-a9f2-4dc53e7637c9\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbzc6"
Nov 01 03:06:36.872000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.871699    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjt9z\" (UniqueName: \"kubernetes.io/projected/73622c04-1428-4f7a-8faf-8eb17604f681-kube-api-access-hjt9z\") pod \"node-ca-stpgr\" (UID: \"73622c04-1428-4f7a-8faf-8eb17604f681\") " pod="openshift-image-registry/node-ca-stpgr"
Nov 01 03:06:36.872000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.871875    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtt7q\" (UniqueName: \"kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-kube-api-access-dtt7q\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:36.872000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.871909    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5fh9\" (UniqueName: \"kubernetes.io/projected/bcbdd4ff-e252-4aee-9418-9b1b05eed574-kube-api-access-z5fh9\") pod \"volume-data-source-validator-58c9f5f46d-62ws6\" (UID: \"bcbdd4ff-e252-4aee-9418-9b1b05eed574\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-62ws6"
Nov 01 03:06:36.872000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.871964    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bhzm\" (UniqueName: \"kubernetes.io/projected/627e3af2-4a0d-4f19-91bc-760af67e01ad-kube-api-access-8bhzm\") pod \"service-ca-5fffc97bdf-wjvnt\" (UID: \"627e3af2-4a0d-4f19-91bc-760af67e01ad\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wjvnt"
Nov 01 03:06:36.872998 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.872341    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbfdt\" (UniqueName: \"kubernetes.io/projected/ca0be51d-87a4-462d-9c83-fcfb8bf224e8-kube-api-access-cbfdt\") pod \"iptables-alerter-ksj2n\" (UID: \"ca0be51d-87a4-462d-9c83-fcfb8bf224e8\") " pod="openshift-network-operator/iptables-alerter-ksj2n"
Nov 01 03:06:36.872998 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.872419    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxfvf\" (UniqueName: \"kubernetes.io/projected/ffdeddae-4086-4fb5-b9f8-c8ffcb765925-kube-api-access-fxfvf\") pod \"kas-connection-checker-685895c7c9-j6kdp\" (UID: \"ffdeddae-4086-4fb5-b9f8-c8ffcb765925\") " pod="kube-system/kas-connection-checker-685895c7c9-j6kdp"
Nov 01 03:06:36.872998 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.872525    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67zjh\" (UniqueName: \"kubernetes.io/projected/581612ca-5a0b-4426-9d07-ee65f85e0412-kube-api-access-67zjh\") pod \"kas-connection-checker-685895c7c9-8plk2\" (UID: \"581612ca-5a0b-4426-9d07-ee65f85e0412\") " pod="kube-system/kas-connection-checker-685895c7c9-8plk2"
Nov 01 03:06:36.872998 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.872654    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9ppn\" (UniqueName: \"kubernetes.io/projected/2ab1baec-cb71-4942-a246-7920cd44526c-kube-api-access-t9ppn\") pod \"network-check-source-6f78d59f99-fsr8j\" (UID: \"2ab1baec-cb71-4942-a246-7920cd44526c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fsr8j"
Nov 01 03:06:36.872998 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.872753    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z96gl\" (UniqueName: \"kubernetes.io/projected/19871ddf-e177-4442-b592-8f1c7df24aaf-kube-api-access-z96gl\") pod \"dns-default-wps29\" (UID: \"19871ddf-e177-4442-b592-8f1c7df24aaf\") " pod="openshift-dns/dns-default-wps29"
Nov 01 03:06:36.872998 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.872854    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x5t7\" (UniqueName: \"kubernetes.io/projected/1d8dba89-5673-4cd3-a6ac-914765f228d6-kube-api-access-5x5t7\") pod \"multus-additional-cni-plugins-9nhh6\" (UID: \"1d8dba89-5673-4cd3-a6ac-914765f228d6\") " pod="openshift-multus/multus-additional-cni-plugins-9nhh6"
Nov 01 03:06:36.873270 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.873153    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h78fl\" (UniqueName: \"kubernetes.io/projected/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-kube-api-access-h78fl\") pod \"router-default-ddbff58bc-5dlt4\" (UID: \"be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88\") " pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:36.873270 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.873177    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b7cm\" (UniqueName: \"kubernetes.io/projected/eee1338d-e2a0-4a45-8f74-b81eb7d23019-kube-api-access-8b7cm\") pod \"insights-operator-7c4986d449-48ssl\" (UID: \"eee1338d-e2a0-4a45-8f74-b81eb7d23019\") " pod="openshift-insights/insights-operator-7c4986d449-48ssl"
Nov 01 03:06:36.873270 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.873187    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfxfn\" (UniqueName: \"kubernetes.io/projected/998efdf0-7719-4768-bfd0-3da04d85d108-kube-api-access-qfxfn\") pod \"aws-ebs-csi-driver-node-rjtwk\" (UID: \"998efdf0-7719-4768-bfd0-3da04d85d108\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk"
Nov 01 03:06:36.876590 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.876549    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9acc0734-3562-41fc-a05a-8c32cbf36443-installation-pull-secrets\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:36.876662 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.876596    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a972ee28-29fa-4755-adcd-b40d476c22f0-original-pull-secret\") pod \"global-pull-secret-syncer-4bvb6\" (UID: \"a972ee28-29fa-4755-adcd-b40d476c22f0\") " pod="kube-system/global-pull-secret-syncer-4bvb6"
Nov 01 03:06:36.877188 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.877171    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d28ff750-94d5-4eff-a8ef-58215bfb99e3-installation-pull-secrets\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:36.888773 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.888756    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-host-slash\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.888829 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.888781    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-cnibin\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.888829 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.888796    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-host-run-k8s-cni-cncf-io\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.888829 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.888812    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.888965 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.888832    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-multus-conf-dir\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.888965 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.888873    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-multus-conf-dir\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.888965 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.888874    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-cnibin\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.888965 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.888893    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-host\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:06:36.888965 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.888899    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.888965 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.888901    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-host-run-k8s-cni-cncf-io\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.888965 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.888873    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-host-slash\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.888965 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.888928    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-host\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:06:36.888965 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.888954    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-tls\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:36.889308 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.888973    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-etc-modprobe-d\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:06:36.889308 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.888989    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2vfgr\" (UID: \"caebbba6-44f3-40c1-95ce-b8a5a3f2667b\") " pod="openshift-insights/insights-runtime-extractor-2vfgr"
Nov 01 03:06:36.889308 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889032    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/998efdf0-7719-4768-bfd0-3da04d85d108-registration-dir\") pod \"aws-ebs-csi-driver-node-rjtwk\" (UID: \"998efdf0-7719-4768-bfd0-3da04d85d108\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk"
Nov 01 03:06:36.889308 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:36.889039    4930 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 03:06:36.889308 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:36.889052    4930 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f44999c56-64k9p: secret "image-registry-tls" not found
Nov 01 03:06:36.889308 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889058    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fecfe177-e63f-490f-94bc-c307041640d5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xbdmd\" (UID: \"fecfe177-e63f-490f-94bc-c307041640d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xbdmd"
Nov 01 03:06:36.889308 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889084    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-multus-cni-dir\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.889308 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889102    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-etc-modprobe-d\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:06:36.889308 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889109    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7dbf71ad-8399-4a40-87d7-98746e830fa9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-t6tw7\" (UID: \"7dbf71ad-8399-4a40-87d7-98746e830fa9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-t6tw7"
Nov 01 03:06:36.889308 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:36.889137    4930 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-tls podName:9acc0734-3562-41fc-a05a-8c32cbf36443 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:37.389116826 +0000 UTC m=+1.912123406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-tls") pod "image-registry-6f44999c56-64k9p" (UID: "9acc0734-3562-41fc-a05a-8c32cbf36443") : secret "image-registry-tls" not found
Nov 01 03:06:36.889308 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889161    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/998efdf0-7719-4768-bfd0-3da04d85d108-registration-dir\") pod \"aws-ebs-csi-driver-node-rjtwk\" (UID: \"998efdf0-7719-4768-bfd0-3da04d85d108\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk"
Nov 01 03:06:36.889308 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:36.889172    4930 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 03:06:36.889308 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889173    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-etc-sysconfig\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:06:36.889308 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:36.889172    4930 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 03:06:36.889308 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889178    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-multus-cni-dir\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.889308 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889234    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-etc-sysconfig\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:06:36.889308 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:36.889246    4930 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dbf71ad-8399-4a40-87d7-98746e830fa9-networking-console-plugin-cert podName:7dbf71ad-8399-4a40-87d7-98746e830fa9 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:37.389230371 +0000 UTC m=+1.912236954 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7dbf71ad-8399-4a40-87d7-98746e830fa9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-t6tw7" (UID: "7dbf71ad-8399-4a40-87d7-98746e830fa9") : secret "networking-console-plugin-cert" not found
Nov 01 03:06:36.889308 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:36.889262    4930 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fecfe177-e63f-490f-94bc-c307041640d5-cluster-monitoring-operator-tls podName:fecfe177-e63f-490f-94bc-c307041640d5 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:37.389254861 +0000 UTC m=+1.912261441 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fecfe177-e63f-490f-94bc-c307041640d5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xbdmd" (UID: "fecfe177-e63f-490f-94bc-c307041640d5") : secret "cluster-monitoring-operator-tls" not found
Nov 01 03:06:36.889308 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889275    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-etc-sysctl-d\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:06:36.889308 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889295    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-metrics-certs\") pod \"router-default-ddbff58bc-5dlt4\" (UID: \"be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88\") " pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889319    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/998efdf0-7719-4768-bfd0-3da04d85d108-sys-fs\") pod \"aws-ebs-csi-driver-node-rjtwk\" (UID: \"998efdf0-7719-4768-bfd0-3da04d85d108\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889348    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-etc-openvswitch\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:36.889366    4930 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889373    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-var-lib-kubelet\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889386    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-etc-sysctl-d\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889399    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bbdccd97-7806-4669-a9cb-6aef70599c54-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gtk6f\" (UID: \"bbdccd97-7806-4669-a9cb-6aef70599c54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gtk6f"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:36.889414    4930 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-metrics-certs podName:be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:37.389402254 +0000 UTC m=+1.912408826 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-metrics-certs") pod "router-default-ddbff58bc-5dlt4" (UID: "be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88") : secret "router-metrics-certs-default" not found
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889416    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-etc-openvswitch\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889441    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-host-run-netns\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:36.889457    4930 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889462    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2vfgr\" (UID: \"caebbba6-44f3-40c1-95ce-b8a5a3f2667b\") " pod="openshift-insights/insights-runtime-extractor-2vfgr"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889469    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-etc-kubernetes\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889441    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/998efdf0-7719-4768-bfd0-3da04d85d108-sys-fs\") pod \"aws-ebs-csi-driver-node-rjtwk\" (UID: \"998efdf0-7719-4768-bfd0-3da04d85d108\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889453    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-var-lib-kubelet\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889489    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-host-run-netns\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889505    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-etc-kubernetes\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:36.889490    4930 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbdccd97-7806-4669-a9cb-6aef70599c54-samples-operator-tls podName:bbdccd97-7806-4669-a9cb-6aef70599c54 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:37.389479694 +0000 UTC m=+1.912486309 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bbdccd97-7806-4669-a9cb-6aef70599c54-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-gtk6f" (UID: "bbdccd97-7806-4669-a9cb-6aef70599c54") : secret "samples-operator-tls" not found
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889580    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a66c33f-29bc-4920-9d7a-0c45215514a0-metrics-certs\") pod \"network-metrics-daemon-mzkbj\" (UID: \"0a66c33f-29bc-4920-9d7a-0c45215514a0\") " pod="openshift-multus/network-metrics-daemon-mzkbj"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889608    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-host-run-ovn-kubernetes\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889657    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-host-run-ovn-kubernetes\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:36.889662    4930 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889637    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/998efdf0-7719-4768-bfd0-3da04d85d108-etc-selinux\") pod \"aws-ebs-csi-driver-node-rjtwk\" (UID: \"998efdf0-7719-4768-bfd0-3da04d85d108\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:36.889713    4930 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a66c33f-29bc-4920-9d7a-0c45215514a0-metrics-certs podName:0a66c33f-29bc-4920-9d7a-0c45215514a0 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:37.389702438 +0000 UTC m=+1.912709007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a66c33f-29bc-4920-9d7a-0c45215514a0-metrics-certs") pod "network-metrics-daemon-mzkbj" (UID: "0a66c33f-29bc-4920-9d7a-0c45215514a0") : secret "metrics-daemon-secret" not found
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889717    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/998efdf0-7719-4768-bfd0-3da04d85d108-etc-selinux\") pod \"aws-ebs-csi-driver-node-rjtwk\" (UID: \"998efdf0-7719-4768-bfd0-3da04d85d108\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889731    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a972ee28-29fa-4755-adcd-b40d476c22f0-dbus\") pod \"global-pull-secret-syncer-4bvb6\" (UID: \"a972ee28-29fa-4755-adcd-b40d476c22f0\") " pod="kube-system/global-pull-secret-syncer-4bvb6"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889747    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-crio-socket\") pod \"insights-runtime-extractor-2vfgr\" (UID: \"caebbba6-44f3-40c1-95ce-b8a5a3f2667b\") " pod="openshift-insights/insights-runtime-extractor-2vfgr"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889763    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-hostroot\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889778    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-etc-systemd\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889792    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d8dba89-5673-4cd3-a6ac-914765f228d6-system-cni-dir\") pod \"multus-additional-cni-plugins-9nhh6\" (UID: \"1d8dba89-5673-4cd3-a6ac-914765f228d6\") " pod="openshift-multus/multus-additional-cni-plugins-9nhh6"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889808    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/998efdf0-7719-4768-bfd0-3da04d85d108-device-dir\") pod \"aws-ebs-csi-driver-node-rjtwk\" (UID: \"998efdf0-7719-4768-bfd0-3da04d85d108\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889822    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-host-cni-netd\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889827    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-crio-socket\") pod \"insights-runtime-extractor-2vfgr\" (UID: \"caebbba6-44f3-40c1-95ce-b8a5a3f2667b\") " pod="openshift-insights/insights-runtime-extractor-2vfgr"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889836    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-system-cni-dir\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889835    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-hostroot\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889864    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a972ee28-29fa-4755-adcd-b40d476c22f0-dbus\") pod \"global-pull-secret-syncer-4bvb6\" (UID: \"a972ee28-29fa-4755-adcd-b40d476c22f0\") " pod="kube-system/global-pull-secret-syncer-4bvb6"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889871    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d8dba89-5673-4cd3-a6ac-914765f228d6-system-cni-dir\") pod \"multus-additional-cni-plugins-9nhh6\" (UID: \"1d8dba89-5673-4cd3-a6ac-914765f228d6\") " pod="openshift-multus/multus-additional-cni-plugins-9nhh6"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889880    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-system-cni-dir\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889875    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-host-cni-netd\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889894    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-run-systemd\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889886    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/998efdf0-7719-4768-bfd0-3da04d85d108-device-dir\") pod \"aws-ebs-csi-driver-node-rjtwk\" (UID: \"998efdf0-7719-4768-bfd0-3da04d85d108\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889866    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-run-systemd\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889920    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/482aebaa-cb7f-4be2-991a-7fb4cd801c7f-hosts-file\") pod \"node-resolver-77nf7\" (UID: \"482aebaa-cb7f-4be2-991a-7fb4cd801c7f\") " pod="openshift-dns/node-resolver-77nf7"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889935    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-run-openvswitch\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889896    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-etc-systemd\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889952    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/482aebaa-cb7f-4be2-991a-7fb4cd801c7f-hosts-file\") pod \"node-resolver-77nf7\" (UID: \"482aebaa-cb7f-4be2-991a-7fb4cd801c7f\") " pod="openshift-dns/node-resolver-77nf7"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889959    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-etc-sysctl-conf\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889980    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2vfgr\" (UID: \"caebbba6-44f3-40c1-95ce-b8a5a3f2667b\") " pod="openshift-insights/insights-runtime-extractor-2vfgr"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889980    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-run-openvswitch\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.889995    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1d8dba89-5673-4cd3-a6ac-914765f228d6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9nhh6\" (UID: \"1d8dba89-5673-4cd3-a6ac-914765f228d6\") " pod="openshift-multus/multus-additional-cni-plugins-9nhh6"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890014    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca0be51d-87a4-462d-9c83-fcfb8bf224e8-host-slash\") pod \"iptables-alerter-ksj2n\" (UID: \"ca0be51d-87a4-462d-9c83-fcfb8bf224e8\") " pod="openshift-network-operator/iptables-alerter-ksj2n"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890036    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-host-var-lib-cni-multus\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890048    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca0be51d-87a4-462d-9c83-fcfb8bf224e8-host-slash\") pod \"iptables-alerter-ksj2n\" (UID: \"ca0be51d-87a4-462d-9c83-fcfb8bf224e8\") " pod="openshift-network-operator/iptables-alerter-ksj2n"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890052    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-service-ca-bundle\") pod \"router-default-ddbff58bc-5dlt4\" (UID: \"be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88\") " pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:36.890064    4930 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890067    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1d8dba89-5673-4cd3-a6ac-914765f228d6-cnibin\") pod \"multus-additional-cni-plugins-9nhh6\" (UID: \"1d8dba89-5673-4cd3-a6ac-914765f228d6\") " pod="openshift-multus/multus-additional-cni-plugins-9nhh6"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890080    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1d8dba89-5673-4cd3-a6ac-914765f228d6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9nhh6\" (UID: \"1d8dba89-5673-4cd3-a6ac-914765f228d6\") " pod="openshift-multus/multus-additional-cni-plugins-9nhh6"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890087    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-systemd-units\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:36.890103    4930 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-insights-runtime-extractor-tls podName:caebbba6-44f3-40c1-95ce-b8a5a3f2667b nodeName:}" failed. No retries permitted until 2026-11-01 03:06:37.390092301 +0000 UTC m=+1.913098896 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2vfgr" (UID: "caebbba6-44f3-40c1-95ce-b8a5a3f2667b") : secret "insights-runtime-extractor-tls" not found
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890121    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-systemd-units\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890124    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-host-var-lib-cni-multus\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890128    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-log-socket\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890132    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1d8dba89-5673-4cd3-a6ac-914765f228d6-cnibin\") pod \"multus-additional-cni-plugins-9nhh6\" (UID: \"1d8dba89-5673-4cd3-a6ac-914765f228d6\") " pod="openshift-multus/multus-additional-cni-plugins-9nhh6"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890151    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/998efdf0-7719-4768-bfd0-3da04d85d108-socket-dir\") pod \"aws-ebs-csi-driver-node-rjtwk\" (UID: \"998efdf0-7719-4768-bfd0-3da04d85d108\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk"
Nov 01 03:06:36.890114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890166    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-var-lib-openvswitch\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890173    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-log-socket\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890172    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-etc-sysctl-conf\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890181    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-host-var-lib-kubelet\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:36.890200    4930 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-service-ca-bundle podName:be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:37.390186552 +0000 UTC m=+1.913193137 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-service-ca-bundle") pod "router-default-ddbff58bc-5dlt4" (UID: "be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88") : configmap references non-existent config key: service-ca.crt
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890215    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/998efdf0-7719-4768-bfd0-3da04d85d108-socket-dir\") pod \"aws-ebs-csi-driver-node-rjtwk\" (UID: \"998efdf0-7719-4768-bfd0-3da04d85d108\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk"
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890217    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-host-var-lib-kubelet\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890218    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-var-lib-openvswitch\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890236    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-host-run-multus-certs\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890262    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-etc-kubernetes\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890278    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-host-run-multus-certs\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890286    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-run\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890305    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-etc-kubernetes\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890311    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a972ee28-29fa-4755-adcd-b40d476c22f0-kubelet-config\") pod \"global-pull-secret-syncer-4bvb6\" (UID: \"a972ee28-29fa-4755-adcd-b40d476c22f0\") " pod="kube-system/global-pull-secret-syncer-4bvb6"
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890338    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-host-cni-bin\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890377    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a972ee28-29fa-4755-adcd-b40d476c22f0-kubelet-config\") pod \"global-pull-secret-syncer-4bvb6\" (UID: \"a972ee28-29fa-4755-adcd-b40d476c22f0\") " pod="kube-system/global-pull-secret-syncer-4bvb6"
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890382    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-host-cni-bin\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890377    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-sys\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890405    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-sys\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890376    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-run\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890424    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-tls\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890451    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-multus-socket-dir-parent\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890476    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-host-run-netns\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890504    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-host-var-lib-cni-bin\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890519    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-host-run-netns\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890523    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-multus-socket-dir-parent\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:36.890528    4930 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:36.890537    4930 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58bc694f-l7vm9: secret "image-registry-tls" not found
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890539    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-node-log\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890550    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f308b2b4-03e5-4d52-97ff-f1eb3ff951b1-host-var-lib-cni-bin\") pod \"multus-p5ght\" (UID: \"f308b2b4-03e5-4d52-97ff-f1eb3ff951b1\") " pod="openshift-multus/multus-p5ght"
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:36.890592    4930 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-tls podName:d28ff750-94d5-4eff-a8ef-58215bfb99e3 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:37.390571783 +0000 UTC m=+1.913578364 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-tls") pod "image-registry-58bc694f-l7vm9" (UID: "d28ff750-94d5-4eff-a8ef-58215bfb99e3") : secret "image-registry-tls" not found
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890605    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/998efdf0-7719-4768-bfd0-3da04d85d108-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rjtwk\" (UID: \"998efdf0-7719-4768-bfd0-3da04d85d108\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk"
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890606    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-node-log\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890639    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-host-kubelet\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890643    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/998efdf0-7719-4768-bfd0-3da04d85d108-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rjtwk\" (UID: \"998efdf0-7719-4768-bfd0-3da04d85d108\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjtwk"
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890648    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-host-kubelet\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890672    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19871ddf-e177-4442-b592-8f1c7df24aaf-metrics-tls\") pod \"dns-default-wps29\" (UID: \"19871ddf-e177-4442-b592-8f1c7df24aaf\") " pod="openshift-dns/dns-default-wps29"
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890699    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/73622c04-1428-4f7a-8faf-8eb17604f681-host\") pod \"node-ca-stpgr\" (UID: \"73622c04-1428-4f7a-8faf-8eb17604f681\") " pod="openshift-image-registry/node-ca-stpgr"
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:36.890709    4930 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890724    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-run-ovn\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:36.890745    4930 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19871ddf-e177-4442-b592-8f1c7df24aaf-metrics-tls podName:19871ddf-e177-4442-b592-8f1c7df24aaf nodeName:}" failed. No retries permitted until 2026-11-01 03:06:37.390734406 +0000 UTC m=+1.913740982 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19871ddf-e177-4442-b592-8f1c7df24aaf-metrics-tls") pod "dns-default-wps29" (UID: "19871ddf-e177-4442-b592-8f1c7df24aaf") : secret "dns-default-metrics-tls" not found
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890758    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/73622c04-1428-4f7a-8faf-8eb17604f681-host\") pod \"node-ca-stpgr\" (UID: \"73622c04-1428-4f7a-8faf-8eb17604f681\") " pod="openshift-image-registry/node-ca-stpgr"
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890761    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9e212e37-6c34-4b3c-941c-b38621f1f184-run-ovn\") pod \"ovnkube-node-h5ksq\" (UID: \"9e212e37-6c34-4b3c-941c-b38621f1f184\") " pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890774    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-lib-modules\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:06:36.892169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.890864    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8190fea3-4e71-4250-9a0f-3a1d78dd32db-lib-modules\") pod \"tuned-l892q\" (UID: \"8190fea3-4e71-4250-9a0f-3a1d78dd32db\") " pod="openshift-cluster-node-tuning-operator/tuned-l892q"
Nov 01 03:06:36.967071 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.967047    4930 scope.go:122] "RemoveContainer" containerID="3d57ddeb233cc6ef9b56e40c0268f52a146af9adc2a471d9dd2158d7b677263c"
Nov 01 03:06:36.981357 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.981338    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-wjvnt"
Nov 01 03:06:36.982292 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:36.982272    4930 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 03:06:37.394983 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:37.394911    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2vfgr\" (UID: \"caebbba6-44f3-40c1-95ce-b8a5a3f2667b\") " pod="openshift-insights/insights-runtime-extractor-2vfgr"
Nov 01 03:06:37.394983 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:37.394950    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-service-ca-bundle\") pod \"router-default-ddbff58bc-5dlt4\" (UID: \"be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88\") " pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:37.394983 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:37.394977    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-tls\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:37.395215 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:37.395015    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19871ddf-e177-4442-b592-8f1c7df24aaf-metrics-tls\") pod \"dns-default-wps29\" (UID: \"19871ddf-e177-4442-b592-8f1c7df24aaf\") " pod="openshift-dns/dns-default-wps29"
Nov 01 03:06:37.395215 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:37.395053    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-tls\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:37.395215 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:37.395081    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fecfe177-e63f-490f-94bc-c307041640d5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xbdmd\" (UID: \"fecfe177-e63f-490f-94bc-c307041640d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xbdmd"
Nov 01 03:06:37.395215 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:37.395089    4930 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 03:06:37.395215 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:37.395110    4930 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58bc694f-l7vm9: secret "image-registry-tls" not found
Nov 01 03:06:37.395215 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:37.395112    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7dbf71ad-8399-4a40-87d7-98746e830fa9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-t6tw7\" (UID: \"7dbf71ad-8399-4a40-87d7-98746e830fa9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-t6tw7"
Nov 01 03:06:37.395215 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:37.395138    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-metrics-certs\") pod \"router-default-ddbff58bc-5dlt4\" (UID: \"be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88\") " pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:37.395215 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:37.395142    4930 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 03:06:37.395215 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:37.395155    4930 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-tls podName:d28ff750-94d5-4eff-a8ef-58215bfb99e3 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:38.395136744 +0000 UTC m=+2.918143327 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-tls") pod "image-registry-58bc694f-l7vm9" (UID: "d28ff750-94d5-4eff-a8ef-58215bfb99e3") : secret "image-registry-tls" not found
Nov 01 03:06:37.395215 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:37.395179    4930 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 03:06:37.395215 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:37.395192    4930 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 03:06:37.395215 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:37.395208    4930 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 03:06:37.395215 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:37.395213    4930 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f44999c56-64k9p: secret "image-registry-tls" not found
Nov 01 03:06:37.395681 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:37.395224    4930 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 03:06:37.395681 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:37.395198    4930 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19871ddf-e177-4442-b592-8f1c7df24aaf-metrics-tls podName:19871ddf-e177-4442-b592-8f1c7df24aaf nodeName:}" failed. No retries permitted until 2026-11-01 03:06:38.395181987 +0000 UTC m=+2.918188572 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19871ddf-e177-4442-b592-8f1c7df24aaf-metrics-tls") pod "dns-default-wps29" (UID: "19871ddf-e177-4442-b592-8f1c7df24aaf") : secret "dns-default-metrics-tls" not found
Nov 01 03:06:37.395681 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:37.395181    4930 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 03:06:37.395681 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:37.395280    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bbdccd97-7806-4669-a9cb-6aef70599c54-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gtk6f\" (UID: \"bbdccd97-7806-4669-a9cb-6aef70599c54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gtk6f"
Nov 01 03:06:37.395681 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:37.395301    4930 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-insights-runtime-extractor-tls podName:caebbba6-44f3-40c1-95ce-b8a5a3f2667b nodeName:}" failed. No retries permitted until 2026-11-01 03:06:38.395293784 +0000 UTC m=+2.918300376 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2vfgr" (UID: "caebbba6-44f3-40c1-95ce-b8a5a3f2667b") : secret "insights-runtime-extractor-tls" not found
Nov 01 03:06:37.395681 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:37.395319    4930 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 03:06:37.395681 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:37.395323    4930 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-service-ca-bundle podName:be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:38.395314778 +0000 UTC m=+2.918321343 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-service-ca-bundle") pod "router-default-ddbff58bc-5dlt4" (UID: "be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88") : configmap references non-existent config key: service-ca.crt
Nov 01 03:06:37.395681 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:37.395336    4930 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fecfe177-e63f-490f-94bc-c307041640d5-cluster-monitoring-operator-tls podName:fecfe177-e63f-490f-94bc-c307041640d5 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:38.395331261 +0000 UTC m=+2.918337827 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fecfe177-e63f-490f-94bc-c307041640d5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xbdmd" (UID: "fecfe177-e63f-490f-94bc-c307041640d5") : secret "cluster-monitoring-operator-tls" not found
Nov 01 03:06:37.395681 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:37.395348    4930 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dbf71ad-8399-4a40-87d7-98746e830fa9-networking-console-plugin-cert podName:7dbf71ad-8399-4a40-87d7-98746e830fa9 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:38.395342306 +0000 UTC m=+2.918348872 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7dbf71ad-8399-4a40-87d7-98746e830fa9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-t6tw7" (UID: "7dbf71ad-8399-4a40-87d7-98746e830fa9") : secret "networking-console-plugin-cert" not found
Nov 01 03:06:37.395681 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:37.395364    4930 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-tls podName:9acc0734-3562-41fc-a05a-8c32cbf36443 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:38.395357254 +0000 UTC m=+2.918363820 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-tls") pod "image-registry-6f44999c56-64k9p" (UID: "9acc0734-3562-41fc-a05a-8c32cbf36443") : secret "image-registry-tls" not found
Nov 01 03:06:37.395681 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:37.395380    4930 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-metrics-certs podName:be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:38.395375346 +0000 UTC m=+2.918381912 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-metrics-certs") pod "router-default-ddbff58bc-5dlt4" (UID: "be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88") : secret "router-metrics-certs-default" not found
Nov 01 03:06:37.395681 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:37.395394    4930 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbdccd97-7806-4669-a9cb-6aef70599c54-samples-operator-tls podName:bbdccd97-7806-4669-a9cb-6aef70599c54 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:38.395389865 +0000 UTC m=+2.918396431 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bbdccd97-7806-4669-a9cb-6aef70599c54-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-gtk6f" (UID: "bbdccd97-7806-4669-a9cb-6aef70599c54") : secret "samples-operator-tls" not found
Nov 01 03:06:37.395681 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:37.395422    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a66c33f-29bc-4920-9d7a-0c45215514a0-metrics-certs\") pod \"network-metrics-daemon-mzkbj\" (UID: \"0a66c33f-29bc-4920-9d7a-0c45215514a0\") " pod="openshift-multus/network-metrics-daemon-mzkbj"
Nov 01 03:06:37.395681 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:37.395520    4930 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 03:06:37.395681 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:37.395546    4930 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a66c33f-29bc-4920-9d7a-0c45215514a0-metrics-certs podName:0a66c33f-29bc-4920-9d7a-0c45215514a0 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:38.395540322 +0000 UTC m=+2.918546917 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a66c33f-29bc-4920-9d7a-0c45215514a0-metrics-certs") pod "network-metrics-daemon-mzkbj" (UID: "0a66c33f-29bc-4920-9d7a-0c45215514a0") : secret "metrics-daemon-secret" not found
Nov 01 03:06:37.670419 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:37.670340    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jh955_f29ea61e-2a78-4c11-9566-f52497e135ff/console-operator/2.log"
Nov 01 03:06:37.670828 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:37.670812    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jh955_f29ea61e-2a78-4c11-9566-f52497e135ff/console-operator/1.log"
Nov 01 03:06:37.670868 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:37.670849    4930 generic.go:356] "Generic (PLEG): container finished" podID="f29ea61e-2a78-4c11-9566-f52497e135ff" containerID="dda9e1c09647771748c99e19f9a3cdf6ec75f8075249dafe90e9414299612b29" exitCode=255
Nov 01 03:06:37.670948 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:37.670930    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jh955" event={"ID":"f29ea61e-2a78-4c11-9566-f52497e135ff","Type":"ContainerDied","Data":"dda9e1c09647771748c99e19f9a3cdf6ec75f8075249dafe90e9414299612b29"}
Nov 01 03:06:37.671008 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:37.670997    4930 scope.go:122] "RemoveContainer" containerID="3d57ddeb233cc6ef9b56e40c0268f52a146af9adc2a471d9dd2158d7b677263c"
Nov 01 03:06:37.671380 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:37.671320    4930 scope.go:122] "RemoveContainer" containerID="dda9e1c09647771748c99e19f9a3cdf6ec75f8075249dafe90e9414299612b29"
Nov 01 03:06:37.671537 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:37.671514    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jh955_openshift-console-operator(f29ea61e-2a78-4c11-9566-f52497e135ff)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jh955" podUID="f29ea61e-2a78-4c11-9566-f52497e135ff"
Nov 01 03:06:37.672435 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:37.672409    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-wjvnt" event={"ID":"627e3af2-4a0d-4f19-91bc-760af67e01ad","Type":"ContainerStarted","Data":"cfc7c528951815b6894a2e8ba06d7bad3bdc3e3a6658fa7f118c2a3d2a210a0a"}
Nov 01 03:06:37.672533 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:37.672449    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-wjvnt" event={"ID":"627e3af2-4a0d-4f19-91bc-760af67e01ad","Type":"ContainerStarted","Data":"4377b50074c12c555f7dd80219a8b159edab09ba6fb89b227a4e218d4492f097"}
Nov 01 03:06:38.231066 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:38.231032    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:38.407390 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:38.407355    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2vfgr\" (UID: \"caebbba6-44f3-40c1-95ce-b8a5a3f2667b\") " pod="openshift-insights/insights-runtime-extractor-2vfgr"
Nov 01 03:06:38.407594 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:38.407403    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-service-ca-bundle\") pod \"router-default-ddbff58bc-5dlt4\" (UID: \"be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88\") " pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:38.407594 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:38.407442    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-tls\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:38.407594 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:38.407472    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19871ddf-e177-4442-b592-8f1c7df24aaf-metrics-tls\") pod \"dns-default-wps29\" (UID: \"19871ddf-e177-4442-b592-8f1c7df24aaf\") " pod="openshift-dns/dns-default-wps29"
Nov 01 03:06:38.407594 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:38.407499    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-tls\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:38.407594 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:38.407514    4930 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 03:06:38.407594 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:38.407524    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fecfe177-e63f-490f-94bc-c307041640d5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xbdmd\" (UID: \"fecfe177-e63f-490f-94bc-c307041640d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xbdmd"
Nov 01 03:06:38.407594 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:38.407551    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7dbf71ad-8399-4a40-87d7-98746e830fa9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-t6tw7\" (UID: \"7dbf71ad-8399-4a40-87d7-98746e830fa9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-t6tw7"
Nov 01 03:06:38.407906 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:38.407620    4930 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-service-ca-bundle podName:be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:40.40759641 +0000 UTC m=+4.930602995 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-service-ca-bundle") pod "router-default-ddbff58bc-5dlt4" (UID: "be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88") : configmap references non-existent config key: service-ca.crt
Nov 01 03:06:38.407906 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:38.407623    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-metrics-certs\") pod \"router-default-ddbff58bc-5dlt4\" (UID: \"be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88\") " pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:38.407906 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:38.407649    4930 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-insights-runtime-extractor-tls podName:caebbba6-44f3-40c1-95ce-b8a5a3f2667b nodeName:}" failed. No retries permitted until 2026-11-01 03:06:40.407638047 +0000 UTC m=+4.930644619 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2vfgr" (UID: "caebbba6-44f3-40c1-95ce-b8a5a3f2667b") : secret "insights-runtime-extractor-tls" not found
Nov 01 03:06:38.407906 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:38.407657    4930 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 01 03:06:38.407906 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:38.407676    4930 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 01 03:06:38.407906 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:38.407690    4930 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 03:06:38.407906 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:38.407700    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bbdccd97-7806-4669-a9cb-6aef70599c54-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gtk6f\" (UID: \"bbdccd97-7806-4669-a9cb-6aef70599c54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gtk6f"
Nov 01 03:06:38.407906 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:38.407706    4930 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f44999c56-64k9p: secret "image-registry-tls" not found
Nov 01 03:06:38.407906 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:38.407719    4930 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 03:06:38.407906 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:38.407739    4930 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dbf71ad-8399-4a40-87d7-98746e830fa9-networking-console-plugin-cert podName:7dbf71ad-8399-4a40-87d7-98746e830fa9 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:40.407715468 +0000 UTC m=+4.930722038 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7dbf71ad-8399-4a40-87d7-98746e830fa9-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-t6tw7" (UID: "7dbf71ad-8399-4a40-87d7-98746e830fa9") : secret "networking-console-plugin-cert" not found
Nov 01 03:06:38.407906 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:38.407743    4930 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 01 03:06:38.407906 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:38.407760    4930 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 03:06:38.407906 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:38.407771    4930 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58bc694f-l7vm9: secret "image-registry-tls" not found
Nov 01 03:06:38.407906 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:38.407792    4930 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 01 03:06:38.407906 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:38.407762    4930 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fecfe177-e63f-490f-94bc-c307041640d5-cluster-monitoring-operator-tls podName:fecfe177-e63f-490f-94bc-c307041640d5 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:40.40775172 +0000 UTC m=+4.930758287 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fecfe177-e63f-490f-94bc-c307041640d5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-xbdmd" (UID: "fecfe177-e63f-490f-94bc-c307041640d5") : secret "cluster-monitoring-operator-tls" not found
Nov 01 03:06:38.407906 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:38.407815    4930 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-tls podName:9acc0734-3562-41fc-a05a-8c32cbf36443 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:40.407806865 +0000 UTC m=+4.930813434 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-tls") pod "image-registry-6f44999c56-64k9p" (UID: "9acc0734-3562-41fc-a05a-8c32cbf36443") : secret "image-registry-tls" not found
Nov 01 03:06:38.407906 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:38.407832    4930 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19871ddf-e177-4442-b592-8f1c7df24aaf-metrics-tls podName:19871ddf-e177-4442-b592-8f1c7df24aaf nodeName:}" failed. No retries permitted until 2026-11-01 03:06:40.407822209 +0000 UTC m=+4.930828775 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19871ddf-e177-4442-b592-8f1c7df24aaf-metrics-tls") pod "dns-default-wps29" (UID: "19871ddf-e177-4442-b592-8f1c7df24aaf") : secret "dns-default-metrics-tls" not found
Nov 01 03:06:38.407906 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:38.407848    4930 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbdccd97-7806-4669-a9cb-6aef70599c54-samples-operator-tls podName:bbdccd97-7806-4669-a9cb-6aef70599c54 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:40.407840222 +0000 UTC m=+4.930846790 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bbdccd97-7806-4669-a9cb-6aef70599c54-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-gtk6f" (UID: "bbdccd97-7806-4669-a9cb-6aef70599c54") : secret "samples-operator-tls" not found
Nov 01 03:06:38.407906 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:38.407869    4930 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-tls podName:d28ff750-94d5-4eff-a8ef-58215bfb99e3 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:40.407860612 +0000 UTC m=+4.930867198 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-tls") pod "image-registry-58bc694f-l7vm9" (UID: "d28ff750-94d5-4eff-a8ef-58215bfb99e3") : secret "image-registry-tls" not found
Nov 01 03:06:38.407906 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:38.407889    4930 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-metrics-certs podName:be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:40.407880573 +0000 UTC m=+4.930887142 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-metrics-certs") pod "router-default-ddbff58bc-5dlt4" (UID: "be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88") : secret "router-metrics-certs-default" not found
Nov 01 03:06:38.408500 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:38.407925    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a66c33f-29bc-4920-9d7a-0c45215514a0-metrics-certs\") pod \"network-metrics-daemon-mzkbj\" (UID: \"0a66c33f-29bc-4920-9d7a-0c45215514a0\") " pod="openshift-multus/network-metrics-daemon-mzkbj"
Nov 01 03:06:38.408500 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:38.408005    4930 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 03:06:38.408500 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:38.408037    4930 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a66c33f-29bc-4920-9d7a-0c45215514a0-metrics-certs podName:0a66c33f-29bc-4920-9d7a-0c45215514a0 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:40.408027236 +0000 UTC m=+4.931033827 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a66c33f-29bc-4920-9d7a-0c45215514a0-metrics-certs") pod "network-metrics-daemon-mzkbj" (UID: "0a66c33f-29bc-4920-9d7a-0c45215514a0") : secret "metrics-daemon-secret" not found
Nov 01 03:06:38.478225 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:38.478195    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:38.667395 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:38.667346    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:38.678254 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:38.678229    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jh955_f29ea61e-2a78-4c11-9566-f52497e135ff/console-operator/2.log"
Nov 01 03:06:38.750075 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:38.750049    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:39.121404 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:39.121362    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:39.450760 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:39.450724    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:39.598150 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:39.598121    4930 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-jh955"
Nov 01 03:06:39.598487 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:39.598471    4930 scope.go:122] "RemoveContainer" containerID="dda9e1c09647771748c99e19f9a3cdf6ec75f8075249dafe90e9414299612b29"
Nov 01 03:06:39.598692 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:39.598663    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jh955_openshift-console-operator(f29ea61e-2a78-4c11-9566-f52497e135ff)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jh955" podUID="f29ea61e-2a78-4c11-9566-f52497e135ff"
Nov 01 03:06:39.683250 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:39.683221    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cbbs4" event={"ID":"f097ea60-e18f-47f0-b6a0-24e2b3a201d0","Type":"ContainerStarted","Data":"aff768970c356f231612346bcd0c541116309cb6d0864ba785b608ac3bb561d9"}
Nov 01 03:06:39.735577 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:39.735491    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:40.172110 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:40.172078    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:40.426471 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:40.426394    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fecfe177-e63f-490f-94bc-c307041640d5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xbdmd\" (UID: \"fecfe177-e63f-490f-94bc-c307041640d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xbdmd"
Nov 01 03:06:40.426471 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:40.426429    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7dbf71ad-8399-4a40-87d7-98746e830fa9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-t6tw7\" (UID: \"7dbf71ad-8399-4a40-87d7-98746e830fa9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-t6tw7"
Nov 01 03:06:40.426471 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:40.426451    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-metrics-certs\") pod \"router-default-ddbff58bc-5dlt4\" (UID: \"be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88\") " pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:40.426762 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:40.426476    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bbdccd97-7806-4669-a9cb-6aef70599c54-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gtk6f\" (UID: \"bbdccd97-7806-4669-a9cb-6aef70599c54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gtk6f"
Nov 01 03:06:40.426762 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:40.426496    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a66c33f-29bc-4920-9d7a-0c45215514a0-metrics-certs\") pod \"network-metrics-daemon-mzkbj\" (UID: \"0a66c33f-29bc-4920-9d7a-0c45215514a0\") " pod="openshift-multus/network-metrics-daemon-mzkbj"
Nov 01 03:06:40.426762 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:40.426604    4930 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 03:06:40.426762 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:40.426633    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2vfgr\" (UID: \"caebbba6-44f3-40c1-95ce-b8a5a3f2667b\") " pod="openshift-insights/insights-runtime-extractor-2vfgr"
Nov 01 03:06:40.426762 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:40.426649    4930 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a66c33f-29bc-4920-9d7a-0c45215514a0-metrics-certs podName:0a66c33f-29bc-4920-9d7a-0c45215514a0 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:44.426637181 +0000 UTC m=+8.949643772 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a66c33f-29bc-4920-9d7a-0c45215514a0-metrics-certs") pod "network-metrics-daemon-mzkbj" (UID: "0a66c33f-29bc-4920-9d7a-0c45215514a0") : secret "metrics-daemon-secret" not found
Nov 01 03:06:40.426762 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:40.426695    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-service-ca-bundle\") pod \"router-default-ddbff58bc-5dlt4\" (UID: \"be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88\") " pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:40.426762 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:40.426739    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-tls\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:40.427101 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:40.426779    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19871ddf-e177-4442-b592-8f1c7df24aaf-metrics-tls\") pod \"dns-default-wps29\" (UID: \"19871ddf-e177-4442-b592-8f1c7df24aaf\") " pod="openshift-dns/dns-default-wps29"
Nov 01 03:06:40.427101 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:40.426822    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-tls\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:40.427101 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:40.426866    4930 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 03:06:40.427101 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:40.426883    4930 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-service-ca-bundle podName:be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:44.426866506 +0000 UTC m=+8.949873088 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-service-ca-bundle") pod "router-default-ddbff58bc-5dlt4" (UID: "be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88") : configmap references non-existent config key: service-ca.crt
Nov 01 03:06:40.427101 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:40.426864    4930 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 03:06:40.427101 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:40.426914    4930 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-insights-runtime-extractor-tls podName:caebbba6-44f3-40c1-95ce-b8a5a3f2667b nodeName:}" failed. No retries permitted until 2026-11-01 03:06:44.426901607 +0000 UTC m=+8.949908191 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2vfgr" (UID: "caebbba6-44f3-40c1-95ce-b8a5a3f2667b") : secret "insights-runtime-extractor-tls" not found
Nov 01 03:06:40.427101 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:40.426915    4930 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 03:06:40.427101 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:40.426927    4930 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 01 03:06:40.427101 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:40.426941    4930 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58bc694f-l7vm9: secret "image-registry-tls" not found
Nov 01 03:06:40.427101 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:40.426972    4930 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-tls podName:d28ff750-94d5-4eff-a8ef-58215bfb99e3 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:44.426962718 +0000 UTC m=+8.949969315 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-tls") pod "image-registry-58bc694f-l7vm9" (UID: "d28ff750-94d5-4eff-a8ef-58215bfb99e3") : secret "image-registry-tls" not found
Nov 01 03:06:40.427101 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:40.426931    4930 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f44999c56-64k9p: secret "image-registry-tls" not found
Nov 01 03:06:40.427101 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:40.427002    4930 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-tls podName:9acc0734-3562-41fc-a05a-8c32cbf36443 nodeName:}" failed. No retries permitted until 2026-11-01 03:06:44.426992101 +0000 UTC m=+8.949998706 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-tls") pod "image-registry-6f44999c56-64k9p" (UID: "9acc0734-3562-41fc-a05a-8c32cbf36443") : secret "image-registry-tls" not found
Nov 01 03:06:40.429322 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:40.429293    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-metrics-certs\") pod \"router-default-ddbff58bc-5dlt4\" (UID: \"be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88\") " pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:40.429453 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:40.429432    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7dbf71ad-8399-4a40-87d7-98746e830fa9-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-t6tw7\" (UID: \"7dbf71ad-8399-4a40-87d7-98746e830fa9\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-t6tw7"
Nov 01 03:06:40.429502 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:40.429491    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bbdccd97-7806-4669-a9cb-6aef70599c54-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-gtk6f\" (UID: \"bbdccd97-7806-4669-a9cb-6aef70599c54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gtk6f"
Nov 01 03:06:40.429543 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:40.429437    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19871ddf-e177-4442-b592-8f1c7df24aaf-metrics-tls\") pod \"dns-default-wps29\" (UID: \"19871ddf-e177-4442-b592-8f1c7df24aaf\") " pod="openshift-dns/dns-default-wps29"
Nov 01 03:06:40.429684 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:40.429669    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fecfe177-e63f-490f-94bc-c307041640d5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-xbdmd\" (UID: \"fecfe177-e63f-490f-94bc-c307041640d5\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xbdmd"
Nov 01 03:06:40.566622 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:40.566598    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xbdmd"
Nov 01 03:06:40.568324 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:40.568301    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gtk6f"
Nov 01 03:06:40.569636 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:40.569619    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-t6tw7"
Nov 01 03:06:40.579396 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:40.579347    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wps29"
Nov 01 03:06:40.680698 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:40.680632    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:40.829810 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:40.829784    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-77nf7_482aebaa-cb7f-4be2-991a-7fb4cd801c7f/dns-node-resolver/0.log"
Nov 01 03:06:40.873062 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:40.873029    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:41.016290 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:41.016211    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-t6tw7"]
Nov 01 03:06:41.019213 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:41.019173    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dbf71ad_8399_4a40_87d7_98746e830fa9.slice/crio-9a6168035c99456ff66682a21b4e658ad862c4fa7cdb0bd214272fde257fb06c WatchSource:0}: Error finding container 9a6168035c99456ff66682a21b4e658ad862c4fa7cdb0bd214272fde257fb06c: Status 404 returned error can't find the container with id 9a6168035c99456ff66682a21b4e658ad862c4fa7cdb0bd214272fde257fb06c
Nov 01 03:06:41.063476 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:41.063453    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wps29"]
Nov 01 03:06:41.063773 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:41.063749    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:41.066494 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:41.066463    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19871ddf_e177_4442_b592_8f1c7df24aaf.slice/crio-2b97b33937dcd133801901f606a96605377cf8e837df5d08b9fe5bb602a7c99f WatchSource:0}: Error finding container 2b97b33937dcd133801901f606a96605377cf8e837df5d08b9fe5bb602a7c99f: Status 404 returned error can't find the container with id 2b97b33937dcd133801901f606a96605377cf8e837df5d08b9fe5bb602a7c99f
Nov 01 03:06:41.068080 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:41.068007    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gtk6f"]
Nov 01 03:06:41.068787 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:41.068766    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-xbdmd"]
Nov 01 03:06:41.101695 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:41.101669    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfecfe177_e63f_490f_94bc_c307041640d5.slice/crio-fe2d1ba7f7662399e29fcde769e3ef69e976393ca64d5dd6e0556732df016ec8 WatchSource:0}: Error finding container fe2d1ba7f7662399e29fcde769e3ef69e976393ca64d5dd6e0556732df016ec8: Status 404 returned error can't find the container with id fe2d1ba7f7662399e29fcde769e3ef69e976393ca64d5dd6e0556732df016ec8
Nov 01 03:06:41.360590 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:41.360540    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:41.372682 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:41.372661    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:41.372864 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:41.372850    4930 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 03:06:41.632371 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:41.632295    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:41.695767 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:41.695742    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wps29" event={"ID":"19871ddf-e177-4442-b592-8f1c7df24aaf","Type":"ContainerStarted","Data":"2b97b33937dcd133801901f606a96605377cf8e837df5d08b9fe5bb602a7c99f"}
Nov 01 03:06:41.696915 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:41.696896    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-t6tw7" event={"ID":"7dbf71ad-8399-4a40-87d7-98746e830fa9","Type":"ContainerStarted","Data":"9a6168035c99456ff66682a21b4e658ad862c4fa7cdb0bd214272fde257fb06c"}
Nov 01 03:06:41.698006 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:41.697973    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xbdmd" event={"ID":"fecfe177-e63f-490f-94bc-c307041640d5","Type":"ContainerStarted","Data":"fe2d1ba7f7662399e29fcde769e3ef69e976393ca64d5dd6e0556732df016ec8"}
Nov 01 03:06:41.699083 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:41.699055    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gtk6f" event={"ID":"bbdccd97-7806-4669-a9cb-6aef70599c54","Type":"ContainerStarted","Data":"232b2ad8ee15c537171207dafeb457a966fe727cedec9c1d9813c6af7711d15e"}
Nov 01 03:06:41.948756 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:41.948671    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:42.023311 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:42.023287    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-stpgr_73622c04-1428-4f7a-8faf-8eb17604f681/node-ca/0.log"
Nov 01 03:06:42.601982 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:42.601947    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:42.766278 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:42.766225    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:42.824457 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:42.824435    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zdrpv_a22c3f93-43f6-4908-9549-b966fb3c04e5/serve-healthcheck-canary/0.log"
Nov 01 03:06:43.024350 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:43.024324    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-cbbs4_f097ea60-e18f-47f0-b6a0-24e2b3a201d0/migrator/0.log"
Nov 01 03:06:43.155987 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:43.155910    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:43.340530 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:43.340495    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:43.386307 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:43.386279    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-62mxc"
Nov 01 03:06:43.434521 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:43.434428    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-chqll_14e2ac70-be2e-4e08-b093-e8a93d6f985a/kube-storage-version-migrator-operator/0.log"
Nov 01 03:06:43.547487 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:43.547448    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:43.712662 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:43.712631    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-t6tw7" event={"ID":"7dbf71ad-8399-4a40-87d7-98746e830fa9","Type":"ContainerStarted","Data":"fa6e73fc35f0b01b9483145320802acd196d727c6d334ff57a183ea584ba3b01"}
Nov 01 03:06:43.753513 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:43.753479    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:43.770663 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:43.770637    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jh955"
Nov 01 03:06:43.771077 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:43.771062    4930 scope.go:122] "RemoveContainer" containerID="dda9e1c09647771748c99e19f9a3cdf6ec75f8075249dafe90e9414299612b29"
Nov 01 03:06:43.771282 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:43.771261    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jh955_openshift-console-operator(f29ea61e-2a78-4c11-9566-f52497e135ff)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jh955" podUID="f29ea61e-2a78-4c11-9566-f52497e135ff"
Nov 01 03:06:44.465081 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:44.465043    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-tls\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:44.465081 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:44.465092    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a66c33f-29bc-4920-9d7a-0c45215514a0-metrics-certs\") pod \"network-metrics-daemon-mzkbj\" (UID: \"0a66c33f-29bc-4920-9d7a-0c45215514a0\") " pod="openshift-multus/network-metrics-daemon-mzkbj"
Nov 01 03:06:44.465638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:44.465123    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2vfgr\" (UID: \"caebbba6-44f3-40c1-95ce-b8a5a3f2667b\") " pod="openshift-insights/insights-runtime-extractor-2vfgr"
Nov 01 03:06:44.465638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:44.465144    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-service-ca-bundle\") pod \"router-default-ddbff58bc-5dlt4\" (UID: \"be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88\") " pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:44.465638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:44.465174    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-tls\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:44.465638 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:44.465365    4930 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 03:06:44.465638 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:06:44.465446    4930 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-insights-runtime-extractor-tls podName:caebbba6-44f3-40c1-95ce-b8a5a3f2667b nodeName:}" failed. No retries permitted until 2026-11-01 03:06:52.465424437 +0000 UTC m=+16.988431025 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2vfgr" (UID: "caebbba6-44f3-40c1-95ce-b8a5a3f2667b") : secret "insights-runtime-extractor-tls" not found
Nov 01 03:06:44.465933 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:44.465909    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88-service-ca-bundle\") pod \"router-default-ddbff58bc-5dlt4\" (UID: \"be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88\") " pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:44.467844 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:44.467814    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a66c33f-29bc-4920-9d7a-0c45215514a0-metrics-certs\") pod \"network-metrics-daemon-mzkbj\" (UID: \"0a66c33f-29bc-4920-9d7a-0c45215514a0\") " pod="openshift-multus/network-metrics-daemon-mzkbj"
Nov 01 03:06:44.467944 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:44.467865    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-tls\") pod \"image-registry-6f44999c56-64k9p\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") " pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:44.468187 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:44.468169    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-tls\") pod \"image-registry-58bc694f-l7vm9\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") " pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:44.469776 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:44.469752    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:44.469858 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:44.469841    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:44.475048 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:44.474872    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mzkbj"
Nov 01 03:06:44.479404 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:44.479384    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:44.718248 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:44.718161    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gtk6f" event={"ID":"bbdccd97-7806-4669-a9cb-6aef70599c54","Type":"ContainerStarted","Data":"1be1dc7bb5de25a17e2a422c28599ff9f31ac07f99c1e3e9cfd94745cb4febca"}
Nov 01 03:06:44.718388 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:44.718255    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-gtk6f" event={"ID":"bbdccd97-7806-4669-a9cb-6aef70599c54","Type":"ContainerStarted","Data":"490c1aa713f95fedb0704f63cc85b8f04b2814ce1936a57e2fa72852e35cddc7"}
Nov 01 03:06:44.720031 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:44.720006    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wps29" event={"ID":"19871ddf-e177-4442-b592-8f1c7df24aaf","Type":"ContainerStarted","Data":"8efd3971b2c51da073aeb59baf1ac60635f8b365d1ad205d91f7fa57139ac3e9"}
Nov 01 03:06:44.720151 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:44.720046    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wps29" event={"ID":"19871ddf-e177-4442-b592-8f1c7df24aaf","Type":"ContainerStarted","Data":"0b5d1223a9946d2f562efe1f53231247f4f11201255723ef30c9071dda9f0dcc"}
Nov 01 03:06:44.720151 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:44.720067    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wps29"
Nov 01 03:06:44.721567 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:44.721534    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-xbdmd" event={"ID":"fecfe177-e63f-490f-94bc-c307041640d5","Type":"ContainerStarted","Data":"1eb63efb0b0ca04ede84344b7bee4110a83120a9c39f91717afc88d1962e4d2a"}
Nov 01 03:06:44.831051 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:44.831020    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h5ksq"
Nov 01 03:06:44.893878 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:44.893822    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:44.957534 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:44.957508    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mzkbj"]
Nov 01 03:06:44.963924 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:44.963899    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a66c33f_29bc_4920_9d7a_0c45215514a0.slice/crio-387aff7604fdc7b367f4be624d279a36831a889829dfe68dc2c1e7a3f3b8ac63 WatchSource:0}: Error finding container 387aff7604fdc7b367f4be624d279a36831a889829dfe68dc2c1e7a3f3b8ac63: Status 404 returned error can't find the container with id 387aff7604fdc7b367f4be624d279a36831a889829dfe68dc2c1e7a3f3b8ac63
Nov 01 03:06:44.969792 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:44.969664    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f44999c56-64k9p"]
Nov 01 03:06:44.970414 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:44.970395    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-ddbff58bc-5dlt4"]
Nov 01 03:06:44.972457 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:44.972431    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe72b1cb_ee7a_4eca_a0f7_a5d0a39bec88.slice/crio-b517456b05b22c04783bdb3f51b6e2bbddc5a08f95f16d008e946db3738648b2 WatchSource:0}: Error finding container b517456b05b22c04783bdb3f51b6e2bbddc5a08f95f16d008e946db3738648b2: Status 404 returned error can't find the container with id b517456b05b22c04783bdb3f51b6e2bbddc5a08f95f16d008e946db3738648b2
Nov 01 03:06:44.973542 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:44.973523    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9acc0734_3562_41fc_a05a_8c32cbf36443.slice/crio-c6bb2bcfd9ff7bc86193195efd0178db2180634619f5610cc4719d1300d54444 WatchSource:0}: Error finding container c6bb2bcfd9ff7bc86193195efd0178db2180634619f5610cc4719d1300d54444: Status 404 returned error can't find the container with id c6bb2bcfd9ff7bc86193195efd0178db2180634619f5610cc4719d1300d54444
Nov 01 03:06:45.012735 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:45.012713    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58bc694f-l7vm9"]
Nov 01 03:06:45.015527 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:06:45.015502    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd28ff750_94d5_4eff_a8ef_58215bfb99e3.slice/crio-13cd6c4350c08ff118651b6c1c0104d0b7db262df770a1566a752e3c65cf5d0f WatchSource:0}: Error finding container 13cd6c4350c08ff118651b6c1c0104d0b7db262df770a1566a752e3c65cf5d0f: Status 404 returned error can't find the container with id 13cd6c4350c08ff118651b6c1c0104d0b7db262df770a1566a752e3c65cf5d0f
Nov 01 03:06:45.073589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:45.073116    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:45.247317 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:45.247233    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:45.424138 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:45.424098    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:45.594585 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:45.594540    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:45.731729 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:45.731690    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mzkbj" event={"ID":"0a66c33f-29bc-4920-9d7a-0c45215514a0","Type":"ContainerStarted","Data":"387aff7604fdc7b367f4be624d279a36831a889829dfe68dc2c1e7a3f3b8ac63"}
Nov 01 03:06:45.733519 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:45.733482    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58bc694f-l7vm9" event={"ID":"d28ff750-94d5-4eff-a8ef-58215bfb99e3","Type":"ContainerStarted","Data":"3bf8fb1d5e7258ce90d215b2eb4d26999f7b01bc5f09b012b7ec066f0f312ba5"}
Nov 01 03:06:45.733656 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:45.733530    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58bc694f-l7vm9" event={"ID":"d28ff750-94d5-4eff-a8ef-58215bfb99e3","Type":"ContainerStarted","Data":"13cd6c4350c08ff118651b6c1c0104d0b7db262df770a1566a752e3c65cf5d0f"}
Nov 01 03:06:45.735135 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:45.735101    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f44999c56-64k9p" event={"ID":"9acc0734-3562-41fc-a05a-8c32cbf36443","Type":"ContainerStarted","Data":"dda51a431a270884ff3b4b90dc88587096b6de83dcd58818eb805b8e91c9a07f"}
Nov 01 03:06:45.735220 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:45.735152    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f44999c56-64k9p" event={"ID":"9acc0734-3562-41fc-a05a-8c32cbf36443","Type":"ContainerStarted","Data":"c6bb2bcfd9ff7bc86193195efd0178db2180634619f5610cc4719d1300d54444"}
Nov 01 03:06:45.736779 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:45.736751    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-ddbff58bc-5dlt4" event={"ID":"be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88","Type":"ContainerStarted","Data":"cd2fe6a2053a7442d3e9d3b65bb8b922eb963743d07cc5a66779be306fcd83aa"}
Nov 01 03:06:45.736872 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:45.736800    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-ddbff58bc-5dlt4" event={"ID":"be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88","Type":"ContainerStarted","Data":"b517456b05b22c04783bdb3f51b6e2bbddc5a08f95f16d008e946db3738648b2"}
Nov 01 03:06:45.765929 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:45.765897    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:45.999172 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:45.999095    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:46.169847 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:46.169816    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:46.470978 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:46.470945    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:46.473366 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:46.473342    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:46.743649 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:46.743573    4930 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 03:06:46.743649 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:46.743573    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mzkbj" event={"ID":"0a66c33f-29bc-4920-9d7a-0c45215514a0","Type":"ContainerStarted","Data":"5db9870618a31b581e8e1a01c61fe1ba1cf549e3c5dbf93f0d3d3e3aa83bb14f"}
Nov 01 03:06:46.743649 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:46.743632    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mzkbj" event={"ID":"0a66c33f-29bc-4920-9d7a-0c45215514a0","Type":"ContainerStarted","Data":"d512d9ee0ee780523fc106015e5a080129def9b52133d8ae6f69e6ea9bf6fe67"}
Nov 01 03:06:46.744148 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:46.743593    4930 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 03:06:46.926828 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:46.926788    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:47.140477 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:47.140437    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:47.374892 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:47.374854    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:47.531811 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:47.531726    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:47.718873 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:47.718837    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:47.746528 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:47.746494    4930 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 03:06:48.613169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:48.613142    4930 scope.go:122] "RemoveContainer" containerID="dda9e1c09647771748c99e19f9a3cdf6ec75f8075249dafe90e9414299612b29"
Nov 01 03:06:48.752169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:48.752142    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jh955_f29ea61e-2a78-4c11-9566-f52497e135ff/console-operator/2.log"
Nov 01 03:06:48.752624 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:48.752196    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jh955" event={"ID":"f29ea61e-2a78-4c11-9566-f52497e135ff","Type":"ContainerStarted","Data":"0eed78e2865abbd9d4bb425d8b4f73d5d2632f6b9195e518dbb83661ec408e9d"}
Nov 01 03:06:48.752624 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:48.752509    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jh955"
Nov 01 03:06:48.753364 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:48.753342    4930 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-jh955 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.133.0.19:8443/readyz\": dial tcp 10.133.0.19:8443: connect: connection refused" start-of-body=
Nov 01 03:06:48.753417 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:48.753381    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-jh955" podUID="f29ea61e-2a78-4c11-9566-f52497e135ff" containerName="console-operator" probeResult="failure" output="Get \"https://10.133.0.19:8443/readyz\": dial tcp 10.133.0.19:8443: connect: connection refused"
Nov 01 03:06:48.798884 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:48.798860    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:06:48.954134 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:48.954061    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:49.148691 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:49.148657    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:49.307030 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:49.306964    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:06:49.338076 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:49.338043    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:49.541831 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:49.541793    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:49.734105 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:49.734074    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:49.759067 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:49.759042    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jh955"
Nov 01 03:06:50.297208 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:50.297167    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:50.480523 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:50.480489    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:50.689901 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:50.689865    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:50.958593 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:50.958495    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:50.959056 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:50.958664    4930 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 03:06:50.959768 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:50.959745    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-ddbff58bc-5dlt4"
Nov 01 03:06:51.143715 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:51.143685    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:51.293099 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:51.293019    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:51.445656 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:51.445614    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:52.530897 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:52.530867    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2vfgr\" (UID: \"caebbba6-44f3-40c1-95ce-b8a5a3f2667b\") " pod="openshift-insights/insights-runtime-extractor-2vfgr"
Nov 01 03:06:52.533179 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:52.533153    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/caebbba6-44f3-40c1-95ce-b8a5a3f2667b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2vfgr\" (UID: \"caebbba6-44f3-40c1-95ce-b8a5a3f2667b\") " pod="openshift-insights/insights-runtime-extractor-2vfgr"
Nov 01 03:06:52.580395 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:52.580369    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2vfgr"
Nov 01 03:06:52.950228 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:52.950196    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2vfgr"]
Nov 01 03:06:53.241990 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:53.241912    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:53.473391 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:53.473358    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:53.772188 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:53.772156    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2vfgr" event={"ID":"caebbba6-44f3-40c1-95ce-b8a5a3f2667b","Type":"ContainerStarted","Data":"95e06f9f3b6e613fad11c15fc0baa8e6531b03c669917af97141ed88d8151b75"}
Nov 01 03:06:53.772567 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:53.772208    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2vfgr" event={"ID":"caebbba6-44f3-40c1-95ce-b8a5a3f2667b","Type":"ContainerStarted","Data":"26fded91f0a357e656d44d1daafdb7abe079bb2e4d221e1c2c0de9d115664127"}
Nov 01 03:06:54.334720 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:54.334694    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:06:54.739538 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:54.739473    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wps29"
Nov 01 03:06:54.777443 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:54.777414    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2vfgr" event={"ID":"caebbba6-44f3-40c1-95ce-b8a5a3f2667b","Type":"ContainerStarted","Data":"6ce39998a5e25972b81b0b90a4731539abb31667b11ece3b43a1f70f796417e0"}
Nov 01 03:06:54.930568 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:54.930536    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-62ccn"
Nov 01 03:06:55.894330 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:55.894300    4930 patch_prober.go:29] interesting pod/image-registry-58bc694f-l7vm9 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 01 03:06:55.894699 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:55.894350    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-58bc694f-l7vm9" podUID="d28ff750-94d5-4eff-a8ef-58215bfb99e3" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:06:56.787201 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:56.787167    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2vfgr" event={"ID":"caebbba6-44f3-40c1-95ce-b8a5a3f2667b","Type":"ContainerStarted","Data":"9ec7e3a490deba3645a70235179171aabadd218a02b7c12ff80b8330e8cf1008"}
Nov 01 03:06:59.408966 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:59.408932    4930 patch_prober.go:29] interesting pod/image-registry-6f44999c56-64k9p container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 01 03:06:59.409352 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:06:59.408984    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6f44999c56-64k9p" podUID="9acc0734-3562-41fc-a05a-8c32cbf36443" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:07:00.357073 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:00.357038    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:03.389132 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:03.389106    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-62mxc"
Nov 01 03:07:03.894717 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:03.894664    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f44999c56-64k9p"]
Nov 01 03:07:03.900078 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:03.899996    4930 patch_prober.go:29] interesting pod/image-registry-6f44999c56-64k9p container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 01 03:07:03.900181 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:03.900089    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6f44999c56-64k9p" podUID="9acc0734-3562-41fc-a05a-8c32cbf36443" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:07:04.205229 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:04.205145    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:04.496803 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:04.496733    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:04.711932 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:04.711899    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:04.912862 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:04.912829    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:08.251393 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:08.251353    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:08.404829 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:08.404795    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:08.590839 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:08.590810    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:08.779421 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:08.779382    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:08.803146 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:08.803121    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:07:09.494640 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:09.494604    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:09.705030 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:09.704991    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:12.026067 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:12.026026    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:13.899578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:13.899530    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:07:13.915898 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:13.915855    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:14.613962 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:14.613925    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:14.840202 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:14.840169    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:15.140194 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:15.140148    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:15.380791 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:15.380746    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:15.613847 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:15.613810    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:15.677236 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:15.677207    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2sb2r"]
Nov 01 03:07:15.685079 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:15.685056    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2sb2r"
Nov 01 03:07:15.787454 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:15.787424    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 01 03:07:15.787684 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:15.787667    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 01 03:07:15.800007 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:15.799984    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d0d60814-1a5f-41d2-a8d2-6b274e743224-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2sb2r\" (UID: \"d0d60814-1a5f-41d2-a8d2-6b274e743224\") " pod="openshift-monitoring/node-exporter-2sb2r"
Nov 01 03:07:15.800104 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:15.800028    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tds4x\" (UniqueName: \"kubernetes.io/projected/d0d60814-1a5f-41d2-a8d2-6b274e743224-kube-api-access-tds4x\") pod \"node-exporter-2sb2r\" (UID: \"d0d60814-1a5f-41d2-a8d2-6b274e743224\") " pod="openshift-monitoring/node-exporter-2sb2r"
Nov 01 03:07:15.800158 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:15.800105    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0d60814-1a5f-41d2-a8d2-6b274e743224-metrics-client-ca\") pod \"node-exporter-2sb2r\" (UID: \"d0d60814-1a5f-41d2-a8d2-6b274e743224\") " pod="openshift-monitoring/node-exporter-2sb2r"
Nov 01 03:07:15.800158 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:15.800148    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d0d60814-1a5f-41d2-a8d2-6b274e743224-node-exporter-accelerators-collector-config\") pod \"node-exporter-2sb2r\" (UID: \"d0d60814-1a5f-41d2-a8d2-6b274e743224\") " pod="openshift-monitoring/node-exporter-2sb2r"
Nov 01 03:07:15.800244 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:15.800177    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d0d60814-1a5f-41d2-a8d2-6b274e743224-node-exporter-tls\") pod \"node-exporter-2sb2r\" (UID: \"d0d60814-1a5f-41d2-a8d2-6b274e743224\") " pod="openshift-monitoring/node-exporter-2sb2r"
Nov 01 03:07:15.800244 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:15.800206    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d0d60814-1a5f-41d2-a8d2-6b274e743224-node-exporter-textfile\") pod \"node-exporter-2sb2r\" (UID: \"d0d60814-1a5f-41d2-a8d2-6b274e743224\") " pod="openshift-monitoring/node-exporter-2sb2r"
Nov 01 03:07:15.800349 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:15.800279    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d0d60814-1a5f-41d2-a8d2-6b274e743224-node-exporter-wtmp\") pod \"node-exporter-2sb2r\" (UID: \"d0d60814-1a5f-41d2-a8d2-6b274e743224\") " pod="openshift-monitoring/node-exporter-2sb2r"
Nov 01 03:07:15.800349 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:15.800323    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d0d60814-1a5f-41d2-a8d2-6b274e743224-root\") pod \"node-exporter-2sb2r\" (UID: \"d0d60814-1a5f-41d2-a8d2-6b274e743224\") " pod="openshift-monitoring/node-exporter-2sb2r"
Nov 01 03:07:15.800430 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:15.800378    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0d60814-1a5f-41d2-a8d2-6b274e743224-sys\") pod \"node-exporter-2sb2r\" (UID: \"d0d60814-1a5f-41d2-a8d2-6b274e743224\") " pod="openshift-monitoring/node-exporter-2sb2r"
Nov 01 03:07:15.823363 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:15.823339    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:15.824795 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:15.824778    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 01 03:07:15.837341 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:15.837321    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jfdhs\""
Nov 01 03:07:15.837341 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:15.837344    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 01 03:07:15.901039 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:15.900970    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d0d60814-1a5f-41d2-a8d2-6b274e743224-root\") pod \"node-exporter-2sb2r\" (UID: \"d0d60814-1a5f-41d2-a8d2-6b274e743224\") " pod="openshift-monitoring/node-exporter-2sb2r"
Nov 01 03:07:15.901039 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:15.901002    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0d60814-1a5f-41d2-a8d2-6b274e743224-sys\") pod \"node-exporter-2sb2r\" (UID: \"d0d60814-1a5f-41d2-a8d2-6b274e743224\") " pod="openshift-monitoring/node-exporter-2sb2r"
Nov 01 03:07:15.901237 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:15.901039    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d0d60814-1a5f-41d2-a8d2-6b274e743224-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2sb2r\" (UID: \"d0d60814-1a5f-41d2-a8d2-6b274e743224\") " pod="openshift-monitoring/node-exporter-2sb2r"
Nov 01 03:07:15.901237 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:15.901064    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tds4x\" (UniqueName: \"kubernetes.io/projected/d0d60814-1a5f-41d2-a8d2-6b274e743224-kube-api-access-tds4x\") pod \"node-exporter-2sb2r\" (UID: \"d0d60814-1a5f-41d2-a8d2-6b274e743224\") " pod="openshift-monitoring/node-exporter-2sb2r"
Nov 01 03:07:15.901237 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:15.901082    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0d60814-1a5f-41d2-a8d2-6b274e743224-metrics-client-ca\") pod \"node-exporter-2sb2r\" (UID: \"d0d60814-1a5f-41d2-a8d2-6b274e743224\") " pod="openshift-monitoring/node-exporter-2sb2r"
Nov 01 03:07:15.901237 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:15.901097    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d0d60814-1a5f-41d2-a8d2-6b274e743224-node-exporter-accelerators-collector-config\") pod \"node-exporter-2sb2r\" (UID: \"d0d60814-1a5f-41d2-a8d2-6b274e743224\") " pod="openshift-monitoring/node-exporter-2sb2r"
Nov 01 03:07:15.901237 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:15.901101    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d0d60814-1a5f-41d2-a8d2-6b274e743224-root\") pod \"node-exporter-2sb2r\" (UID: \"d0d60814-1a5f-41d2-a8d2-6b274e743224\") " pod="openshift-monitoring/node-exporter-2sb2r"
Nov 01 03:07:15.901237 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:15.901124    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d0d60814-1a5f-41d2-a8d2-6b274e743224-node-exporter-tls\") pod \"node-exporter-2sb2r\" (UID: \"d0d60814-1a5f-41d2-a8d2-6b274e743224\") " pod="openshift-monitoring/node-exporter-2sb2r"
Nov 01 03:07:15.901237 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:15.901101    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0d60814-1a5f-41d2-a8d2-6b274e743224-sys\") pod \"node-exporter-2sb2r\" (UID: \"d0d60814-1a5f-41d2-a8d2-6b274e743224\") " pod="openshift-monitoring/node-exporter-2sb2r"
Nov 01 03:07:15.901237 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:15.901217    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d0d60814-1a5f-41d2-a8d2-6b274e743224-node-exporter-textfile\") pod \"node-exporter-2sb2r\" (UID: \"d0d60814-1a5f-41d2-a8d2-6b274e743224\") " pod="openshift-monitoring/node-exporter-2sb2r"
Nov 01 03:07:15.901732 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:15.901247    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d0d60814-1a5f-41d2-a8d2-6b274e743224-node-exporter-wtmp\") pod \"node-exporter-2sb2r\" (UID: \"d0d60814-1a5f-41d2-a8d2-6b274e743224\") " pod="openshift-monitoring/node-exporter-2sb2r"
Nov 01 03:07:15.901732 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:15.901414    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d0d60814-1a5f-41d2-a8d2-6b274e743224-node-exporter-wtmp\") pod \"node-exporter-2sb2r\" (UID: \"d0d60814-1a5f-41d2-a8d2-6b274e743224\") " pod="openshift-monitoring/node-exporter-2sb2r"
Nov 01 03:07:15.901732 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:15.901631    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d0d60814-1a5f-41d2-a8d2-6b274e743224-node-exporter-textfile\") pod \"node-exporter-2sb2r\" (UID: \"d0d60814-1a5f-41d2-a8d2-6b274e743224\") " pod="openshift-monitoring/node-exporter-2sb2r"
Nov 01 03:07:15.901732 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:15.901700    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d0d60814-1a5f-41d2-a8d2-6b274e743224-node-exporter-accelerators-collector-config\") pod \"node-exporter-2sb2r\" (UID: \"d0d60814-1a5f-41d2-a8d2-6b274e743224\") " pod="openshift-monitoring/node-exporter-2sb2r"
Nov 01 03:07:15.901932 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:15.901829    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0d60814-1a5f-41d2-a8d2-6b274e743224-metrics-client-ca\") pod \"node-exporter-2sb2r\" (UID: \"d0d60814-1a5f-41d2-a8d2-6b274e743224\") " pod="openshift-monitoring/node-exporter-2sb2r"
Nov 01 03:07:15.903345 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:15.903326    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d0d60814-1a5f-41d2-a8d2-6b274e743224-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2sb2r\" (UID: \"d0d60814-1a5f-41d2-a8d2-6b274e743224\") " pod="openshift-monitoring/node-exporter-2sb2r"
Nov 01 03:07:15.903523 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:15.903509    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d0d60814-1a5f-41d2-a8d2-6b274e743224-node-exporter-tls\") pod \"node-exporter-2sb2r\" (UID: \"d0d60814-1a5f-41d2-a8d2-6b274e743224\") " pod="openshift-monitoring/node-exporter-2sb2r"
Nov 01 03:07:15.961807 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:15.961787    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tds4x\" (UniqueName: \"kubernetes.io/projected/d0d60814-1a5f-41d2-a8d2-6b274e743224-kube-api-access-tds4x\") pod \"node-exporter-2sb2r\" (UID: \"d0d60814-1a5f-41d2-a8d2-6b274e743224\") " pod="openshift-monitoring/node-exporter-2sb2r"
Nov 01 03:07:15.985753 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:15.985728    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:16.027156 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:16.027129    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2sb2r"
Nov 01 03:07:16.066973 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:07:16.066933    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0d60814_1a5f_41d2_a8d2_6b274e743224.slice/crio-4c4deaaa00bd680ed724ad922cec768c0fb88ca1cf14058e9b5ee2adf38f15c4 WatchSource:0}: Error finding container 4c4deaaa00bd680ed724ad922cec768c0fb88ca1cf14058e9b5ee2adf38f15c4: Status 404 returned error can't find the container with id 4c4deaaa00bd680ed724ad922cec768c0fb88ca1cf14058e9b5ee2adf38f15c4
Nov 01 03:07:16.194535 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:16.194473    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:16.410184 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:16.410154    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-58bc694f-l7vm9"]
Nov 01 03:07:16.630456 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:16.630425    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:16.851443 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:16.851353    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:16.867652 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:16.867625    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2sb2r" event={"ID":"d0d60814-1a5f-41d2-a8d2-6b274e743224","Type":"ContainerStarted","Data":"4c4deaaa00bd680ed724ad922cec768c0fb88ca1cf14058e9b5ee2adf38f15c4"}
Nov 01 03:07:17.032386 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:17.032312    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:17.207877 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:17.207839    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:17.854839 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:17.854798    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:17.873261 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:17.873229    4930 generic.go:356] "Generic (PLEG): container finished" podID="d0d60814-1a5f-41d2-a8d2-6b274e743224" containerID="c38c9f041902689b07dee50395bfbd8471879e43534ad178ec2f0b700dedee7c" exitCode=0
Nov 01 03:07:17.873405 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:17.873273    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2sb2r" event={"ID":"d0d60814-1a5f-41d2-a8d2-6b274e743224","Type":"ContainerDied","Data":"c38c9f041902689b07dee50395bfbd8471879e43534ad178ec2f0b700dedee7c"}
Nov 01 03:07:18.883908 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:18.883872    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2sb2r" event={"ID":"d0d60814-1a5f-41d2-a8d2-6b274e743224","Type":"ContainerStarted","Data":"48f87b880ad4c3dea6b8be47f26871950f8f422a7ea8959fb16c26c541439797"}
Nov 01 03:07:18.884401 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:18.883942    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2sb2r" event={"ID":"d0d60814-1a5f-41d2-a8d2-6b274e743224","Type":"ContainerStarted","Data":"41d001187f3dcfe185b04797105dd5a1910a43316161660b5c7351df39157bdf"}
Nov 01 03:07:19.008329 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:19.008277    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2sb2r" podStartSLOduration=2.619252538 podStartE2EDuration="4.008258552s" totalImagesPullingTime="1.318449799s" totalInitContainerRuntime="70.556215ms" isStatefulPod=true podCreationTimestamp="2026-11-01 03:07:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:07:19.007274653 +0000 UTC m=+43.530281234" watchObservedRunningTime="2026-11-01 03:07:19.008258552 +0000 UTC m=+43.531265140"
Nov 01 03:07:19.120735 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:07:19.120694    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:07:19.832443 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:19.832404    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:20.058038 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:20.058003    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:20.301899 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:20.301862    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:20.560290 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:20.560184    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:21.417033 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:21.416999    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:21.591591 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:21.591527    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:21.752711 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:21.752613    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:23.780687 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:23.780651    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:23.985389 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:23.985353    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:24.925467 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:24.925434    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:25.717491 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:25.717450    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:26.001018 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:26.000921    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:27.358985 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:27.358942    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:27.554969 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:27.554933    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:27.827880 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:27.827795    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:27.991308 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:27.991274    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:28.917361 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:28.917327    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6f44999c56-64k9p" podUID="9acc0734-3562-41fc-a05a-8c32cbf36443" containerName="registry" containerID="cri-o://dda51a431a270884ff3b4b90dc88587096b6de83dcd58818eb805b8e91c9a07f" gracePeriod=30
Nov 01 03:07:29.178236 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:29.178180    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:07:29.313617 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:29.313587    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-kube-api-access-phjdn\" (UniqueName: \"kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-kube-api-access-phjdn\") pod \"9acc0734-3562-41fc-a05a-8c32cbf36443\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") "
Nov 01 03:07:29.313786 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:29.313631    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-tls\" (UniqueName: \"kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-tls\") pod \"9acc0734-3562-41fc-a05a-8c32cbf36443\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") "
Nov 01 03:07:29.313786 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:29.313658    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9acc0734-3562-41fc-a05a-8c32cbf36443-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9acc0734-3562-41fc-a05a-8c32cbf36443-ca-trust-extracted\") pod \"9acc0734-3562-41fc-a05a-8c32cbf36443\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") "
Nov 01 03:07:29.313786 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:29.313721    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9acc0734-3562-41fc-a05a-8c32cbf36443-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9acc0734-3562-41fc-a05a-8c32cbf36443-trusted-ca\") pod \"9acc0734-3562-41fc-a05a-8c32cbf36443\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") "
Nov 01 03:07:29.313786 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:29.313758    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-certificates\") pod \"9acc0734-3562-41fc-a05a-8c32cbf36443\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") "
Nov 01 03:07:29.313995 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:29.313793    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9acc0734-3562-41fc-a05a-8c32cbf36443-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9acc0734-3562-41fc-a05a-8c32cbf36443-installation-pull-secrets\") pod \"9acc0734-3562-41fc-a05a-8c32cbf36443\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") "
Nov 01 03:07:29.313995 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:29.313838    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-bound-sa-token\") pod \"9acc0734-3562-41fc-a05a-8c32cbf36443\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") "
Nov 01 03:07:29.313995 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:29.313905    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9acc0734-3562-41fc-a05a-8c32cbf36443-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9acc0734-3562-41fc-a05a-8c32cbf36443-image-registry-private-configuration\") pod \"9acc0734-3562-41fc-a05a-8c32cbf36443\" (UID: \"9acc0734-3562-41fc-a05a-8c32cbf36443\") "
Nov 01 03:07:29.314248 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:29.314214    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9acc0734-3562-41fc-a05a-8c32cbf36443-trusted-ca" pod "9acc0734-3562-41fc-a05a-8c32cbf36443" (UID: "9acc0734-3562-41fc-a05a-8c32cbf36443"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:07:29.314316 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:29.314231    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-certificates" pod "9acc0734-3562-41fc-a05a-8c32cbf36443" (UID: "9acc0734-3562-41fc-a05a-8c32cbf36443"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:07:29.316314 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:29.316239    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-tls" pod "9acc0734-3562-41fc-a05a-8c32cbf36443" (UID: "9acc0734-3562-41fc-a05a-8c32cbf36443"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:07:29.316314 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:29.316282    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-kube-api-access-phjdn" pod "9acc0734-3562-41fc-a05a-8c32cbf36443" (UID: "9acc0734-3562-41fc-a05a-8c32cbf36443"). InnerVolumeSpecName "kube-api-access-phjdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:07:29.316508 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:29.316478    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-bound-sa-token" pod "9acc0734-3562-41fc-a05a-8c32cbf36443" (UID: "9acc0734-3562-41fc-a05a-8c32cbf36443"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:07:29.316595 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:29.316547    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9acc0734-3562-41fc-a05a-8c32cbf36443-image-registry-private-configuration" pod "9acc0734-3562-41fc-a05a-8c32cbf36443" (UID: "9acc0734-3562-41fc-a05a-8c32cbf36443"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:07:29.316768 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:29.316746    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9acc0734-3562-41fc-a05a-8c32cbf36443-installation-pull-secrets" pod "9acc0734-3562-41fc-a05a-8c32cbf36443" (UID: "9acc0734-3562-41fc-a05a-8c32cbf36443"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:07:29.321011 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:29.320985    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9acc0734-3562-41fc-a05a-8c32cbf36443-ca-trust-extracted" pod "9acc0734-3562-41fc-a05a-8c32cbf36443" (UID: "9acc0734-3562-41fc-a05a-8c32cbf36443"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:07:29.414997 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:29.414970    4930 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9acc0734-3562-41fc-a05a-8c32cbf36443-image-registry-private-configuration\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:07:29.414997 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:29.414994    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phjdn\" (UniqueName: \"kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-kube-api-access-phjdn\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:07:29.415165 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:29.415005    4930 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-tls\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:07:29.415165 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:29.415014    4930 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9acc0734-3562-41fc-a05a-8c32cbf36443-ca-trust-extracted\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:07:29.415165 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:29.415023    4930 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9acc0734-3562-41fc-a05a-8c32cbf36443-trusted-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:07:29.415165 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:29.415031    4930 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9acc0734-3562-41fc-a05a-8c32cbf36443-registry-certificates\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:07:29.415165 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:29.415041    4930 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9acc0734-3562-41fc-a05a-8c32cbf36443-installation-pull-secrets\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:07:29.415165 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:29.415050    4930 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9acc0734-3562-41fc-a05a-8c32cbf36443-bound-sa-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:07:29.929676 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:29.929600    4930 generic.go:356] "Generic (PLEG): container finished" podID="9acc0734-3562-41fc-a05a-8c32cbf36443" containerID="dda51a431a270884ff3b4b90dc88587096b6de83dcd58818eb805b8e91c9a07f" exitCode=0
Nov 01 03:07:29.929676 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:29.929641    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f44999c56-64k9p" event={"ID":"9acc0734-3562-41fc-a05a-8c32cbf36443","Type":"ContainerDied","Data":"dda51a431a270884ff3b4b90dc88587096b6de83dcd58818eb805b8e91c9a07f"}
Nov 01 03:07:29.929676 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:29.929656    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f44999c56-64k9p"
Nov 01 03:07:29.930113 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:29.929705    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f44999c56-64k9p" event={"ID":"9acc0734-3562-41fc-a05a-8c32cbf36443","Type":"ContainerDied","Data":"c6bb2bcfd9ff7bc86193195efd0178db2180634619f5610cc4719d1300d54444"}
Nov 01 03:07:29.930113 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:29.929725    4930 scope.go:122] "RemoveContainer" containerID="dda51a431a270884ff3b4b90dc88587096b6de83dcd58818eb805b8e91c9a07f"
Nov 01 03:07:29.937788 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:29.937761    4930 scope.go:122] "RemoveContainer" containerID="dda51a431a270884ff3b4b90dc88587096b6de83dcd58818eb805b8e91c9a07f"
Nov 01 03:07:29.938042 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:07:29.938021    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dda51a431a270884ff3b4b90dc88587096b6de83dcd58818eb805b8e91c9a07f\": container with ID starting with dda51a431a270884ff3b4b90dc88587096b6de83dcd58818eb805b8e91c9a07f not found: ID does not exist" containerID="dda51a431a270884ff3b4b90dc88587096b6de83dcd58818eb805b8e91c9a07f"
Nov 01 03:07:29.938115 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:29.938053    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dda51a431a270884ff3b4b90dc88587096b6de83dcd58818eb805b8e91c9a07f"} err="failed to get container status \"dda51a431a270884ff3b4b90dc88587096b6de83dcd58818eb805b8e91c9a07f\": rpc error: code = NotFound desc = could not find container \"dda51a431a270884ff3b4b90dc88587096b6de83dcd58818eb805b8e91c9a07f\": container with ID starting with dda51a431a270884ff3b4b90dc88587096b6de83dcd58818eb805b8e91c9a07f not found: ID does not exist"
Nov 01 03:07:30.150603 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:30.150571    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f44999c56-64k9p"]
Nov 01 03:07:30.198345 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:30.198272    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6f44999c56-64k9p"]
Nov 01 03:07:30.677762 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:30.677729    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:30.972737 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:30.972649    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:31.618131 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:31.618099    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9acc0734-3562-41fc-a05a-8c32cbf36443" path="/var/lib/kubelet/pods/9acc0734-3562-41fc-a05a-8c32cbf36443/volumes"
Nov 01 03:07:31.978056 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:31.977965    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:32.123268 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:32.123229    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:38.229882 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:38.229846    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:38.376105 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:38.376072    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:41.429757 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:41.429697    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-58bc694f-l7vm9" podUID="d28ff750-94d5-4eff-a8ef-58215bfb99e3" containerName="registry" containerID="cri-o://3bf8fb1d5e7258ce90d215b2eb4d26999f7b01bc5f09b012b7ec066f0f312ba5" gracePeriod=30
Nov 01 03:07:41.681283 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:41.681231    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:07:41.706037 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:41.706015    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-kube-api-access-dtt7q\" (UniqueName: \"kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-kube-api-access-dtt7q\") pod \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") "
Nov 01 03:07:41.706143 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:41.706044    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d28ff750-94d5-4eff-a8ef-58215bfb99e3-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d28ff750-94d5-4eff-a8ef-58215bfb99e3-installation-pull-secrets\") pod \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") "
Nov 01 03:07:41.706143 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:41.706083    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-tls\" (UniqueName: \"kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-tls\") pod \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") "
Nov 01 03:07:41.706143 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:41.706127    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-bound-sa-token\") pod \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") "
Nov 01 03:07:41.706248 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:41.706153    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d28ff750-94d5-4eff-a8ef-58215bfb99e3-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d28ff750-94d5-4eff-a8ef-58215bfb99e3-image-registry-private-configuration\") pod \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") "
Nov 01 03:07:41.706248 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:41.706186    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d28ff750-94d5-4eff-a8ef-58215bfb99e3-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d28ff750-94d5-4eff-a8ef-58215bfb99e3-trusted-ca\") pod \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") "
Nov 01 03:07:41.706248 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:41.706213    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d28ff750-94d5-4eff-a8ef-58215bfb99e3-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d28ff750-94d5-4eff-a8ef-58215bfb99e3-ca-trust-extracted\") pod \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") "
Nov 01 03:07:41.706248 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:41.706243    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-certificates\") pod \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\" (UID: \"d28ff750-94d5-4eff-a8ef-58215bfb99e3\") "
Nov 01 03:07:41.707116 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:41.706702    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d28ff750-94d5-4eff-a8ef-58215bfb99e3-trusted-ca" pod "d28ff750-94d5-4eff-a8ef-58215bfb99e3" (UID: "d28ff750-94d5-4eff-a8ef-58215bfb99e3"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:07:41.707116 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:41.706765    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-certificates" pod "d28ff750-94d5-4eff-a8ef-58215bfb99e3" (UID: "d28ff750-94d5-4eff-a8ef-58215bfb99e3"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:07:41.708521 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:41.708494    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d28ff750-94d5-4eff-a8ef-58215bfb99e3-image-registry-private-configuration" pod "d28ff750-94d5-4eff-a8ef-58215bfb99e3" (UID: "d28ff750-94d5-4eff-a8ef-58215bfb99e3"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:07:41.708851 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:41.708828    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-bound-sa-token" pod "d28ff750-94d5-4eff-a8ef-58215bfb99e3" (UID: "d28ff750-94d5-4eff-a8ef-58215bfb99e3"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:07:41.708937 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:41.708848    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-tls" pod "d28ff750-94d5-4eff-a8ef-58215bfb99e3" (UID: "d28ff750-94d5-4eff-a8ef-58215bfb99e3"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:07:41.708937 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:41.708850    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d28ff750-94d5-4eff-a8ef-58215bfb99e3-installation-pull-secrets" pod "d28ff750-94d5-4eff-a8ef-58215bfb99e3" (UID: "d28ff750-94d5-4eff-a8ef-58215bfb99e3"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:07:41.709027 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:41.708966    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-kube-api-access-dtt7q" pod "d28ff750-94d5-4eff-a8ef-58215bfb99e3" (UID: "d28ff750-94d5-4eff-a8ef-58215bfb99e3"). InnerVolumeSpecName "kube-api-access-dtt7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:07:41.713201 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:41.713170    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28ff750-94d5-4eff-a8ef-58215bfb99e3-ca-trust-extracted" pod "d28ff750-94d5-4eff-a8ef-58215bfb99e3" (UID: "d28ff750-94d5-4eff-a8ef-58215bfb99e3"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:07:41.807188 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:41.807162    4930 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-tls\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:07:41.807188 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:41.807185    4930 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-bound-sa-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:07:41.807318 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:41.807196    4930 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d28ff750-94d5-4eff-a8ef-58215bfb99e3-image-registry-private-configuration\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:07:41.807318 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:41.807205    4930 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d28ff750-94d5-4eff-a8ef-58215bfb99e3-trusted-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:07:41.807318 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:41.807214    4930 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d28ff750-94d5-4eff-a8ef-58215bfb99e3-ca-trust-extracted\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:07:41.807318 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:41.807223    4930 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d28ff750-94d5-4eff-a8ef-58215bfb99e3-registry-certificates\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:07:41.807318 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:41.807232    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtt7q\" (UniqueName: \"kubernetes.io/projected/d28ff750-94d5-4eff-a8ef-58215bfb99e3-kube-api-access-dtt7q\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:07:41.807318 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:41.807241    4930 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d28ff750-94d5-4eff-a8ef-58215bfb99e3-installation-pull-secrets\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:07:41.976302 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:41.976222    4930 generic.go:356] "Generic (PLEG): container finished" podID="d28ff750-94d5-4eff-a8ef-58215bfb99e3" containerID="3bf8fb1d5e7258ce90d215b2eb4d26999f7b01bc5f09b012b7ec066f0f312ba5" exitCode=0
Nov 01 03:07:41.976302 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:41.976274    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58bc694f-l7vm9"
Nov 01 03:07:41.976302 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:41.976281    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58bc694f-l7vm9" event={"ID":"d28ff750-94d5-4eff-a8ef-58215bfb99e3","Type":"ContainerDied","Data":"3bf8fb1d5e7258ce90d215b2eb4d26999f7b01bc5f09b012b7ec066f0f312ba5"}
Nov 01 03:07:41.976492 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:41.976328    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58bc694f-l7vm9" event={"ID":"d28ff750-94d5-4eff-a8ef-58215bfb99e3","Type":"ContainerDied","Data":"13cd6c4350c08ff118651b6c1c0104d0b7db262df770a1566a752e3c65cf5d0f"}
Nov 01 03:07:41.976492 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:41.976354    4930 scope.go:122] "RemoveContainer" containerID="3bf8fb1d5e7258ce90d215b2eb4d26999f7b01bc5f09b012b7ec066f0f312ba5"
Nov 01 03:07:41.984791 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:41.984771    4930 scope.go:122] "RemoveContainer" containerID="3bf8fb1d5e7258ce90d215b2eb4d26999f7b01bc5f09b012b7ec066f0f312ba5"
Nov 01 03:07:41.985089 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:07:41.985073    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bf8fb1d5e7258ce90d215b2eb4d26999f7b01bc5f09b012b7ec066f0f312ba5\": container with ID starting with 3bf8fb1d5e7258ce90d215b2eb4d26999f7b01bc5f09b012b7ec066f0f312ba5 not found: ID does not exist" containerID="3bf8fb1d5e7258ce90d215b2eb4d26999f7b01bc5f09b012b7ec066f0f312ba5"
Nov 01 03:07:41.985134 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:41.985098    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bf8fb1d5e7258ce90d215b2eb4d26999f7b01bc5f09b012b7ec066f0f312ba5"} err="failed to get container status \"3bf8fb1d5e7258ce90d215b2eb4d26999f7b01bc5f09b012b7ec066f0f312ba5\": rpc error: code = NotFound desc = could not find container \"3bf8fb1d5e7258ce90d215b2eb4d26999f7b01bc5f09b012b7ec066f0f312ba5\": container with ID starting with 3bf8fb1d5e7258ce90d215b2eb4d26999f7b01bc5f09b012b7ec066f0f312ba5 not found: ID does not exist"
Nov 01 03:07:42.140145 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:42.140112    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-58bc694f-l7vm9"]
Nov 01 03:07:42.142945 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:42.142918    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-58bc694f-l7vm9"]
Nov 01 03:07:42.981419 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:42.981388    4930 generic.go:356] "Generic (PLEG): container finished" podID="45c60c0c-dcff-421d-a9f2-4dc53e7637c9" containerID="5d41367b4bede9a07ce8382f8a3e5be6d36791a840f9cac09cde85411eec36cd" exitCode=0
Nov 01 03:07:42.981851 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:42.981458    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbzc6" event={"ID":"45c60c0c-dcff-421d-a9f2-4dc53e7637c9","Type":"ContainerDied","Data":"5d41367b4bede9a07ce8382f8a3e5be6d36791a840f9cac09cde85411eec36cd"}
Nov 01 03:07:42.981851 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:42.981768    4930 scope.go:122] "RemoveContainer" containerID="5d41367b4bede9a07ce8382f8a3e5be6d36791a840f9cac09cde85411eec36cd"
Nov 01 03:07:43.621241 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:43.621209    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d28ff750-94d5-4eff-a8ef-58215bfb99e3" path="/var/lib/kubelet/pods/d28ff750-94d5-4eff-a8ef-58215bfb99e3/volumes"
Nov 01 03:07:43.987401 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:43.987325    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-cbzc6" event={"ID":"45c60c0c-dcff-421d-a9f2-4dc53e7637c9","Type":"ContainerStarted","Data":"39ab239228ad5c88d491fb847014cfa82cab4a0990fb7987bb9dc36cf14f483f"}
Nov 01 03:07:49.119983 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:07:49.119952    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:07:53.441291 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:53.441249    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:07:53.641222 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:07:53.641193    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:08:08.077271 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:08:08.077240    4930 generic.go:356] "Generic (PLEG): container finished" podID="14e2ac70-be2e-4e08-b093-e8a93d6f985a" containerID="59c7db500b6891ccf55e74cdb53331819b25a34a9739617c3dd553b787bcd588" exitCode=0
Nov 01 03:08:08.077647 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:08:08.077309    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chqll" event={"ID":"14e2ac70-be2e-4e08-b093-e8a93d6f985a","Type":"ContainerDied","Data":"59c7db500b6891ccf55e74cdb53331819b25a34a9739617c3dd553b787bcd588"}
Nov 01 03:08:08.077647 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:08:08.077605    4930 scope.go:122] "RemoveContainer" containerID="59c7db500b6891ccf55e74cdb53331819b25a34a9739617c3dd553b787bcd588"
Nov 01 03:08:09.083073 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:08:09.083036    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-chqll" event={"ID":"14e2ac70-be2e-4e08-b093-e8a93d6f985a","Type":"ContainerStarted","Data":"1776d12a17124bdeb4443663511311a4e63b3e25a69d794276217c9a9ed0694d"}
Nov 01 03:08:11.094813 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:08:11.094778    4930 generic.go:356] "Generic (PLEG): container finished" podID="eee1338d-e2a0-4a45-8f74-b81eb7d23019" containerID="2aded05819eae7519ba2246448f4f7ae3bc7689c18225f58266e75d32c3fa6e0" exitCode=0
Nov 01 03:08:11.095199 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:08:11.094851    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-48ssl" event={"ID":"eee1338d-e2a0-4a45-8f74-b81eb7d23019","Type":"ContainerDied","Data":"2aded05819eae7519ba2246448f4f7ae3bc7689c18225f58266e75d32c3fa6e0"}
Nov 01 03:08:11.095199 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:08:11.095167    4930 scope.go:122] "RemoveContainer" containerID="2aded05819eae7519ba2246448f4f7ae3bc7689c18225f58266e75d32c3fa6e0"
Nov 01 03:08:12.099955 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:08:12.099919    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-48ssl" event={"ID":"eee1338d-e2a0-4a45-8f74-b81eb7d23019","Type":"ContainerStarted","Data":"eae84b419507f4678457e5cce5d02f577afbe3d61022716f5ad5560fcb2ed97d"}
Nov 01 03:08:19.119938 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:08:19.119903    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:08:26.055849 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:08:26.055812    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:08:34.491653 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:08:34.491604    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:08:34.640426 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:08:34.640398    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:08:36.052572 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:08:36.052483    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:08:36.824319 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:08:36.824288    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:08:37.238409 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:08:37.238378    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:08:42.507984 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:08:42.507950    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:08:47.230046 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:08:47.229957    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:08:47.398018 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:08:47.397979    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:08:49.119460 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:08:49.119427    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:08:53.409309 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:08:53.409270    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:08:53.576373 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:08:53.576337    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:09:10.185068 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:09:10.185034    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:09:19.119965 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:09:19.119929    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:09:26.410656 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:09:26.410605    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:09:36.404851 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:09:36.404814    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:09:49.119160 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:09:49.119112    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:10:19.119695 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:10:19.119620    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:10:26.741811 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:26.741767    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:10:32.754631 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:32.754597    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:10:32.940062 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:32.940025    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:10:33.093853 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:33.093822    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:10:34.517630 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:34.517592    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:10:34.710305 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:34.710272    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:10:35.869058 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:35.869026    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:10:36.810382 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:36.810346    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:10:38.170988 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.170954    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sgtm5"]
Nov 01 03:10:38.171863 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.171298    4930 cpu_manager.go:414] "containerMap: removing container" podUID="d28ff750-94d5-4eff-a8ef-58215bfb99e3" containerName="registry"
Nov 01 03:10:38.171863 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.171314    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d28ff750-94d5-4eff-a8ef-58215bfb99e3" containerName="registry"
Nov 01 03:10:38.171863 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.171334    4930 cpu_manager.go:414] "containerMap: removing container" podUID="9acc0734-3562-41fc-a05a-8c32cbf36443" containerName="registry"
Nov 01 03:10:38.171863 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.171338    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9acc0734-3562-41fc-a05a-8c32cbf36443" containerName="registry"
Nov 01 03:10:38.171863 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.171390    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="d28ff750-94d5-4eff-a8ef-58215bfb99e3" containerName="registry"
Nov 01 03:10:38.171863 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.171401    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="9acc0734-3562-41fc-a05a-8c32cbf36443" containerName="registry"
Nov 01 03:10:38.174387 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.174372    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sgtm5"
Nov 01 03:10:38.226930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.226904    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 01 03:10:38.276265 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.276243    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 01 03:10:38.276619 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.276601    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-7k2nl\""
Nov 01 03:10:38.277573 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.277543    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sgtm5"]
Nov 01 03:10:38.278287 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.278268    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jz7b2"]
Nov 01 03:10:38.281416 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.281402    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jz7b2"
Nov 01 03:10:38.302873 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.302849    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6916682f-50fe-4ab1-a516-7648fdf1a5b8-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sgtm5\" (UID: \"6916682f-50fe-4ab1-a516-7648fdf1a5b8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sgtm5"
Nov 01 03:10:38.302970 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.302889    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r2lh\" (UniqueName: \"kubernetes.io/projected/6916682f-50fe-4ab1-a516-7648fdf1a5b8-kube-api-access-9r2lh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sgtm5\" (UID: \"6916682f-50fe-4ab1-a516-7648fdf1a5b8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sgtm5"
Nov 01 03:10:38.302970 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.302910    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6916682f-50fe-4ab1-a516-7648fdf1a5b8-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sgtm5\" (UID: \"6916682f-50fe-4ab1-a516-7648fdf1a5b8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sgtm5"
Nov 01 03:10:38.334045 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.334022    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-p5pv9\""
Nov 01 03:10:38.351241 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.351213    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jz7b2"]
Nov 01 03:10:38.386802 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.386780    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax8zjg"]
Nov 01 03:10:38.390129 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.390114    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax8zjg"
Nov 01 03:10:38.403832 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.403813    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6fa8951a-5ddb-40c9-962e-4c44f84c2974-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jz7b2\" (UID: \"6fa8951a-5ddb-40c9-962e-4c44f84c2974\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jz7b2"
Nov 01 03:10:38.403932 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.403842    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnhdr\" (UniqueName: \"kubernetes.io/projected/6fa8951a-5ddb-40c9-962e-4c44f84c2974-kube-api-access-gnhdr\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jz7b2\" (UID: \"6fa8951a-5ddb-40c9-962e-4c44f84c2974\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jz7b2"
Nov 01 03:10:38.403932 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.403865    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6916682f-50fe-4ab1-a516-7648fdf1a5b8-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sgtm5\" (UID: \"6916682f-50fe-4ab1-a516-7648fdf1a5b8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sgtm5"
Nov 01 03:10:38.403932 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.403893    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r2lh\" (UniqueName: \"kubernetes.io/projected/6916682f-50fe-4ab1-a516-7648fdf1a5b8-kube-api-access-9r2lh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sgtm5\" (UID: \"6916682f-50fe-4ab1-a516-7648fdf1a5b8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sgtm5"
Nov 01 03:10:38.404075 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.404053    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6916682f-50fe-4ab1-a516-7648fdf1a5b8-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sgtm5\" (UID: \"6916682f-50fe-4ab1-a516-7648fdf1a5b8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sgtm5"
Nov 01 03:10:38.404114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.404103    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6fa8951a-5ddb-40c9-962e-4c44f84c2974-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jz7b2\" (UID: \"6fa8951a-5ddb-40c9-962e-4c44f84c2974\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jz7b2"
Nov 01 03:10:38.404232 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.404215    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6916682f-50fe-4ab1-a516-7648fdf1a5b8-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sgtm5\" (UID: \"6916682f-50fe-4ab1-a516-7648fdf1a5b8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sgtm5"
Nov 01 03:10:38.404373 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.404358    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6916682f-50fe-4ab1-a516-7648fdf1a5b8-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sgtm5\" (UID: \"6916682f-50fe-4ab1-a516-7648fdf1a5b8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sgtm5"
Nov 01 03:10:38.444473 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.444424    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-dks4b\""
Nov 01 03:10:38.452402 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.452382    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax8zjg"]
Nov 01 03:10:38.469353 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.469325    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r2lh\" (UniqueName: \"kubernetes.io/projected/6916682f-50fe-4ab1-a516-7648fdf1a5b8-kube-api-access-9r2lh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sgtm5\" (UID: \"6916682f-50fe-4ab1-a516-7648fdf1a5b8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sgtm5"
Nov 01 03:10:38.472435 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.472414    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjq5xv"]
Nov 01 03:10:38.475829 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.475812    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjq5xv"
Nov 01 03:10:38.489871 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.489851    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sgtm5"
Nov 01 03:10:38.505305 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.505281    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnhdr\" (UniqueName: \"kubernetes.io/projected/6fa8951a-5ddb-40c9-962e-4c44f84c2974-kube-api-access-gnhdr\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jz7b2\" (UID: \"6fa8951a-5ddb-40c9-962e-4c44f84c2974\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jz7b2"
Nov 01 03:10:38.505405 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.505331    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e457c005-7a56-408f-a56f-93a1d29cbc3c-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax8zjg\" (UID: \"e457c005-7a56-408f-a56f-93a1d29cbc3c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax8zjg"
Nov 01 03:10:38.505473 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.505449    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e457c005-7a56-408f-a56f-93a1d29cbc3c-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax8zjg\" (UID: \"e457c005-7a56-408f-a56f-93a1d29cbc3c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax8zjg"
Nov 01 03:10:38.505548 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.505532    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6fa8951a-5ddb-40c9-962e-4c44f84c2974-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jz7b2\" (UID: \"6fa8951a-5ddb-40c9-962e-4c44f84c2974\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jz7b2"
Nov 01 03:10:38.505622 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.505598    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gwcw\" (UniqueName: \"kubernetes.io/projected/e457c005-7a56-408f-a56f-93a1d29cbc3c-kube-api-access-9gwcw\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax8zjg\" (UID: \"e457c005-7a56-408f-a56f-93a1d29cbc3c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax8zjg"
Nov 01 03:10:38.505682 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.505644    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6fa8951a-5ddb-40c9-962e-4c44f84c2974-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jz7b2\" (UID: \"6fa8951a-5ddb-40c9-962e-4c44f84c2974\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jz7b2"
Nov 01 03:10:38.505875 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.505855    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6fa8951a-5ddb-40c9-962e-4c44f84c2974-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jz7b2\" (UID: \"6fa8951a-5ddb-40c9-962e-4c44f84c2974\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jz7b2"
Nov 01 03:10:38.505985 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.505965    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6fa8951a-5ddb-40c9-962e-4c44f84c2974-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jz7b2\" (UID: \"6fa8951a-5ddb-40c9-962e-4c44f84c2974\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jz7b2"
Nov 01 03:10:38.528285 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.528264    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-c2bpq\""
Nov 01 03:10:38.537407 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.537213    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjq5xv"]
Nov 01 03:10:38.567782 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.567752    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnhdr\" (UniqueName: \"kubernetes.io/projected/6fa8951a-5ddb-40c9-962e-4c44f84c2974-kube-api-access-gnhdr\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jz7b2\" (UID: \"6fa8951a-5ddb-40c9-962e-4c44f84c2974\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jz7b2"
Nov 01 03:10:38.596102 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.596083    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jz7b2"
Nov 01 03:10:38.609223 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.607130    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/175d8311-cd8c-41d9-9fc5-8068a2471a31-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjq5xv\" (UID: \"175d8311-cd8c-41d9-9fc5-8068a2471a31\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjq5xv"
Nov 01 03:10:38.609223 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.607190    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t77hb\" (UniqueName: \"kubernetes.io/projected/175d8311-cd8c-41d9-9fc5-8068a2471a31-kube-api-access-t77hb\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjq5xv\" (UID: \"175d8311-cd8c-41d9-9fc5-8068a2471a31\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjq5xv"
Nov 01 03:10:38.609223 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.607232    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e457c005-7a56-408f-a56f-93a1d29cbc3c-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax8zjg\" (UID: \"e457c005-7a56-408f-a56f-93a1d29cbc3c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax8zjg"
Nov 01 03:10:38.609223 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.607270    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e457c005-7a56-408f-a56f-93a1d29cbc3c-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax8zjg\" (UID: \"e457c005-7a56-408f-a56f-93a1d29cbc3c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax8zjg"
Nov 01 03:10:38.609223 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.607333    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/175d8311-cd8c-41d9-9fc5-8068a2471a31-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjq5xv\" (UID: \"175d8311-cd8c-41d9-9fc5-8068a2471a31\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjq5xv"
Nov 01 03:10:38.609223 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.607384    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gwcw\" (UniqueName: \"kubernetes.io/projected/e457c005-7a56-408f-a56f-93a1d29cbc3c-kube-api-access-9gwcw\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax8zjg\" (UID: \"e457c005-7a56-408f-a56f-93a1d29cbc3c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax8zjg"
Nov 01 03:10:38.609223 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.608107    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e457c005-7a56-408f-a56f-93a1d29cbc3c-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax8zjg\" (UID: \"e457c005-7a56-408f-a56f-93a1d29cbc3c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax8zjg"
Nov 01 03:10:38.609223 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.608372    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e457c005-7a56-408f-a56f-93a1d29cbc3c-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax8zjg\" (UID: \"e457c005-7a56-408f-a56f-93a1d29cbc3c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax8zjg"
Nov 01 03:10:38.668455 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.668427    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gwcw\" (UniqueName: \"kubernetes.io/projected/e457c005-7a56-408f-a56f-93a1d29cbc3c-kube-api-access-9gwcw\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax8zjg\" (UID: \"e457c005-7a56-408f-a56f-93a1d29cbc3c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax8zjg"
Nov 01 03:10:38.705139 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.705069    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax8zjg"
Nov 01 03:10:38.708422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.708393    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/175d8311-cd8c-41d9-9fc5-8068a2471a31-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjq5xv\" (UID: \"175d8311-cd8c-41d9-9fc5-8068a2471a31\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjq5xv"
Nov 01 03:10:38.708527 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.708451    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/175d8311-cd8c-41d9-9fc5-8068a2471a31-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjq5xv\" (UID: \"175d8311-cd8c-41d9-9fc5-8068a2471a31\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjq5xv"
Nov 01 03:10:38.708527 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.708488    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t77hb\" (UniqueName: \"kubernetes.io/projected/175d8311-cd8c-41d9-9fc5-8068a2471a31-kube-api-access-t77hb\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjq5xv\" (UID: \"175d8311-cd8c-41d9-9fc5-8068a2471a31\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjq5xv"
Nov 01 03:10:38.708859 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.708841    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/175d8311-cd8c-41d9-9fc5-8068a2471a31-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjq5xv\" (UID: \"175d8311-cd8c-41d9-9fc5-8068a2471a31\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjq5xv"
Nov 01 03:10:38.708917 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.708884    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/175d8311-cd8c-41d9-9fc5-8068a2471a31-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjq5xv\" (UID: \"175d8311-cd8c-41d9-9fc5-8068a2471a31\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjq5xv"
Nov 01 03:10:38.769486 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.769457    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t77hb\" (UniqueName: \"kubernetes.io/projected/175d8311-cd8c-41d9-9fc5-8068a2471a31-kube-api-access-t77hb\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjq5xv\" (UID: \"175d8311-cd8c-41d9-9fc5-8068a2471a31\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjq5xv"
Nov 01 03:10:38.791339 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.791316    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjq5xv"
Nov 01 03:10:38.860705 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.860673    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sgtm5"]
Nov 01 03:10:38.864731 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:10:38.864702    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6916682f_50fe_4ab1_a516_7648fdf1a5b8.slice/crio-189fc5b66a8a8654787dda39a0cb3afd0b5fa472536341c19ee19ace75dee0a4 WatchSource:0}: Error finding container 189fc5b66a8a8654787dda39a0cb3afd0b5fa472536341c19ee19ace75dee0a4: Status 404 returned error can't find the container with id 189fc5b66a8a8654787dda39a0cb3afd0b5fa472536341c19ee19ace75dee0a4
Nov 01 03:10:38.971524 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:38.971469    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jz7b2"]
Nov 01 03:10:38.974387 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:10:38.974363    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fa8951a_5ddb_40c9_962e_4c44f84c2974.slice/crio-efb82ac6739ba63e1d80e063e33d22e85a1c0315ffeeb2002647f84a962c274a WatchSource:0}: Error finding container efb82ac6739ba63e1d80e063e33d22e85a1c0315ffeeb2002647f84a962c274a: Status 404 returned error can't find the container with id efb82ac6739ba63e1d80e063e33d22e85a1c0315ffeeb2002647f84a962c274a
Nov 01 03:10:39.074652 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:39.074526    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax8zjg"]
Nov 01 03:10:39.076979 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:10:39.076951    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode457c005_7a56_408f_a56f_93a1d29cbc3c.slice/crio-e838945874c73a20411060730db0e026c8eb9c2877287ba0389720160c980d23 WatchSource:0}: Error finding container e838945874c73a20411060730db0e026c8eb9c2877287ba0389720160c980d23: Status 404 returned error can't find the container with id e838945874c73a20411060730db0e026c8eb9c2877287ba0389720160c980d23
Nov 01 03:10:39.157994 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:39.157973    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjq5xv"]
Nov 01 03:10:39.160351 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:10:39.160331    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod175d8311_cd8c_41d9_9fc5_8068a2471a31.slice/crio-d384554a16cfb89d2c7c75594f58f854dcb7cbde75358a9c56475ef6e72dffe3 WatchSource:0}: Error finding container d384554a16cfb89d2c7c75594f58f854dcb7cbde75358a9c56475ef6e72dffe3: Status 404 returned error can't find the container with id d384554a16cfb89d2c7c75594f58f854dcb7cbde75358a9c56475ef6e72dffe3
Nov 01 03:10:39.619869 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:39.619837    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sgtm5" event={"ID":"6916682f-50fe-4ab1-a516-7648fdf1a5b8","Type":"ContainerStarted","Data":"189fc5b66a8a8654787dda39a0cb3afd0b5fa472536341c19ee19ace75dee0a4"}
Nov 01 03:10:39.620228 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:39.619884    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jz7b2" event={"ID":"6fa8951a-5ddb-40c9-962e-4c44f84c2974","Type":"ContainerStarted","Data":"efb82ac6739ba63e1d80e063e33d22e85a1c0315ffeeb2002647f84a962c274a"}
Nov 01 03:10:39.620228 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:39.619902    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjq5xv" event={"ID":"175d8311-cd8c-41d9-9fc5-8068a2471a31","Type":"ContainerStarted","Data":"d384554a16cfb89d2c7c75594f58f854dcb7cbde75358a9c56475ef6e72dffe3"}
Nov 01 03:10:39.620228 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:39.619919    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax8zjg" event={"ID":"e457c005-7a56-408f-a56f-93a1d29cbc3c","Type":"ContainerStarted","Data":"e838945874c73a20411060730db0e026c8eb9c2877287ba0389720160c980d23"}
Nov 01 03:10:39.988641 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:39.988549    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64472qbb"]
Nov 01 03:10:39.992721 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:39.992162    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64472qbb"
Nov 01 03:10:40.044959 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:40.044928    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-gw99p\""
Nov 01 03:10:40.096019 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:40.095993    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64472qbb"]
Nov 01 03:10:40.121491 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:40.120770    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1be18826-da5d-4c10-8599-382ebdc1d828-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64472qbb\" (UID: \"1be18826-da5d-4c10-8599-382ebdc1d828\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64472qbb"
Nov 01 03:10:40.121491 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:40.120829    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffj4g\" (UniqueName: \"kubernetes.io/projected/1be18826-da5d-4c10-8599-382ebdc1d828-kube-api-access-ffj4g\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64472qbb\" (UID: \"1be18826-da5d-4c10-8599-382ebdc1d828\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64472qbb"
Nov 01 03:10:40.121491 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:40.120894    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1be18826-da5d-4c10-8599-382ebdc1d828-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64472qbb\" (UID: \"1be18826-da5d-4c10-8599-382ebdc1d828\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64472qbb"
Nov 01 03:10:40.221999 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:40.221963    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1be18826-da5d-4c10-8599-382ebdc1d828-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64472qbb\" (UID: \"1be18826-da5d-4c10-8599-382ebdc1d828\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64472qbb"
Nov 01 03:10:40.222221 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:40.222044    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1be18826-da5d-4c10-8599-382ebdc1d828-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64472qbb\" (UID: \"1be18826-da5d-4c10-8599-382ebdc1d828\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64472qbb"
Nov 01 03:10:40.222221 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:40.222094    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffj4g\" (UniqueName: \"kubernetes.io/projected/1be18826-da5d-4c10-8599-382ebdc1d828-kube-api-access-ffj4g\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64472qbb\" (UID: \"1be18826-da5d-4c10-8599-382ebdc1d828\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64472qbb"
Nov 01 03:10:40.222822 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:40.222688    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1be18826-da5d-4c10-8599-382ebdc1d828-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64472qbb\" (UID: \"1be18826-da5d-4c10-8599-382ebdc1d828\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64472qbb"
Nov 01 03:10:40.222822 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:40.222774    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1be18826-da5d-4c10-8599-382ebdc1d828-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64472qbb\" (UID: \"1be18826-da5d-4c10-8599-382ebdc1d828\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64472qbb"
Nov 01 03:10:40.225155 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:40.225131    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6zcbg"]
Nov 01 03:10:40.229342 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:40.229322    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6zcbg"
Nov 01 03:10:40.282150 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:40.282072    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-q87nw\""
Nov 01 03:10:40.292389 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:40.292336    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffj4g\" (UniqueName: \"kubernetes.io/projected/1be18826-da5d-4c10-8599-382ebdc1d828-kube-api-access-ffj4g\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64472qbb\" (UID: \"1be18826-da5d-4c10-8599-382ebdc1d828\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64472qbb"
Nov 01 03:10:40.295025 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:40.295003    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6zcbg"]
Nov 01 03:10:40.312124 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:40.312084    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64472qbb"
Nov 01 03:10:40.322816 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:40.322790    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqlmb\" (UniqueName: \"kubernetes.io/projected/ac53872b-4bda-4070-ba1e-dc51073c3729-kube-api-access-lqlmb\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6zcbg\" (UID: \"ac53872b-4bda-4070-ba1e-dc51073c3729\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6zcbg"
Nov 01 03:10:40.322921 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:40.322833    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac53872b-4bda-4070-ba1e-dc51073c3729-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6zcbg\" (UID: \"ac53872b-4bda-4070-ba1e-dc51073c3729\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6zcbg"
Nov 01 03:10:40.322921 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:40.322865    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac53872b-4bda-4070-ba1e-dc51073c3729-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6zcbg\" (UID: \"ac53872b-4bda-4070-ba1e-dc51073c3729\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6zcbg"
Nov 01 03:10:40.424324 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:40.423819    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqlmb\" (UniqueName: \"kubernetes.io/projected/ac53872b-4bda-4070-ba1e-dc51073c3729-kube-api-access-lqlmb\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6zcbg\" (UID: \"ac53872b-4bda-4070-ba1e-dc51073c3729\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6zcbg"
Nov 01 03:10:40.424324 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:40.423886    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac53872b-4bda-4070-ba1e-dc51073c3729-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6zcbg\" (UID: \"ac53872b-4bda-4070-ba1e-dc51073c3729\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6zcbg"
Nov 01 03:10:40.424324 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:40.423940    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac53872b-4bda-4070-ba1e-dc51073c3729-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6zcbg\" (UID: \"ac53872b-4bda-4070-ba1e-dc51073c3729\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6zcbg"
Nov 01 03:10:40.424671 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:40.424329    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac53872b-4bda-4070-ba1e-dc51073c3729-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6zcbg\" (UID: \"ac53872b-4bda-4070-ba1e-dc51073c3729\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6zcbg"
Nov 01 03:10:40.424671 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:40.424546    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac53872b-4bda-4070-ba1e-dc51073c3729-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6zcbg\" (UID: \"ac53872b-4bda-4070-ba1e-dc51073c3729\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6zcbg"
Nov 01 03:10:40.486442 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:40.486419    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqlmb\" (UniqueName: \"kubernetes.io/projected/ac53872b-4bda-4070-ba1e-dc51073c3729-kube-api-access-lqlmb\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6zcbg\" (UID: \"ac53872b-4bda-4070-ba1e-dc51073c3729\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6zcbg"
Nov 01 03:10:40.551485 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:40.551134    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6zcbg"
Nov 01 03:10:40.712068 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:40.712034    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64472qbb"]
Nov 01 03:10:40.717381 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:10:40.717347    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1be18826_da5d_4c10_8599_382ebdc1d828.slice/crio-498ae61259bd248c7935124531d83076361a684dcae976f2ba6e5bb4d66eb358 WatchSource:0}: Error finding container 498ae61259bd248c7935124531d83076361a684dcae976f2ba6e5bb4d66eb358: Status 404 returned error can't find the container with id 498ae61259bd248c7935124531d83076361a684dcae976f2ba6e5bb4d66eb358
Nov 01 03:10:40.950383 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:40.950348    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6zcbg"]
Nov 01 03:10:40.956160 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:10:40.955313    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac53872b_4bda_4070_ba1e_dc51073c3729.slice/crio-56943cad8e0348dca7a1f89ea70ac2ca9be6e264972b4541c0115ccbd3525951 WatchSource:0}: Error finding container 56943cad8e0348dca7a1f89ea70ac2ca9be6e264972b4541c0115ccbd3525951: Status 404 returned error can't find the container with id 56943cad8e0348dca7a1f89ea70ac2ca9be6e264972b4541c0115ccbd3525951
Nov 01 03:10:41.635916 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:41.635823    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6zcbg" event={"ID":"ac53872b-4bda-4070-ba1e-dc51073c3729","Type":"ContainerStarted","Data":"56943cad8e0348dca7a1f89ea70ac2ca9be6e264972b4541c0115ccbd3525951"}
Nov 01 03:10:41.637816 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:41.637783    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64472qbb" event={"ID":"1be18826-da5d-4c10-8599-382ebdc1d828","Type":"ContainerStarted","Data":"498ae61259bd248c7935124531d83076361a684dcae976f2ba6e5bb4d66eb358"}
Nov 01 03:10:42.276573 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:42.276514    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:10:42.470905 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:42.470857    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:10:42.694391 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:42.694355    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh"]
Nov 01 03:10:42.698647 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:42.698109    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh"
Nov 01 03:10:42.752953 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:42.752198    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 01 03:10:42.788894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:42.788763    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-csbp4\""
Nov 01 03:10:42.793408 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:42.793351    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:10:42.805897 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:42.805837    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh"]
Nov 01 03:10:42.846844 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:42.846809    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26292f6-f496-4947-968d-55497ed63981-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh\" (UID: \"c26292f6-f496-4947-968d-55497ed63981\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh"
Nov 01 03:10:42.847003 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:42.846891    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c26292f6-f496-4947-968d-55497ed63981-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh\" (UID: \"c26292f6-f496-4947-968d-55497ed63981\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh"
Nov 01 03:10:42.847003 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:42.846915    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c26292f6-f496-4947-968d-55497ed63981-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh\" (UID: \"c26292f6-f496-4947-968d-55497ed63981\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh"
Nov 01 03:10:42.847003 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:42.846965    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c26292f6-f496-4947-968d-55497ed63981-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh\" (UID: \"c26292f6-f496-4947-968d-55497ed63981\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh"
Nov 01 03:10:42.847003 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:42.846989    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c26292f6-f496-4947-968d-55497ed63981-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh\" (UID: \"c26292f6-f496-4947-968d-55497ed63981\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh"
Nov 01 03:10:42.847224 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:42.847108    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c26292f6-f496-4947-968d-55497ed63981-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh\" (UID: \"c26292f6-f496-4947-968d-55497ed63981\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh"
Nov 01 03:10:42.847224 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:42.847151    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26292f6-f496-4947-968d-55497ed63981-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh\" (UID: \"c26292f6-f496-4947-968d-55497ed63981\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh"
Nov 01 03:10:42.847320 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:42.847223    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d66rn\" (UniqueName: \"kubernetes.io/projected/c26292f6-f496-4947-968d-55497ed63981-kube-api-access-d66rn\") pod \"openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh\" (UID: \"c26292f6-f496-4947-968d-55497ed63981\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh"
Nov 01 03:10:42.847320 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:42.847287    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c26292f6-f496-4947-968d-55497ed63981-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh\" (UID: \"c26292f6-f496-4947-968d-55497ed63981\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh"
Nov 01 03:10:42.948132 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:42.948046    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c26292f6-f496-4947-968d-55497ed63981-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh\" (UID: \"c26292f6-f496-4947-968d-55497ed63981\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh"
Nov 01 03:10:42.948132 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:42.948107    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c26292f6-f496-4947-968d-55497ed63981-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh\" (UID: \"c26292f6-f496-4947-968d-55497ed63981\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh"
Nov 01 03:10:42.948363 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:42.948143    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c26292f6-f496-4947-968d-55497ed63981-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh\" (UID: \"c26292f6-f496-4947-968d-55497ed63981\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh"
Nov 01 03:10:42.948363 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:42.948182    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26292f6-f496-4947-968d-55497ed63981-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh\" (UID: \"c26292f6-f496-4947-968d-55497ed63981\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh"
Nov 01 03:10:42.948363 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:42.948211    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d66rn\" (UniqueName: \"kubernetes.io/projected/c26292f6-f496-4947-968d-55497ed63981-kube-api-access-d66rn\") pod \"openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh\" (UID: \"c26292f6-f496-4947-968d-55497ed63981\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh"
Nov 01 03:10:42.948363 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:42.948245    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c26292f6-f496-4947-968d-55497ed63981-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh\" (UID: \"c26292f6-f496-4947-968d-55497ed63981\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh"
Nov 01 03:10:42.948363 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:42.948273    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26292f6-f496-4947-968d-55497ed63981-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh\" (UID: \"c26292f6-f496-4947-968d-55497ed63981\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh"
Nov 01 03:10:42.948363 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:42.948298    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c26292f6-f496-4947-968d-55497ed63981-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh\" (UID: \"c26292f6-f496-4947-968d-55497ed63981\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh"
Nov 01 03:10:42.948363 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:42.948327    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c26292f6-f496-4947-968d-55497ed63981-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh\" (UID: \"c26292f6-f496-4947-968d-55497ed63981\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh"
Nov 01 03:10:42.948868 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:42.948838    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c26292f6-f496-4947-968d-55497ed63981-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh\" (UID: \"c26292f6-f496-4947-968d-55497ed63981\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh"
Nov 01 03:10:42.950917 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:42.949969    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c26292f6-f496-4947-968d-55497ed63981-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh\" (UID: \"c26292f6-f496-4947-968d-55497ed63981\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh"
Nov 01 03:10:42.950917 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:42.950134    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26292f6-f496-4947-968d-55497ed63981-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh\" (UID: \"c26292f6-f496-4947-968d-55497ed63981\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh"
Nov 01 03:10:42.950917 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:42.950231    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26292f6-f496-4947-968d-55497ed63981-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh\" (UID: \"c26292f6-f496-4947-968d-55497ed63981\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh"
Nov 01 03:10:42.950917 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:42.950855    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c26292f6-f496-4947-968d-55497ed63981-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh\" (UID: \"c26292f6-f496-4947-968d-55497ed63981\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh"
Nov 01 03:10:42.958419 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:42.958395    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c26292f6-f496-4947-968d-55497ed63981-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh\" (UID: \"c26292f6-f496-4947-968d-55497ed63981\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh"
Nov 01 03:10:42.960321 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:42.960278    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c26292f6-f496-4947-968d-55497ed63981-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh\" (UID: \"c26292f6-f496-4947-968d-55497ed63981\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh"
Nov 01 03:10:42.992380 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:42.992293    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:10:43.015839 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:43.015804    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c26292f6-f496-4947-968d-55497ed63981-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh\" (UID: \"c26292f6-f496-4947-968d-55497ed63981\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh"
Nov 01 03:10:43.016124 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:43.016088    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"]
Nov 01 03:10:43.018308 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:43.018051    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d66rn\" (UniqueName: \"kubernetes.io/projected/c26292f6-f496-4947-968d-55497ed63981-kube-api-access-d66rn\") pod \"openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh\" (UID: \"c26292f6-f496-4947-968d-55497ed63981\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh"
Nov 01 03:10:43.021028 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:43.020997    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"
Nov 01 03:10:43.082745 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:43.082655    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"]
Nov 01 03:10:43.143893 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:43.143783    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-v7nmz\""
Nov 01 03:10:43.150653 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:43.150467    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e9b838-6a66-49b0-9aec-c43bfd9221ae-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zscr5\" (UID: \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"
Nov 01 03:10:43.150653 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:43.150531    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e9b838-6a66-49b0-9aec-c43bfd9221ae-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zscr5\" (UID: \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"
Nov 01 03:10:43.150653 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:43.150576    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts9nd\" (UniqueName: \"kubernetes.io/projected/24e9b838-6a66-49b0-9aec-c43bfd9221ae-kube-api-access-ts9nd\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zscr5\" (UID: \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"
Nov 01 03:10:43.150653 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:43.150611    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24e9b838-6a66-49b0-9aec-c43bfd9221ae-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zscr5\" (UID: \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"
Nov 01 03:10:43.150968 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:43.150678    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24e9b838-6a66-49b0-9aec-c43bfd9221ae-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zscr5\" (UID: \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"
Nov 01 03:10:43.150968 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:43.150783    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24e9b838-6a66-49b0-9aec-c43bfd9221ae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zscr5\" (UID: \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"
Nov 01 03:10:43.150968 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:43.150854    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24e9b838-6a66-49b0-9aec-c43bfd9221ae-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zscr5\" (UID: \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"
Nov 01 03:10:43.150968 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:43.150887    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24e9b838-6a66-49b0-9aec-c43bfd9221ae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zscr5\" (UID: \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"
Nov 01 03:10:43.150968 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:43.150910    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24e9b838-6a66-49b0-9aec-c43bfd9221ae-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zscr5\" (UID: \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"
Nov 01 03:10:43.151814 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:43.151707    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:10:43.251920 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:43.251834    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e9b838-6a66-49b0-9aec-c43bfd9221ae-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zscr5\" (UID: \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"
Nov 01 03:10:43.251920 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:43.251900    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e9b838-6a66-49b0-9aec-c43bfd9221ae-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zscr5\" (UID: \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"
Nov 01 03:10:43.252217 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:43.251932    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts9nd\" (UniqueName: \"kubernetes.io/projected/24e9b838-6a66-49b0-9aec-c43bfd9221ae-kube-api-access-ts9nd\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zscr5\" (UID: \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"
Nov 01 03:10:43.252217 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:43.251972    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24e9b838-6a66-49b0-9aec-c43bfd9221ae-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zscr5\" (UID: \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"
Nov 01 03:10:43.252217 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:43.251995    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24e9b838-6a66-49b0-9aec-c43bfd9221ae-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zscr5\" (UID: \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"
Nov 01 03:10:43.252217 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:43.252025    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24e9b838-6a66-49b0-9aec-c43bfd9221ae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zscr5\" (UID: \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"
Nov 01 03:10:43.252217 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:43.252067    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24e9b838-6a66-49b0-9aec-c43bfd9221ae-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zscr5\" (UID: \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"
Nov 01 03:10:43.252217 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:43.252098    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24e9b838-6a66-49b0-9aec-c43bfd9221ae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zscr5\" (UID: \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"
Nov 01 03:10:43.252217 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:43.252124    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24e9b838-6a66-49b0-9aec-c43bfd9221ae-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zscr5\" (UID: \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"
Nov 01 03:10:43.253015 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:43.252988    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e9b838-6a66-49b0-9aec-c43bfd9221ae-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zscr5\" (UID: \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"
Nov 01 03:10:43.253290 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:43.253268    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e9b838-6a66-49b0-9aec-c43bfd9221ae-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zscr5\" (UID: \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"
Nov 01 03:10:43.253928 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:43.253905    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24e9b838-6a66-49b0-9aec-c43bfd9221ae-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zscr5\" (UID: \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"
Nov 01 03:10:43.254182 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:43.254160    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24e9b838-6a66-49b0-9aec-c43bfd9221ae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zscr5\" (UID: \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"
Nov 01 03:10:43.254269 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:43.254159    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24e9b838-6a66-49b0-9aec-c43bfd9221ae-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zscr5\" (UID: \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"
Nov 01 03:10:43.264145 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:43.264116    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24e9b838-6a66-49b0-9aec-c43bfd9221ae-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zscr5\" (UID: \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"
Nov 01 03:10:43.273689 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:43.273641    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24e9b838-6a66-49b0-9aec-c43bfd9221ae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zscr5\" (UID: \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"
Nov 01 03:10:43.313959 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:43.313928    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:10:43.317275 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:43.317251    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh"
Nov 01 03:10:43.319158 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:43.319133    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts9nd\" (UniqueName: \"kubernetes.io/projected/24e9b838-6a66-49b0-9aec-c43bfd9221ae-kube-api-access-ts9nd\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zscr5\" (UID: \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"
Nov 01 03:10:43.325365 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:43.325283    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24e9b838-6a66-49b0-9aec-c43bfd9221ae-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zscr5\" (UID: \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"
Nov 01 03:10:43.357273 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:43.356992    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"
Nov 01 03:10:43.864739 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:43.864640    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh"]
Nov 01 03:10:43.870290 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:10:43.870213    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc26292f6_f496_4947_968d_55497ed63981.slice/crio-0a837a4840853dbb276bcab01045ff7a948b1a9df5dc5098bd980a71b9788200 WatchSource:0}: Error finding container 0a837a4840853dbb276bcab01045ff7a948b1a9df5dc5098bd980a71b9788200: Status 404 returned error can't find the container with id 0a837a4840853dbb276bcab01045ff7a948b1a9df5dc5098bd980a71b9788200
Nov 01 03:10:43.914514 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:43.914447    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"]
Nov 01 03:10:44.665222 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:44.665168    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5" event={"ID":"24e9b838-6a66-49b0-9aec-c43bfd9221ae","Type":"ContainerStarted","Data":"a026cb953a800c68f752276f2786b3b3a714f4ee360e5d30ec995c28ccba09c7"}
Nov 01 03:10:44.669945 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:44.669913    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh" event={"ID":"c26292f6-f496-4947-968d-55497ed63981","Type":"ContainerStarted","Data":"0a837a4840853dbb276bcab01045ff7a948b1a9df5dc5098bd980a71b9788200"}
Nov 01 03:10:46.357067 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:46.356966    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:10:49.156079 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:10:49.156036    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:10:52.505809 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:52.505771    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:10:52.506207 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:52.505858    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:10:52.506207 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:52.505898    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:10:52.756639 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:52.756574    4930 generic.go:356] "Generic (PLEG): container finished" podID="6916682f-50fe-4ab1-a516-7648fdf1a5b8" containerID="4de4697f2cd2eac1e4d233b9d35abb5b0972170652603e7f18f1960f29ce1acc" exitCode=0
Nov 01 03:10:52.756766 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:52.756668    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sgtm5" event={"ID":"6916682f-50fe-4ab1-a516-7648fdf1a5b8","Type":"ContainerDied","Data":"4de4697f2cd2eac1e4d233b9d35abb5b0972170652603e7f18f1960f29ce1acc"}
Nov 01 03:10:52.760583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:52.759943    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh" event={"ID":"c26292f6-f496-4947-968d-55497ed63981","Type":"ContainerStarted","Data":"5ee539c0d9126d819755754f96a251dec59808cca65ed9ff658f219f3943cb9c"}
Nov 01 03:10:53.016049 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:53.015949    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh" podStartSLOduration=2.383967475 podStartE2EDuration="11.015928748s" totalImagesPullingTime="8.631961273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:10:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:10:53.014882275 +0000 UTC m=+257.537888878" watchObservedRunningTime="2026-11-01 03:10:53.015928748 +0000 UTC m=+257.538935338"
Nov 01 03:10:53.078925 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:53.078883    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:10:53.079035 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:53.079015    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:10:53.079091 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:53.079069    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:10:53.318160 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:53.318094    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh"
Nov 01 03:10:53.324356 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:53.324131    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh"
Nov 01 03:10:53.767070 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:53.766959    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5" event={"ID":"24e9b838-6a66-49b0-9aec-c43bfd9221ae","Type":"ContainerStarted","Data":"bf4395ebac7bfe9834d0aa8d5038db9353479279937c9e6d106f6e515b87399f"}
Nov 01 03:10:53.769156 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:53.769126    4930 generic.go:356] "Generic (PLEG): container finished" podID="6fa8951a-5ddb-40c9-962e-4c44f84c2974" containerID="16e30a265b35d78edfd067c22510c2cc1067210b9a4df2eb7eb6efddf61956c2" exitCode=0
Nov 01 03:10:53.769277 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:53.769253    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jz7b2" event={"ID":"6fa8951a-5ddb-40c9-962e-4c44f84c2974","Type":"ContainerDied","Data":"16e30a265b35d78edfd067c22510c2cc1067210b9a4df2eb7eb6efddf61956c2"}
Nov 01 03:10:53.769870 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:53.769847    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh"
Nov 01 03:10:53.770899 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:53.770676    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh"
Nov 01 03:10:53.890708 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:53.889321    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5" podStartSLOduration=2.720766689 podStartE2EDuration="11.889304372s" totalImagesPullingTime="9.168537683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:10:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:10:53.888460759 +0000 UTC m=+258.411467349" watchObservedRunningTime="2026-11-01 03:10:53.889304372 +0000 UTC m=+258.412310960"
Nov 01 03:10:54.358361 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:54.358341    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"
Nov 01 03:10:54.364178 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:54.364156    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"
Nov 01 03:10:54.774243 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:54.774212    4930 generic.go:356] "Generic (PLEG): container finished" podID="e457c005-7a56-408f-a56f-93a1d29cbc3c" containerID="4b0e9d9427046874ec2c74213de8b1ce845f14731e9cffe2a89fc67676c4b199" exitCode=0
Nov 01 03:10:54.774659 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:54.774301    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax8zjg" event={"ID":"e457c005-7a56-408f-a56f-93a1d29cbc3c","Type":"ContainerDied","Data":"4b0e9d9427046874ec2c74213de8b1ce845f14731e9cffe2a89fc67676c4b199"}
Nov 01 03:10:54.776406 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:54.776388    4930 generic.go:356] "Generic (PLEG): container finished" podID="6916682f-50fe-4ab1-a516-7648fdf1a5b8" containerID="aa32a634b89fba1b63e0470902a1ca4d103ccc3e6dffa990f8685fd954c4c540" exitCode=0
Nov 01 03:10:54.776506 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:54.776478    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sgtm5" event={"ID":"6916682f-50fe-4ab1-a516-7648fdf1a5b8","Type":"ContainerDied","Data":"aa32a634b89fba1b63e0470902a1ca4d103ccc3e6dffa990f8685fd954c4c540"}
Nov 01 03:10:54.778124 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:54.778103    4930 generic.go:356] "Generic (PLEG): container finished" podID="1be18826-da5d-4c10-8599-382ebdc1d828" containerID="8994694803cd88cffa7b8ebfdde73ebfb51bedfe301317386b2e90ffe41dece6" exitCode=0
Nov 01 03:10:54.778233 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:54.778175    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64472qbb" event={"ID":"1be18826-da5d-4c10-8599-382ebdc1d828","Type":"ContainerDied","Data":"8994694803cd88cffa7b8ebfdde73ebfb51bedfe301317386b2e90ffe41dece6"}
Nov 01 03:10:54.779704 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:54.779681    4930 generic.go:356] "Generic (PLEG): container finished" podID="175d8311-cd8c-41d9-9fc5-8068a2471a31" containerID="9f7e36774daa7fb1fb6bf4c836e786f0b2f8805833f72567909f22b0919bd07f" exitCode=0
Nov 01 03:10:54.779813 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:54.779709    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjq5xv" event={"ID":"175d8311-cd8c-41d9-9fc5-8068a2471a31","Type":"ContainerDied","Data":"9f7e36774daa7fb1fb6bf4c836e786f0b2f8805833f72567909f22b0919bd07f"}
Nov 01 03:10:54.781346 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:54.781288    4930 generic.go:356] "Generic (PLEG): container finished" podID="ac53872b-4bda-4070-ba1e-dc51073c3729" containerID="bc891af0cfc5976547c4296d8c5b1dde26706469346b29d3c1865a9b4dc98332" exitCode=0
Nov 01 03:10:54.781417 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:54.781370    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6zcbg" event={"ID":"ac53872b-4bda-4070-ba1e-dc51073c3729","Type":"ContainerDied","Data":"bc891af0cfc5976547c4296d8c5b1dde26706469346b29d3c1865a9b4dc98332"}
Nov 01 03:10:54.781952 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:54.781806    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"
Nov 01 03:10:54.782736 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:54.782706    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"
Nov 01 03:10:55.787176 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:55.787144    4930 generic.go:356] "Generic (PLEG): container finished" podID="6fa8951a-5ddb-40c9-962e-4c44f84c2974" containerID="7db93f63daca80f5539c2d799456e0fc9aa1ed22d44c7dd19e6b42467d27b6e9" exitCode=0
Nov 01 03:10:55.787518 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:55.787224    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jz7b2" event={"ID":"6fa8951a-5ddb-40c9-962e-4c44f84c2974","Type":"ContainerDied","Data":"7db93f63daca80f5539c2d799456e0fc9aa1ed22d44c7dd19e6b42467d27b6e9"}
Nov 01 03:10:56.794798 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:56.794711    4930 generic.go:356] "Generic (PLEG): container finished" podID="175d8311-cd8c-41d9-9fc5-8068a2471a31" containerID="11bb51cd35db8ddd9d0432f9f76f567a5d60f9ca047f331cfd90d8f9db30953e" exitCode=0
Nov 01 03:10:56.795285 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:56.794856    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjq5xv" event={"ID":"175d8311-cd8c-41d9-9fc5-8068a2471a31","Type":"ContainerDied","Data":"11bb51cd35db8ddd9d0432f9f76f567a5d60f9ca047f331cfd90d8f9db30953e"}
Nov 01 03:10:56.797362 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:56.797289    4930 generic.go:356] "Generic (PLEG): container finished" podID="e457c005-7a56-408f-a56f-93a1d29cbc3c" containerID="e2650585b51c0a454e4688019ffc135cf895109e5acca25239204bbde47c6dc6" exitCode=0
Nov 01 03:10:56.797466 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:56.797377    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax8zjg" event={"ID":"e457c005-7a56-408f-a56f-93a1d29cbc3c","Type":"ContainerDied","Data":"e2650585b51c0a454e4688019ffc135cf895109e5acca25239204bbde47c6dc6"}
Nov 01 03:10:58.977805 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:58.977573    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:10:59.979235 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:10:59.977686    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-qwt9t"]
Nov 01 03:11:00.061938 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:00.061889    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:00.062959 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:00.062930    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-qwt9t"
Nov 01 03:11:00.105921 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:00.105847    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-qwt9t"]
Nov 01 03:11:00.153000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:00.152971    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 01 03:11:00.153151 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:00.153069    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 01 03:11:00.153151 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:00.153143    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 01 03:11:00.153250 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:00.153222    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-445ws\""
Nov 01 03:11:00.153858 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:00.153835    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 01 03:11:00.230363 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:00.230041    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/edd2b590-b470-4d56-8756-028d3aa2765f-tmp\") pod \"opendatahub-operator-controller-manager-576db5b58f-qwt9t\" (UID: \"edd2b590-b470-4d56-8756-028d3aa2765f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-qwt9t"
Nov 01 03:11:00.230363 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:00.230096    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/edd2b590-b470-4d56-8756-028d3aa2765f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-576db5b58f-qwt9t\" (UID: \"edd2b590-b470-4d56-8756-028d3aa2765f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-qwt9t"
Nov 01 03:11:00.230363 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:00.230166    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/edd2b590-b470-4d56-8756-028d3aa2765f-metrics-cert\") pod \"opendatahub-operator-controller-manager-576db5b58f-qwt9t\" (UID: \"edd2b590-b470-4d56-8756-028d3aa2765f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-qwt9t"
Nov 01 03:11:00.231080 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:00.230804    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/edd2b590-b470-4d56-8756-028d3aa2765f-webhook-cert\") pod \"opendatahub-operator-controller-manager-576db5b58f-qwt9t\" (UID: \"edd2b590-b470-4d56-8756-028d3aa2765f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-qwt9t"
Nov 01 03:11:00.231080 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:00.230903    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpc8j\" (UniqueName: \"kubernetes.io/projected/edd2b590-b470-4d56-8756-028d3aa2765f-kube-api-access-kpc8j\") pod \"opendatahub-operator-controller-manager-576db5b58f-qwt9t\" (UID: \"edd2b590-b470-4d56-8756-028d3aa2765f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-qwt9t"
Nov 01 03:11:00.231080 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:00.230944    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/edd2b590-b470-4d56-8756-028d3aa2765f-manifests\") pod \"opendatahub-operator-controller-manager-576db5b58f-qwt9t\" (UID: \"edd2b590-b470-4d56-8756-028d3aa2765f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-qwt9t"
Nov 01 03:11:00.269022 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:00.268986    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 01 03:11:00.332683 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:00.331408    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpc8j\" (UniqueName: \"kubernetes.io/projected/edd2b590-b470-4d56-8756-028d3aa2765f-kube-api-access-kpc8j\") pod \"opendatahub-operator-controller-manager-576db5b58f-qwt9t\" (UID: \"edd2b590-b470-4d56-8756-028d3aa2765f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-qwt9t"
Nov 01 03:11:00.332683 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:00.331465    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/edd2b590-b470-4d56-8756-028d3aa2765f-manifests\") pod \"opendatahub-operator-controller-manager-576db5b58f-qwt9t\" (UID: \"edd2b590-b470-4d56-8756-028d3aa2765f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-qwt9t"
Nov 01 03:11:00.332683 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:00.331516    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/edd2b590-b470-4d56-8756-028d3aa2765f-tmp\") pod \"opendatahub-operator-controller-manager-576db5b58f-qwt9t\" (UID: \"edd2b590-b470-4d56-8756-028d3aa2765f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-qwt9t"
Nov 01 03:11:00.332683 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:00.331542    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/edd2b590-b470-4d56-8756-028d3aa2765f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-576db5b58f-qwt9t\" (UID: \"edd2b590-b470-4d56-8756-028d3aa2765f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-qwt9t"
Nov 01 03:11:00.332683 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:00.331616    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/edd2b590-b470-4d56-8756-028d3aa2765f-metrics-cert\") pod \"opendatahub-operator-controller-manager-576db5b58f-qwt9t\" (UID: \"edd2b590-b470-4d56-8756-028d3aa2765f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-qwt9t"
Nov 01 03:11:00.332683 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:00.331644    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/edd2b590-b470-4d56-8756-028d3aa2765f-webhook-cert\") pod \"opendatahub-operator-controller-manager-576db5b58f-qwt9t\" (UID: \"edd2b590-b470-4d56-8756-028d3aa2765f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-qwt9t"
Nov 01 03:11:00.333490 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:00.333441    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/edd2b590-b470-4d56-8756-028d3aa2765f-manifests\") pod \"opendatahub-operator-controller-manager-576db5b58f-qwt9t\" (UID: \"edd2b590-b470-4d56-8756-028d3aa2765f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-qwt9t"
Nov 01 03:11:00.336384 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:00.336294    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/edd2b590-b470-4d56-8756-028d3aa2765f-metrics-cert\") pod \"opendatahub-operator-controller-manager-576db5b58f-qwt9t\" (UID: \"edd2b590-b470-4d56-8756-028d3aa2765f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-qwt9t"
Nov 01 03:11:00.338003 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:00.337974    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/edd2b590-b470-4d56-8756-028d3aa2765f-webhook-cert\") pod \"opendatahub-operator-controller-manager-576db5b58f-qwt9t\" (UID: \"edd2b590-b470-4d56-8756-028d3aa2765f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-qwt9t"
Nov 01 03:11:00.338445 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:00.338400    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/edd2b590-b470-4d56-8756-028d3aa2765f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-576db5b58f-qwt9t\" (UID: \"edd2b590-b470-4d56-8756-028d3aa2765f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-qwt9t"
Nov 01 03:11:00.341800 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:00.341776    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/edd2b590-b470-4d56-8756-028d3aa2765f-tmp\") pod \"opendatahub-operator-controller-manager-576db5b58f-qwt9t\" (UID: \"edd2b590-b470-4d56-8756-028d3aa2765f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-qwt9t"
Nov 01 03:11:00.366644 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:00.366545    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:00.403196 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:00.403133    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpc8j\" (UniqueName: \"kubernetes.io/projected/edd2b590-b470-4d56-8756-028d3aa2765f-kube-api-access-kpc8j\") pod \"opendatahub-operator-controller-manager-576db5b58f-qwt9t\" (UID: \"edd2b590-b470-4d56-8756-028d3aa2765f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-qwt9t"
Nov 01 03:11:00.449966 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:00.449515    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-qwt9t"
Nov 01 03:11:00.562150 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:00.562044    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:00.715522 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:00.715485    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:01.137995 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:01.137970    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-qwt9t"]
Nov 01 03:11:01.140356 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:11:01.140264    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedd2b590_b470_4d56_8756_028d3aa2765f.slice/crio-2a964cf68677fe6d0f77f967b38f1e2fe0adbbcdc535e33dbbdb52a3c0ba00b8 WatchSource:0}: Error finding container 2a964cf68677fe6d0f77f967b38f1e2fe0adbbcdc535e33dbbdb52a3c0ba00b8: Status 404 returned error can't find the container with id 2a964cf68677fe6d0f77f967b38f1e2fe0adbbcdc535e33dbbdb52a3c0ba00b8
Nov 01 03:11:01.833798 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:01.833756    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-qwt9t" event={"ID":"edd2b590-b470-4d56-8756-028d3aa2765f","Type":"ContainerStarted","Data":"2a964cf68677fe6d0f77f967b38f1e2fe0adbbcdc535e33dbbdb52a3c0ba00b8"}
Nov 01 03:11:01.837032 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:01.837004    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64472qbb" event={"ID":"1be18826-da5d-4c10-8599-382ebdc1d828","Type":"ContainerStarted","Data":"8cb080de6f490da80607fb4423f59c5f12b788b1c7f733351735d6dc9df485a4"}
Nov 01 03:11:01.840302 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:01.840274    4930 generic.go:356] "Generic (PLEG): container finished" podID="ac53872b-4bda-4070-ba1e-dc51073c3729" containerID="e8a986fe4f9640ad1ed93f78570e10bb58b7b0dc360ee5e4620c5c2ba124445b" exitCode=0
Nov 01 03:11:01.840397 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:01.840330    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6zcbg" event={"ID":"ac53872b-4bda-4070-ba1e-dc51073c3729","Type":"ContainerDied","Data":"e8a986fe4f9640ad1ed93f78570e10bb58b7b0dc360ee5e4620c5c2ba124445b"}
Nov 01 03:11:02.391614 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:02.391544    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:02.544337 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:02.544232    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:02.857887 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:02.857848    4930 generic.go:356] "Generic (PLEG): container finished" podID="1be18826-da5d-4c10-8599-382ebdc1d828" containerID="8cb080de6f490da80607fb4423f59c5f12b788b1c7f733351735d6dc9df485a4" exitCode=0
Nov 01 03:11:02.858102 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:02.857894    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64472qbb" event={"ID":"1be18826-da5d-4c10-8599-382ebdc1d828","Type":"ContainerDied","Data":"8cb080de6f490da80607fb4423f59c5f12b788b1c7f733351735d6dc9df485a4"}
Nov 01 03:11:06.331057 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:06.330941    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:06.485263 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:06.485136    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:08.921522 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:08.921489    4930 generic.go:356] "Generic (PLEG): container finished" podID="6916682f-50fe-4ab1-a516-7648fdf1a5b8" containerID="b618b98e5b88c01d9166521981e5ac4ec7b0421e1261e02c64f2341c6d883a31" exitCode=0
Nov 01 03:11:08.921947 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:08.921712    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sgtm5" event={"ID":"6916682f-50fe-4ab1-a516-7648fdf1a5b8","Type":"ContainerDied","Data":"b618b98e5b88c01d9166521981e5ac4ec7b0421e1261e02c64f2341c6d883a31"}
Nov 01 03:11:09.931434 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:09.931394    4930 generic.go:356] "Generic (PLEG): container finished" podID="6fa8951a-5ddb-40c9-962e-4c44f84c2974" containerID="8826a67cac3aa2d377a28baa3161ba28f270b7c47a41c12cf301c575f1917b71" exitCode=0
Nov 01 03:11:09.931940 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:09.931472    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jz7b2" event={"ID":"6fa8951a-5ddb-40c9-962e-4c44f84c2974","Type":"ContainerDied","Data":"8826a67cac3aa2d377a28baa3161ba28f270b7c47a41c12cf301c575f1917b71"}
Nov 01 03:11:09.934141 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:09.934113    4930 generic.go:356] "Generic (PLEG): container finished" podID="edd2b590-b470-4d56-8756-028d3aa2765f" containerID="aa537767d55c750fe4d87c498e761d44730e8a9b7857eb6fec9ba84e3f1c7cd5" exitCode=0
Nov 01 03:11:09.934258 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:09.934238    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-qwt9t" event={"ID":"edd2b590-b470-4d56-8756-028d3aa2765f","Type":"ContainerDied","Data":"aa537767d55c750fe4d87c498e761d44730e8a9b7857eb6fec9ba84e3f1c7cd5"}
Nov 01 03:11:09.940054 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:09.940024    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjq5xv" event={"ID":"175d8311-cd8c-41d9-9fc5-8068a2471a31","Type":"ContainerStarted","Data":"ac17510ebb39f1b257458f861f139a5ddf4b5614117ae23ee7e4048e39daac1c"}
Nov 01 03:11:10.107531 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:10.107495    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sgtm5"
Nov 01 03:11:10.138353 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:10.138322    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6916682f-50fe-4ab1-a516-7648fdf1a5b8-util\" (UniqueName: \"kubernetes.io/empty-dir/6916682f-50fe-4ab1-a516-7648fdf1a5b8-util\") pod \"6916682f-50fe-4ab1-a516-7648fdf1a5b8\" (UID: \"6916682f-50fe-4ab1-a516-7648fdf1a5b8\") "
Nov 01 03:11:10.138500 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:10.138367    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6916682f-50fe-4ab1-a516-7648fdf1a5b8-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6916682f-50fe-4ab1-a516-7648fdf1a5b8-bundle\") pod \"6916682f-50fe-4ab1-a516-7648fdf1a5b8\" (UID: \"6916682f-50fe-4ab1-a516-7648fdf1a5b8\") "
Nov 01 03:11:10.138500 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:10.138419    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6916682f-50fe-4ab1-a516-7648fdf1a5b8-kube-api-access-9r2lh\" (UniqueName: \"kubernetes.io/projected/6916682f-50fe-4ab1-a516-7648fdf1a5b8-kube-api-access-9r2lh\") pod \"6916682f-50fe-4ab1-a516-7648fdf1a5b8\" (UID: \"6916682f-50fe-4ab1-a516-7648fdf1a5b8\") "
Nov 01 03:11:10.139316 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:10.139249    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6916682f-50fe-4ab1-a516-7648fdf1a5b8-bundle" pod "6916682f-50fe-4ab1-a516-7648fdf1a5b8" (UID: "6916682f-50fe-4ab1-a516-7648fdf1a5b8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:11:10.144647 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:10.144620    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6916682f-50fe-4ab1-a516-7648fdf1a5b8-util" pod "6916682f-50fe-4ab1-a516-7648fdf1a5b8" (UID: "6916682f-50fe-4ab1-a516-7648fdf1a5b8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:11:10.147124 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:10.147090    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6916682f-50fe-4ab1-a516-7648fdf1a5b8-kube-api-access-9r2lh" pod "6916682f-50fe-4ab1-a516-7648fdf1a5b8" (UID: "6916682f-50fe-4ab1-a516-7648fdf1a5b8"). InnerVolumeSpecName "kube-api-access-9r2lh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:11:10.239188 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:10.239163    4930 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6916682f-50fe-4ab1-a516-7648fdf1a5b8-util\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:11:10.239308 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:10.239193    4930 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6916682f-50fe-4ab1-a516-7648fdf1a5b8-bundle\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:11:10.239308 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:10.239204    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r2lh\" (UniqueName: \"kubernetes.io/projected/6916682f-50fe-4ab1-a516-7648fdf1a5b8-kube-api-access-9r2lh\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:11:10.341738 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:10.341686    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjq5xv" podStartSLOduration=2.731129117 podStartE2EDuration="32.341672927s" totalImagesPullingTime="29.61054381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:10:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:11:10.20079611 +0000 UTC m=+274.723802700" watchObservedRunningTime="2026-11-01 03:11:10.341672927 +0000 UTC m=+274.864679514"
Nov 01 03:11:10.948498 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:10.948045    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64472qbb" event={"ID":"1be18826-da5d-4c10-8599-382ebdc1d828","Type":"ContainerStarted","Data":"e0bbbf3ed5ab397679ca250e67cd63ab4209e43adc1efd9b11162feb4cd76a8d"}
Nov 01 03:11:10.950367 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:10.950339    4930 generic.go:356] "Generic (PLEG): container finished" podID="175d8311-cd8c-41d9-9fc5-8068a2471a31" containerID="ac17510ebb39f1b257458f861f139a5ddf4b5614117ae23ee7e4048e39daac1c" exitCode=0
Nov 01 03:11:10.950475 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:10.950426    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjq5xv" event={"ID":"175d8311-cd8c-41d9-9fc5-8068a2471a31","Type":"ContainerDied","Data":"ac17510ebb39f1b257458f861f139a5ddf4b5614117ae23ee7e4048e39daac1c"}
Nov 01 03:11:10.952887 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:10.952865    4930 generic.go:356] "Generic (PLEG): container finished" podID="ac53872b-4bda-4070-ba1e-dc51073c3729" containerID="4cbd6871972bdaadd30211f4a398e7074858f32752f2a77fc7ed3f55f8f618c7" exitCode=0
Nov 01 03:11:10.953079 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:10.953060    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6zcbg" event={"ID":"ac53872b-4bda-4070-ba1e-dc51073c3729","Type":"ContainerDied","Data":"4cbd6871972bdaadd30211f4a398e7074858f32752f2a77fc7ed3f55f8f618c7"}
Nov 01 03:11:10.955301 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:10.955229    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sgtm5" event={"ID":"6916682f-50fe-4ab1-a516-7648fdf1a5b8","Type":"ContainerDied","Data":"189fc5b66a8a8654787dda39a0cb3afd0b5fa472536341c19ee19ace75dee0a4"}
Nov 01 03:11:10.955301 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:10.955272    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sgtm5"
Nov 01 03:11:10.955301 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:10.955282    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="189fc5b66a8a8654787dda39a0cb3afd0b5fa472536341c19ee19ace75dee0a4"
Nov 01 03:11:11.074219 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:11.073759    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64472qbb" podStartSLOduration=3.8142816980000003 podStartE2EDuration="32.073738621s" totalImagesPullingTime="28.259456923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:10:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:11:11.071308375 +0000 UTC m=+275.594314964" watchObservedRunningTime="2026-11-01 03:11:11.073738621 +0000 UTC m=+275.596745210"
Nov 01 03:11:11.116307 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:11.116283    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jz7b2"
Nov 01 03:11:11.147475 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:11.147454    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fa8951a-5ddb-40c9-962e-4c44f84c2974-util\" (UniqueName: \"kubernetes.io/empty-dir/6fa8951a-5ddb-40c9-962e-4c44f84c2974-util\") pod \"6fa8951a-5ddb-40c9-962e-4c44f84c2974\" (UID: \"6fa8951a-5ddb-40c9-962e-4c44f84c2974\") "
Nov 01 03:11:11.147590 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:11.147546    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fa8951a-5ddb-40c9-962e-4c44f84c2974-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6fa8951a-5ddb-40c9-962e-4c44f84c2974-bundle\") pod \"6fa8951a-5ddb-40c9-962e-4c44f84c2974\" (UID: \"6fa8951a-5ddb-40c9-962e-4c44f84c2974\") "
Nov 01 03:11:11.147659 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:11.147600    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fa8951a-5ddb-40c9-962e-4c44f84c2974-kube-api-access-gnhdr\" (UniqueName: \"kubernetes.io/projected/6fa8951a-5ddb-40c9-962e-4c44f84c2974-kube-api-access-gnhdr\") pod \"6fa8951a-5ddb-40c9-962e-4c44f84c2974\" (UID: \"6fa8951a-5ddb-40c9-962e-4c44f84c2974\") "
Nov 01 03:11:11.147949 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:11.147922    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fa8951a-5ddb-40c9-962e-4c44f84c2974-bundle" pod "6fa8951a-5ddb-40c9-962e-4c44f84c2974" (UID: "6fa8951a-5ddb-40c9-962e-4c44f84c2974"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:11:11.149673 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:11.149650    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fa8951a-5ddb-40c9-962e-4c44f84c2974-kube-api-access-gnhdr" pod "6fa8951a-5ddb-40c9-962e-4c44f84c2974" (UID: "6fa8951a-5ddb-40c9-962e-4c44f84c2974"). InnerVolumeSpecName "kube-api-access-gnhdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:11:11.152353 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:11.152330    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fa8951a-5ddb-40c9-962e-4c44f84c2974-util" pod "6fa8951a-5ddb-40c9-962e-4c44f84c2974" (UID: "6fa8951a-5ddb-40c9-962e-4c44f84c2974"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:11:11.248336 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:11.248310    4930 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6fa8951a-5ddb-40c9-962e-4c44f84c2974-bundle\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:11:11.248336 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:11.248336    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnhdr\" (UniqueName: \"kubernetes.io/projected/6fa8951a-5ddb-40c9-962e-4c44f84c2974-kube-api-access-gnhdr\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:11:11.248527 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:11.248346    4930 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6fa8951a-5ddb-40c9-962e-4c44f84c2974-util\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:11:11.961896 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:11.961864    4930 generic.go:356] "Generic (PLEG): container finished" podID="e457c005-7a56-408f-a56f-93a1d29cbc3c" containerID="3733a671364ffb43f4b0a81fad8a5686cfdae6fb49e577df6e2cb2c7235f8b43" exitCode=0
Nov 01 03:11:11.962276 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:11.961958    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax8zjg" event={"ID":"e457c005-7a56-408f-a56f-93a1d29cbc3c","Type":"ContainerDied","Data":"3733a671364ffb43f4b0a81fad8a5686cfdae6fb49e577df6e2cb2c7235f8b43"}
Nov 01 03:11:11.963923 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:11.963899    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jz7b2" event={"ID":"6fa8951a-5ddb-40c9-962e-4c44f84c2974","Type":"ContainerDied","Data":"efb82ac6739ba63e1d80e063e33d22e85a1c0315ffeeb2002647f84a962c274a"}
Nov 01 03:11:11.964037 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:11.963951    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efb82ac6739ba63e1d80e063e33d22e85a1c0315ffeeb2002647f84a962c274a"
Nov 01 03:11:11.964037 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:11.963905    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jz7b2"
Nov 01 03:11:11.965805 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:11.965779    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-qwt9t" event={"ID":"edd2b590-b470-4d56-8756-028d3aa2765f","Type":"ContainerStarted","Data":"354486caaa4dbeeb34f57ff863619e4ad3db2c212d0206eca5fb52e11665d8a8"}
Nov 01 03:11:11.965929 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:11.965862    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-qwt9t"
Nov 01 03:11:11.967924 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:11.967903    4930 generic.go:356] "Generic (PLEG): container finished" podID="1be18826-da5d-4c10-8599-382ebdc1d828" containerID="e0bbbf3ed5ab397679ca250e67cd63ab4209e43adc1efd9b11162feb4cd76a8d" exitCode=0
Nov 01 03:11:11.968021 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:11.967956    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64472qbb" event={"ID":"1be18826-da5d-4c10-8599-382ebdc1d828","Type":"ContainerDied","Data":"e0bbbf3ed5ab397679ca250e67cd63ab4209e43adc1efd9b11162feb4cd76a8d"}
Nov 01 03:11:12.103725 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:12.103705    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjq5xv"
Nov 01 03:11:12.115223 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:12.115203    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6zcbg"
Nov 01 03:11:12.154341 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:12.154315    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/175d8311-cd8c-41d9-9fc5-8068a2471a31-util\" (UniqueName: \"kubernetes.io/empty-dir/175d8311-cd8c-41d9-9fc5-8068a2471a31-util\") pod \"175d8311-cd8c-41d9-9fc5-8068a2471a31\" (UID: \"175d8311-cd8c-41d9-9fc5-8068a2471a31\") "
Nov 01 03:11:12.154454 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:12.154373    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac53872b-4bda-4070-ba1e-dc51073c3729-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac53872b-4bda-4070-ba1e-dc51073c3729-bundle\") pod \"ac53872b-4bda-4070-ba1e-dc51073c3729\" (UID: \"ac53872b-4bda-4070-ba1e-dc51073c3729\") "
Nov 01 03:11:12.154454 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:12.154417    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac53872b-4bda-4070-ba1e-dc51073c3729-kube-api-access-lqlmb\" (UniqueName: \"kubernetes.io/projected/ac53872b-4bda-4070-ba1e-dc51073c3729-kube-api-access-lqlmb\") pod \"ac53872b-4bda-4070-ba1e-dc51073c3729\" (UID: \"ac53872b-4bda-4070-ba1e-dc51073c3729\") "
Nov 01 03:11:12.154454 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:12.154446    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/175d8311-cd8c-41d9-9fc5-8068a2471a31-kube-api-access-t77hb\" (UniqueName: \"kubernetes.io/projected/175d8311-cd8c-41d9-9fc5-8068a2471a31-kube-api-access-t77hb\") pod \"175d8311-cd8c-41d9-9fc5-8068a2471a31\" (UID: \"175d8311-cd8c-41d9-9fc5-8068a2471a31\") "
Nov 01 03:11:12.154649 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:12.154477    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac53872b-4bda-4070-ba1e-dc51073c3729-util\" (UniqueName: \"kubernetes.io/empty-dir/ac53872b-4bda-4070-ba1e-dc51073c3729-util\") pod \"ac53872b-4bda-4070-ba1e-dc51073c3729\" (UID: \"ac53872b-4bda-4070-ba1e-dc51073c3729\") "
Nov 01 03:11:12.154649 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:12.154518    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/175d8311-cd8c-41d9-9fc5-8068a2471a31-bundle\" (UniqueName: \"kubernetes.io/empty-dir/175d8311-cd8c-41d9-9fc5-8068a2471a31-bundle\") pod \"175d8311-cd8c-41d9-9fc5-8068a2471a31\" (UID: \"175d8311-cd8c-41d9-9fc5-8068a2471a31\") "
Nov 01 03:11:12.155148 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:12.155117    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/175d8311-cd8c-41d9-9fc5-8068a2471a31-bundle" pod "175d8311-cd8c-41d9-9fc5-8068a2471a31" (UID: "175d8311-cd8c-41d9-9fc5-8068a2471a31"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:11:12.155249 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:12.155203    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac53872b-4bda-4070-ba1e-dc51073c3729-bundle" pod "ac53872b-4bda-4070-ba1e-dc51073c3729" (UID: "ac53872b-4bda-4070-ba1e-dc51073c3729"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:11:12.157035 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:12.157007    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac53872b-4bda-4070-ba1e-dc51073c3729-kube-api-access-lqlmb" pod "ac53872b-4bda-4070-ba1e-dc51073c3729" (UID: "ac53872b-4bda-4070-ba1e-dc51073c3729"). InnerVolumeSpecName "kube-api-access-lqlmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:11:12.157116 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:12.157079    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/175d8311-cd8c-41d9-9fc5-8068a2471a31-kube-api-access-t77hb" pod "175d8311-cd8c-41d9-9fc5-8068a2471a31" (UID: "175d8311-cd8c-41d9-9fc5-8068a2471a31"). InnerVolumeSpecName "kube-api-access-t77hb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:11:12.160075 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:12.160051    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/175d8311-cd8c-41d9-9fc5-8068a2471a31-util" pod "175d8311-cd8c-41d9-9fc5-8068a2471a31" (UID: "175d8311-cd8c-41d9-9fc5-8068a2471a31"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:11:12.160429 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:12.160408    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac53872b-4bda-4070-ba1e-dc51073c3729-util" pod "ac53872b-4bda-4070-ba1e-dc51073c3729" (UID: "ac53872b-4bda-4070-ba1e-dc51073c3729"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:11:12.255405 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:12.255382    4930 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/175d8311-cd8c-41d9-9fc5-8068a2471a31-util\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:11:12.255501 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:12.255408    4930 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac53872b-4bda-4070-ba1e-dc51073c3729-bundle\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:11:12.255501 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:12.255422    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqlmb\" (UniqueName: \"kubernetes.io/projected/ac53872b-4bda-4070-ba1e-dc51073c3729-kube-api-access-lqlmb\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:11:12.255501 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:12.255433    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t77hb\" (UniqueName: \"kubernetes.io/projected/175d8311-cd8c-41d9-9fc5-8068a2471a31-kube-api-access-t77hb\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:11:12.255501 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:12.255442    4930 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac53872b-4bda-4070-ba1e-dc51073c3729-util\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:11:12.255501 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:12.255450    4930 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/175d8311-cd8c-41d9-9fc5-8068a2471a31-bundle\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:11:12.274859 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:12.274814    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-qwt9t" podStartSLOduration=3.9782409960000003 podStartE2EDuration="13.27480144s" totalImagesPullingTime="8.589659947s" totalInitContainerRuntime="706.900497ms" isStatefulPod=false podCreationTimestamp="2026-11-01 03:10:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:11:12.273204034 +0000 UTC m=+276.796210629" watchObservedRunningTime="2026-11-01 03:11:12.27480144 +0000 UTC m=+276.797808027"
Nov 01 03:11:12.974234 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:12.974199    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjq5xv" event={"ID":"175d8311-cd8c-41d9-9fc5-8068a2471a31","Type":"ContainerDied","Data":"d384554a16cfb89d2c7c75594f58f854dcb7cbde75358a9c56475ef6e72dffe3"}
Nov 01 03:11:12.974689 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:12.974250    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d384554a16cfb89d2c7c75594f58f854dcb7cbde75358a9c56475ef6e72dffe3"
Nov 01 03:11:12.974689 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:12.974210    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjq5xv"
Nov 01 03:11:12.976278 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:12.976258    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6zcbg"
Nov 01 03:11:12.976388 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:12.976280    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6zcbg" event={"ID":"ac53872b-4bda-4070-ba1e-dc51073c3729","Type":"ContainerDied","Data":"56943cad8e0348dca7a1f89ea70ac2ca9be6e264972b4541c0115ccbd3525951"}
Nov 01 03:11:12.976388 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:12.976322    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56943cad8e0348dca7a1f89ea70ac2ca9be6e264972b4541c0115ccbd3525951"
Nov 01 03:11:13.129482 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:13.129461    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax8zjg"
Nov 01 03:11:13.131635 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:13.131616    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64472qbb"
Nov 01 03:11:13.161966 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:13.161940    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e457c005-7a56-408f-a56f-93a1d29cbc3c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e457c005-7a56-408f-a56f-93a1d29cbc3c-bundle\") pod \"e457c005-7a56-408f-a56f-93a1d29cbc3c\" (UID: \"e457c005-7a56-408f-a56f-93a1d29cbc3c\") "
Nov 01 03:11:13.162067 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:13.162017    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e457c005-7a56-408f-a56f-93a1d29cbc3c-kube-api-access-9gwcw\" (UniqueName: \"kubernetes.io/projected/e457c005-7a56-408f-a56f-93a1d29cbc3c-kube-api-access-9gwcw\") pod \"e457c005-7a56-408f-a56f-93a1d29cbc3c\" (UID: \"e457c005-7a56-408f-a56f-93a1d29cbc3c\") "
Nov 01 03:11:13.162067 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:13.162062    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e457c005-7a56-408f-a56f-93a1d29cbc3c-util\" (UniqueName: \"kubernetes.io/empty-dir/e457c005-7a56-408f-a56f-93a1d29cbc3c-util\") pod \"e457c005-7a56-408f-a56f-93a1d29cbc3c\" (UID: \"e457c005-7a56-408f-a56f-93a1d29cbc3c\") "
Nov 01 03:11:13.162177 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:13.162089    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1be18826-da5d-4c10-8599-382ebdc1d828-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1be18826-da5d-4c10-8599-382ebdc1d828-bundle\") pod \"1be18826-da5d-4c10-8599-382ebdc1d828\" (UID: \"1be18826-da5d-4c10-8599-382ebdc1d828\") "
Nov 01 03:11:13.162177 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:13.162153    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1be18826-da5d-4c10-8599-382ebdc1d828-util\" (UniqueName: \"kubernetes.io/empty-dir/1be18826-da5d-4c10-8599-382ebdc1d828-util\") pod \"1be18826-da5d-4c10-8599-382ebdc1d828\" (UID: \"1be18826-da5d-4c10-8599-382ebdc1d828\") "
Nov 01 03:11:13.162283 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:13.162201    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1be18826-da5d-4c10-8599-382ebdc1d828-kube-api-access-ffj4g\" (UniqueName: \"kubernetes.io/projected/1be18826-da5d-4c10-8599-382ebdc1d828-kube-api-access-ffj4g\") pod \"1be18826-da5d-4c10-8599-382ebdc1d828\" (UID: \"1be18826-da5d-4c10-8599-382ebdc1d828\") "
Nov 01 03:11:13.162594 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:13.162545    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1be18826-da5d-4c10-8599-382ebdc1d828-bundle" pod "1be18826-da5d-4c10-8599-382ebdc1d828" (UID: "1be18826-da5d-4c10-8599-382ebdc1d828"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:11:13.162843 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:13.162811    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e457c005-7a56-408f-a56f-93a1d29cbc3c-bundle" pod "e457c005-7a56-408f-a56f-93a1d29cbc3c" (UID: "e457c005-7a56-408f-a56f-93a1d29cbc3c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:11:13.164107 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:13.164084    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e457c005-7a56-408f-a56f-93a1d29cbc3c-kube-api-access-9gwcw" pod "e457c005-7a56-408f-a56f-93a1d29cbc3c" (UID: "e457c005-7a56-408f-a56f-93a1d29cbc3c"). InnerVolumeSpecName "kube-api-access-9gwcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:11:13.164254 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:13.164238    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1be18826-da5d-4c10-8599-382ebdc1d828-kube-api-access-ffj4g" pod "1be18826-da5d-4c10-8599-382ebdc1d828" (UID: "1be18826-da5d-4c10-8599-382ebdc1d828"). InnerVolumeSpecName "kube-api-access-ffj4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:11:13.168462 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:13.168434    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e457c005-7a56-408f-a56f-93a1d29cbc3c-util" pod "e457c005-7a56-408f-a56f-93a1d29cbc3c" (UID: "e457c005-7a56-408f-a56f-93a1d29cbc3c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:11:13.169808 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:13.169787    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1be18826-da5d-4c10-8599-382ebdc1d828-util" pod "1be18826-da5d-4c10-8599-382ebdc1d828" (UID: "1be18826-da5d-4c10-8599-382ebdc1d828"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:11:13.263254 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:13.263183    4930 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e457c005-7a56-408f-a56f-93a1d29cbc3c-bundle\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:11:13.263254 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:13.263220    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gwcw\" (UniqueName: \"kubernetes.io/projected/e457c005-7a56-408f-a56f-93a1d29cbc3c-kube-api-access-9gwcw\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:11:13.263254 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:13.263237    4930 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e457c005-7a56-408f-a56f-93a1d29cbc3c-util\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:11:13.263254 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:13.263252    4930 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1be18826-da5d-4c10-8599-382ebdc1d828-bundle\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:11:13.263464 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:13.263266    4930 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1be18826-da5d-4c10-8599-382ebdc1d828-util\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:11:13.263464 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:13.263280    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffj4g\" (UniqueName: \"kubernetes.io/projected/1be18826-da5d-4c10-8599-382ebdc1d828-kube-api-access-ffj4g\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:11:13.982126 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:13.982048    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64472qbb"
Nov 01 03:11:13.982126 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:13.982060    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64472qbb" event={"ID":"1be18826-da5d-4c10-8599-382ebdc1d828","Type":"ContainerDied","Data":"498ae61259bd248c7935124531d83076361a684dcae976f2ba6e5bb4d66eb358"}
Nov 01 03:11:13.982126 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:13.982112    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="498ae61259bd248c7935124531d83076361a684dcae976f2ba6e5bb4d66eb358"
Nov 01 03:11:13.984090 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:13.984066    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax8zjg" event={"ID":"e457c005-7a56-408f-a56f-93a1d29cbc3c","Type":"ContainerDied","Data":"e838945874c73a20411060730db0e026c8eb9c2877287ba0389720160c980d23"}
Nov 01 03:11:13.984212 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:13.984110    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e838945874c73a20411060730db0e026c8eb9c2877287ba0389720160c980d23"
Nov 01 03:11:13.984212 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:13.984179    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ax8zjg"
Nov 01 03:11:17.647663 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:17.647625    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:17.870955 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:17.870919    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:19.120136 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:11:19.120100    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:11:22.979703 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:22.979675    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-576db5b58f-qwt9t"
Nov 01 03:11:23.300455 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:23.300379    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:27.062886 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.062850    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7bqn7"]
Nov 01 03:11:27.063248 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.063138    4930 cpu_manager.go:414] "containerMap: removing container" podUID="175d8311-cd8c-41d9-9fc5-8068a2471a31" containerName="pull"
Nov 01 03:11:27.063248 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.063149    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="175d8311-cd8c-41d9-9fc5-8068a2471a31" containerName="pull"
Nov 01 03:11:27.063248 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.063165    4930 cpu_manager.go:414] "containerMap: removing container" podUID="e457c005-7a56-408f-a56f-93a1d29cbc3c" containerName="util"
Nov 01 03:11:27.063248 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.063170    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e457c005-7a56-408f-a56f-93a1d29cbc3c" containerName="util"
Nov 01 03:11:27.063248 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.063177    4930 cpu_manager.go:414] "containerMap: removing container" podUID="6fa8951a-5ddb-40c9-962e-4c44f84c2974" containerName="pull"
Nov 01 03:11:27.063248 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.063182    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fa8951a-5ddb-40c9-962e-4c44f84c2974" containerName="pull"
Nov 01 03:11:27.063248 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.063191    4930 cpu_manager.go:414] "containerMap: removing container" podUID="175d8311-cd8c-41d9-9fc5-8068a2471a31" containerName="extract"
Nov 01 03:11:27.063248 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.063196    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="175d8311-cd8c-41d9-9fc5-8068a2471a31" containerName="extract"
Nov 01 03:11:27.063248 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.063202    4930 cpu_manager.go:414] "containerMap: removing container" podUID="e457c005-7a56-408f-a56f-93a1d29cbc3c" containerName="extract"
Nov 01 03:11:27.063248 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.063206    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e457c005-7a56-408f-a56f-93a1d29cbc3c" containerName="extract"
Nov 01 03:11:27.063248 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.063213    4930 cpu_manager.go:414] "containerMap: removing container" podUID="6916682f-50fe-4ab1-a516-7648fdf1a5b8" containerName="util"
Nov 01 03:11:27.063248 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.063217    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6916682f-50fe-4ab1-a516-7648fdf1a5b8" containerName="util"
Nov 01 03:11:27.063248 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.063225    4930 cpu_manager.go:414] "containerMap: removing container" podUID="6916682f-50fe-4ab1-a516-7648fdf1a5b8" containerName="pull"
Nov 01 03:11:27.063248 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.063230    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6916682f-50fe-4ab1-a516-7648fdf1a5b8" containerName="pull"
Nov 01 03:11:27.063248 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.063243    4930 cpu_manager.go:414] "containerMap: removing container" podUID="6fa8951a-5ddb-40c9-962e-4c44f84c2974" containerName="util"
Nov 01 03:11:27.063248 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.063247    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fa8951a-5ddb-40c9-962e-4c44f84c2974" containerName="util"
Nov 01 03:11:27.063248 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.063256    4930 cpu_manager.go:414] "containerMap: removing container" podUID="ac53872b-4bda-4070-ba1e-dc51073c3729" containerName="extract"
Nov 01 03:11:27.063788 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.063261    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac53872b-4bda-4070-ba1e-dc51073c3729" containerName="extract"
Nov 01 03:11:27.063788 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.063268    4930 cpu_manager.go:414] "containerMap: removing container" podUID="ac53872b-4bda-4070-ba1e-dc51073c3729" containerName="pull"
Nov 01 03:11:27.063788 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.063272    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac53872b-4bda-4070-ba1e-dc51073c3729" containerName="pull"
Nov 01 03:11:27.063788 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.063279    4930 cpu_manager.go:414] "containerMap: removing container" podUID="6916682f-50fe-4ab1-a516-7648fdf1a5b8" containerName="extract"
Nov 01 03:11:27.063788 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.063283    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6916682f-50fe-4ab1-a516-7648fdf1a5b8" containerName="extract"
Nov 01 03:11:27.063788 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.063290    4930 cpu_manager.go:414] "containerMap: removing container" podUID="e457c005-7a56-408f-a56f-93a1d29cbc3c" containerName="pull"
Nov 01 03:11:27.063788 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.063294    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e457c005-7a56-408f-a56f-93a1d29cbc3c" containerName="pull"
Nov 01 03:11:27.063788 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.063302    4930 cpu_manager.go:414] "containerMap: removing container" podUID="1be18826-da5d-4c10-8599-382ebdc1d828" containerName="extract"
Nov 01 03:11:27.063788 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.063306    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1be18826-da5d-4c10-8599-382ebdc1d828" containerName="extract"
Nov 01 03:11:27.063788 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.063313    4930 cpu_manager.go:414] "containerMap: removing container" podUID="1be18826-da5d-4c10-8599-382ebdc1d828" containerName="pull"
Nov 01 03:11:27.063788 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.063317    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1be18826-da5d-4c10-8599-382ebdc1d828" containerName="pull"
Nov 01 03:11:27.063788 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.063322    4930 cpu_manager.go:414] "containerMap: removing container" podUID="6fa8951a-5ddb-40c9-962e-4c44f84c2974" containerName="extract"
Nov 01 03:11:27.063788 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.063327    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fa8951a-5ddb-40c9-962e-4c44f84c2974" containerName="extract"
Nov 01 03:11:27.063788 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.063333    4930 cpu_manager.go:414] "containerMap: removing container" podUID="175d8311-cd8c-41d9-9fc5-8068a2471a31" containerName="util"
Nov 01 03:11:27.063788 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.063337    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="175d8311-cd8c-41d9-9fc5-8068a2471a31" containerName="util"
Nov 01 03:11:27.063788 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.063344    4930 cpu_manager.go:414] "containerMap: removing container" podUID="1be18826-da5d-4c10-8599-382ebdc1d828" containerName="util"
Nov 01 03:11:27.063788 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.063348    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1be18826-da5d-4c10-8599-382ebdc1d828" containerName="util"
Nov 01 03:11:27.063788 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.063354    4930 cpu_manager.go:414] "containerMap: removing container" podUID="ac53872b-4bda-4070-ba1e-dc51073c3729" containerName="util"
Nov 01 03:11:27.063788 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.063358    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac53872b-4bda-4070-ba1e-dc51073c3729" containerName="util"
Nov 01 03:11:27.063788 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.063425    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="175d8311-cd8c-41d9-9fc5-8068a2471a31" containerName="extract"
Nov 01 03:11:27.063788 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.063433    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="e457c005-7a56-408f-a56f-93a1d29cbc3c" containerName="extract"
Nov 01 03:11:27.063788 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.063442    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac53872b-4bda-4070-ba1e-dc51073c3729" containerName="extract"
Nov 01 03:11:27.063788 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.063447    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="6916682f-50fe-4ab1-a516-7648fdf1a5b8" containerName="extract"
Nov 01 03:11:27.063788 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.063452    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="1be18826-da5d-4c10-8599-382ebdc1d828" containerName="extract"
Nov 01 03:11:27.063788 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.063457    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fa8951a-5ddb-40c9-962e-4c44f84c2974" containerName="extract"
Nov 01 03:11:27.065196 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.065181    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7bqn7"
Nov 01 03:11:27.122985 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.122960    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:27.167785 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.167760    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Nov 01 03:11:27.167952 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.167767    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Nov 01 03:11:27.168038 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.168019    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-zmztb\""
Nov 01 03:11:27.169708 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.169691    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7bqn7"]
Nov 01 03:11:27.171407 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.171391    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqrcw\" (UniqueName: \"kubernetes.io/projected/71c14375-3d66-4183-8b90-18f9ed38256f-kube-api-access-dqrcw\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-7bqn7\" (UID: \"71c14375-3d66-4183-8b90-18f9ed38256f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7bqn7"
Nov 01 03:11:27.171453 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.171427    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71c14375-3d66-4183-8b90-18f9ed38256f-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-7bqn7\" (UID: \"71c14375-3d66-4183-8b90-18f9ed38256f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7bqn7"
Nov 01 03:11:27.272091 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.272061    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqrcw\" (UniqueName: \"kubernetes.io/projected/71c14375-3d66-4183-8b90-18f9ed38256f-kube-api-access-dqrcw\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-7bqn7\" (UID: \"71c14375-3d66-4183-8b90-18f9ed38256f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7bqn7"
Nov 01 03:11:27.272213 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.272104    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71c14375-3d66-4183-8b90-18f9ed38256f-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-7bqn7\" (UID: \"71c14375-3d66-4183-8b90-18f9ed38256f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7bqn7"
Nov 01 03:11:27.272436 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.272421    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71c14375-3d66-4183-8b90-18f9ed38256f-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-7bqn7\" (UID: \"71c14375-3d66-4183-8b90-18f9ed38256f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7bqn7"
Nov 01 03:11:27.338276 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.338247    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqrcw\" (UniqueName: \"kubernetes.io/projected/71c14375-3d66-4183-8b90-18f9ed38256f-kube-api-access-dqrcw\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-7bqn7\" (UID: \"71c14375-3d66-4183-8b90-18f9ed38256f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7bqn7"
Nov 01 03:11:27.375289 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.375269    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7bqn7"
Nov 01 03:11:27.846135 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:27.846106    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7bqn7"]
Nov 01 03:11:27.884271 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:11:27.884242    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71c14375_3d66_4183_8b90_18f9ed38256f.slice/crio-5f0042ebc0df86556c8bcfa07491a2678efd432bda55cfcf5aa24a80b4e60064 WatchSource:0}: Error finding container 5f0042ebc0df86556c8bcfa07491a2678efd432bda55cfcf5aa24a80b4e60064: Status 404 returned error can't find the container with id 5f0042ebc0df86556c8bcfa07491a2678efd432bda55cfcf5aa24a80b4e60064
Nov 01 03:11:28.048184 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:28.048143    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7bqn7" event={"ID":"71c14375-3d66-4183-8b90-18f9ed38256f","Type":"ContainerStarted","Data":"5f0042ebc0df86556c8bcfa07491a2678efd432bda55cfcf5aa24a80b4e60064"}
Nov 01 03:11:29.460389 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:29.460350    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:31.064524 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:31.064446    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7bqn7" event={"ID":"71c14375-3d66-4183-8b90-18f9ed38256f","Type":"ContainerStarted","Data":"6dd3c127d012f57233218bd716d965a2561366f9147e71a439ce76af2508c7bc"}
Nov 01 03:11:31.079425 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:31.079391    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:31.283532 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:31.283474    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7bqn7" podStartSLOduration=2.5580916399999998 podStartE2EDuration="5.283456899s" totalImagesPullingTime="2.725365259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:11:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:11:31.23308007 +0000 UTC m=+295.756086657" watchObservedRunningTime="2026-11-01 03:11:31.283456899 +0000 UTC m=+295.806463491"
Nov 01 03:11:32.739741 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:32.739707    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-rgb2s"]
Nov 01 03:11:32.742010 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:32.741993    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rgb2s"
Nov 01 03:11:32.818983 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:32.818955    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd2p8\" (UniqueName: \"kubernetes.io/projected/347b9e3e-b16b-416f-abeb-137af0e6e8b0-kube-api-access-vd2p8\") pod \"limitador-operator-controller-manager-5778bb4449-rgb2s\" (UID: \"347b9e3e-b16b-416f-abeb-137af0e6e8b0\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rgb2s"
Nov 01 03:11:32.844956 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:32.844930    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 01 03:11:32.845069 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:32.844994    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 01 03:11:32.845230 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:32.845204    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-6r2hn\""
Nov 01 03:11:32.846999 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:32.846980    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-rgb2s"]
Nov 01 03:11:32.920199 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:32.920173    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd2p8\" (UniqueName: \"kubernetes.io/projected/347b9e3e-b16b-416f-abeb-137af0e6e8b0-kube-api-access-vd2p8\") pod \"limitador-operator-controller-manager-5778bb4449-rgb2s\" (UID: \"347b9e3e-b16b-416f-abeb-137af0e6e8b0\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rgb2s"
Nov 01 03:11:32.984958 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:32.984934    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd2p8\" (UniqueName: \"kubernetes.io/projected/347b9e3e-b16b-416f-abeb-137af0e6e8b0-kube-api-access-vd2p8\") pod \"limitador-operator-controller-manager-5778bb4449-rgb2s\" (UID: \"347b9e3e-b16b-416f-abeb-137af0e6e8b0\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rgb2s"
Nov 01 03:11:33.053730 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:33.053673    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rgb2s"
Nov 01 03:11:33.439603 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:33.439577    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-rgb2s"]
Nov 01 03:11:33.441644 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:11:33.441619    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod347b9e3e_b16b_416f_abeb_137af0e6e8b0.slice/crio-ec09d33fbd499eb294150e959b071790800f0f0d45f17909ad8499da326b3d6a WatchSource:0}: Error finding container ec09d33fbd499eb294150e959b071790800f0f0d45f17909ad8499da326b3d6a: Status 404 returned error can't find the container with id ec09d33fbd499eb294150e959b071790800f0f0d45f17909ad8499da326b3d6a
Nov 01 03:11:33.491750 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:33.491713    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:33.665684 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:33.665656    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:34.079480 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:34.079452    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rgb2s" event={"ID":"347b9e3e-b16b-416f-abeb-137af0e6e8b0","Type":"ContainerStarted","Data":"ec09d33fbd499eb294150e959b071790800f0f0d45f17909ad8499da326b3d6a"}
Nov 01 03:11:34.113419 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:34.113390    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:34.660951 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:34.660914    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:35.631039 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:35.631009    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jh955_f29ea61e-2a78-4c11-9566-f52497e135ff/console-operator/2.log"
Nov 01 03:11:35.637109 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:35.637085    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jh955_f29ea61e-2a78-4c11-9566-f52497e135ff/console-operator/2.log"
Nov 01 03:11:35.734025 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:35.733987    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:36.333454 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:36.333415    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:36.719719 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:36.719682    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:37.100853 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:37.100816    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:38.104976 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:38.104940    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rgb2s" event={"ID":"347b9e3e-b16b-416f-abeb-137af0e6e8b0","Type":"ContainerStarted","Data":"50213642afdd8a0a7eef15d20947f18b4a7691a09aedf55f9f5a021bc3b6c6a1"}
Nov 01 03:11:38.105377 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:38.105085    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rgb2s"
Nov 01 03:11:38.226567 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:38.226500    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rgb2s" podStartSLOduration=2.093129995 podStartE2EDuration="6.226485502s" totalImagesPullingTime="4.133355507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:11:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:11:38.221682596 +0000 UTC m=+302.744689183" watchObservedRunningTime="2026-11-01 03:11:38.226485502 +0000 UTC m=+302.749492089"
Nov 01 03:11:38.537372 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:38.537291    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:38.706456 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:38.706421    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:40.156415 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:40.156380    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b4sdg"]
Nov 01 03:11:40.158703 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:40.158686    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b4sdg"
Nov 01 03:11:40.217188 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:40.217162    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b4sdg"]
Nov 01 03:11:40.264246 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:40.264220    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-mvk79\""
Nov 01 03:11:40.280495 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:40.280465    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58tn4\" (UniqueName: \"kubernetes.io/projected/e4edaa96-e7f5-480f-9c85-b7189aedb11c-kube-api-access-58tn4\") pod \"kuadrant-operator-controller-manager-678d87766f-b4sdg\" (UID: \"e4edaa96-e7f5-480f-9c85-b7189aedb11c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b4sdg"
Nov 01 03:11:40.280639 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:40.280531    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e4edaa96-e7f5-480f-9c85-b7189aedb11c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-b4sdg\" (UID: \"e4edaa96-e7f5-480f-9c85-b7189aedb11c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b4sdg"
Nov 01 03:11:40.381873 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:40.381840    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58tn4\" (UniqueName: \"kubernetes.io/projected/e4edaa96-e7f5-480f-9c85-b7189aedb11c-kube-api-access-58tn4\") pod \"kuadrant-operator-controller-manager-678d87766f-b4sdg\" (UID: \"e4edaa96-e7f5-480f-9c85-b7189aedb11c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b4sdg"
Nov 01 03:11:40.382036 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:40.381890    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e4edaa96-e7f5-480f-9c85-b7189aedb11c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-b4sdg\" (UID: \"e4edaa96-e7f5-480f-9c85-b7189aedb11c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b4sdg"
Nov 01 03:11:40.382263 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:40.382243    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e4edaa96-e7f5-480f-9c85-b7189aedb11c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-b4sdg\" (UID: \"e4edaa96-e7f5-480f-9c85-b7189aedb11c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b4sdg"
Nov 01 03:11:40.443360 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:40.443297    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58tn4\" (UniqueName: \"kubernetes.io/projected/e4edaa96-e7f5-480f-9c85-b7189aedb11c-kube-api-access-58tn4\") pod \"kuadrant-operator-controller-manager-678d87766f-b4sdg\" (UID: \"e4edaa96-e7f5-480f-9c85-b7189aedb11c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b4sdg"
Nov 01 03:11:40.469807 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:40.469781    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b4sdg"
Nov 01 03:11:40.893315 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:40.893280    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:40.945973 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:40.945953    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b4sdg"]
Nov 01 03:11:40.948067 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:11:40.948042    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4edaa96_e7f5_480f_9c85_b7189aedb11c.slice/crio-416be4b2e947684c2cc3799ab343fb45dbb0f8216e72e905e467235ceaa50d67 WatchSource:0}: Error finding container 416be4b2e947684c2cc3799ab343fb45dbb0f8216e72e905e467235ceaa50d67: Status 404 returned error can't find the container with id 416be4b2e947684c2cc3799ab343fb45dbb0f8216e72e905e467235ceaa50d67
Nov 01 03:11:40.950852 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:40.950830    4930 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 03:11:41.090073 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:41.090033    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:41.121031 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:41.121007    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b4sdg" event={"ID":"e4edaa96-e7f5-480f-9c85-b7189aedb11c","Type":"ContainerStarted","Data":"416be4b2e947684c2cc3799ab343fb45dbb0f8216e72e905e467235ceaa50d67"}
Nov 01 03:11:41.264122 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:41.264051    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:41.804706 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:41.804671    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:43.055958 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:43.055917    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:43.337694 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:43.337659    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:43.619857 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:43.619763    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:44.155386 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:44.155350    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:45.533455 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:45.533415    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:45.575567 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:45.575524    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-k7f54"]
Nov 01 03:11:45.577815 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:45.577798    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-k7f54"
Nov 01 03:11:45.685190 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:45.685156    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-dklmm\""
Nov 01 03:11:45.686837 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:45.686818    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-k7f54"]
Nov 01 03:11:45.719552 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:45.719529    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fvvn\" (UniqueName: \"kubernetes.io/projected/b06cb628-601b-44ce-9667-aa39fdbe1c01-kube-api-access-5fvvn\") pod \"authorino-operator-7b7bb4bc77-k7f54\" (UID: \"b06cb628-601b-44ce-9667-aa39fdbe1c01\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-k7f54"
Nov 01 03:11:45.734678 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:45.734653    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:45.820448 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:45.820373    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fvvn\" (UniqueName: \"kubernetes.io/projected/b06cb628-601b-44ce-9667-aa39fdbe1c01-kube-api-access-5fvvn\") pod \"authorino-operator-7b7bb4bc77-k7f54\" (UID: \"b06cb628-601b-44ce-9667-aa39fdbe1c01\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-k7f54"
Nov 01 03:11:45.899484 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:45.899452    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fvvn\" (UniqueName: \"kubernetes.io/projected/b06cb628-601b-44ce-9667-aa39fdbe1c01-kube-api-access-5fvvn\") pod \"authorino-operator-7b7bb4bc77-k7f54\" (UID: \"b06cb628-601b-44ce-9667-aa39fdbe1c01\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-k7f54"
Nov 01 03:11:46.132692 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:46.132653    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:46.189936 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:46.189913    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-k7f54"
Nov 01 03:11:46.306926 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:46.306888    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:46.569218 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:46.569187    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-k7f54"]
Nov 01 03:11:46.570853 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:11:46.570829    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb06cb628_601b_44ce_9667_aa39fdbe1c01.slice/crio-5c441de339b5c304b8e08516aadc8800ae81718e1a897d5671bb5d2de8766dc3 WatchSource:0}: Error finding container 5c441de339b5c304b8e08516aadc8800ae81718e1a897d5671bb5d2de8766dc3: Status 404 returned error can't find the container with id 5c441de339b5c304b8e08516aadc8800ae81718e1a897d5671bb5d2de8766dc3
Nov 01 03:11:47.149358 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:47.149321    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-k7f54" event={"ID":"b06cb628-601b-44ce-9667-aa39fdbe1c01","Type":"ContainerStarted","Data":"5c441de339b5c304b8e08516aadc8800ae81718e1a897d5671bb5d2de8766dc3"}
Nov 01 03:11:47.491502 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:47.491425    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:47.775309 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:47.775224    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:47.932274 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:47.932240    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:48.109142 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:48.109115    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:48.280239 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:48.280209    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9"]
Nov 01 03:11:48.282390 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:48.282368    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9"
Nov 01 03:11:48.340010 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:48.339985    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-mh4mn\""
Nov 01 03:11:48.375807 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:48.375722    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:48.391537 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:48.391516    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9"]
Nov 01 03:11:48.440341 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:48.440318    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7785e6a9-3334-4855-9744-23a8295e3116-istio-data\") pod \"data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9\" (UID: \"7785e6a9-3334-4855-9744-23a8295e3116\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9"
Nov 01 03:11:48.440470 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:48.440356    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7785e6a9-3334-4855-9744-23a8295e3116-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9\" (UID: \"7785e6a9-3334-4855-9744-23a8295e3116\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9"
Nov 01 03:11:48.440470 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:48.440374    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7785e6a9-3334-4855-9744-23a8295e3116-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9\" (UID: \"7785e6a9-3334-4855-9744-23a8295e3116\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9"
Nov 01 03:11:48.440616 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:48.440512    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7785e6a9-3334-4855-9744-23a8295e3116-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9\" (UID: \"7785e6a9-3334-4855-9744-23a8295e3116\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9"
Nov 01 03:11:48.440616 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:48.440575    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7785e6a9-3334-4855-9744-23a8295e3116-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9\" (UID: \"7785e6a9-3334-4855-9744-23a8295e3116\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9"
Nov 01 03:11:48.440705 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:48.440633    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mswrd\" (UniqueName: \"kubernetes.io/projected/7785e6a9-3334-4855-9744-23a8295e3116-kube-api-access-mswrd\") pod \"data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9\" (UID: \"7785e6a9-3334-4855-9744-23a8295e3116\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9"
Nov 01 03:11:48.440705 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:48.440685    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7785e6a9-3334-4855-9744-23a8295e3116-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9\" (UID: \"7785e6a9-3334-4855-9744-23a8295e3116\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9"
Nov 01 03:11:48.440803 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:48.440708    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7785e6a9-3334-4855-9744-23a8295e3116-istio-token\") pod \"data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9\" (UID: \"7785e6a9-3334-4855-9744-23a8295e3116\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9"
Nov 01 03:11:48.440803 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:48.440793    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7785e6a9-3334-4855-9744-23a8295e3116-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9\" (UID: \"7785e6a9-3334-4855-9744-23a8295e3116\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9"
Nov 01 03:11:48.541517 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:48.541472    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7785e6a9-3334-4855-9744-23a8295e3116-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9\" (UID: \"7785e6a9-3334-4855-9744-23a8295e3116\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9"
Nov 01 03:11:48.541700 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:48.541543    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7785e6a9-3334-4855-9744-23a8295e3116-istio-data\") pod \"data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9\" (UID: \"7785e6a9-3334-4855-9744-23a8295e3116\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9"
Nov 01 03:11:48.541700 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:48.541602    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7785e6a9-3334-4855-9744-23a8295e3116-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9\" (UID: \"7785e6a9-3334-4855-9744-23a8295e3116\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9"
Nov 01 03:11:48.541700 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:48.541633    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7785e6a9-3334-4855-9744-23a8295e3116-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9\" (UID: \"7785e6a9-3334-4855-9744-23a8295e3116\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9"
Nov 01 03:11:48.541700 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:48.541678    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7785e6a9-3334-4855-9744-23a8295e3116-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9\" (UID: \"7785e6a9-3334-4855-9744-23a8295e3116\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9"
Nov 01 03:11:48.541884 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:48.541720    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7785e6a9-3334-4855-9744-23a8295e3116-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9\" (UID: \"7785e6a9-3334-4855-9744-23a8295e3116\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9"
Nov 01 03:11:48.541884 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:48.541756    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mswrd\" (UniqueName: \"kubernetes.io/projected/7785e6a9-3334-4855-9744-23a8295e3116-kube-api-access-mswrd\") pod \"data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9\" (UID: \"7785e6a9-3334-4855-9744-23a8295e3116\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9"
Nov 01 03:11:48.541884 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:48.541802    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7785e6a9-3334-4855-9744-23a8295e3116-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9\" (UID: \"7785e6a9-3334-4855-9744-23a8295e3116\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9"
Nov 01 03:11:48.541884 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:48.541830    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7785e6a9-3334-4855-9744-23a8295e3116-istio-token\") pod \"data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9\" (UID: \"7785e6a9-3334-4855-9744-23a8295e3116\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9"
Nov 01 03:11:48.542070 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:48.542043    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7785e6a9-3334-4855-9744-23a8295e3116-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9\" (UID: \"7785e6a9-3334-4855-9744-23a8295e3116\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9"
Nov 01 03:11:48.542070 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:48.542062    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7785e6a9-3334-4855-9744-23a8295e3116-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9\" (UID: \"7785e6a9-3334-4855-9744-23a8295e3116\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9"
Nov 01 03:11:48.542172 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:48.542149    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7785e6a9-3334-4855-9744-23a8295e3116-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9\" (UID: \"7785e6a9-3334-4855-9744-23a8295e3116\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9"
Nov 01 03:11:48.542313 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:48.542285    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7785e6a9-3334-4855-9744-23a8295e3116-istio-data\") pod \"data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9\" (UID: \"7785e6a9-3334-4855-9744-23a8295e3116\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9"
Nov 01 03:11:48.542755 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:48.542720    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7785e6a9-3334-4855-9744-23a8295e3116-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9\" (UID: \"7785e6a9-3334-4855-9744-23a8295e3116\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9"
Nov 01 03:11:48.544407 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:48.544383    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7785e6a9-3334-4855-9744-23a8295e3116-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9\" (UID: \"7785e6a9-3334-4855-9744-23a8295e3116\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9"
Nov 01 03:11:48.544894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:48.544873    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7785e6a9-3334-4855-9744-23a8295e3116-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9\" (UID: \"7785e6a9-3334-4855-9744-23a8295e3116\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9"
Nov 01 03:11:48.609147 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:48.609121    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7785e6a9-3334-4855-9744-23a8295e3116-istio-token\") pod \"data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9\" (UID: \"7785e6a9-3334-4855-9744-23a8295e3116\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9"
Nov 01 03:11:48.610688 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:48.610664    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mswrd\" (UniqueName: \"kubernetes.io/projected/7785e6a9-3334-4855-9744-23a8295e3116-kube-api-access-mswrd\") pod \"data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9\" (UID: \"7785e6a9-3334-4855-9744-23a8295e3116\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9"
Nov 01 03:11:48.698497 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:48.698426    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:48.896192 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:48.896155    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9"
Nov 01 03:11:49.112971 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:49.112942    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-rgb2s"
Nov 01 03:11:49.124429 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:11:49.124394    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:11:49.397811 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:49.397786    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9"]
Nov 01 03:11:49.400028 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:11:49.400000    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7785e6a9_3334_4855_9744_23a8295e3116.slice/crio-78876188de9cb377798ed932c1e81d6ddd86e5a5c5190de373d71597dd3611e1 WatchSource:0}: Error finding container 78876188de9cb377798ed932c1e81d6ddd86e5a5c5190de373d71597dd3611e1: Status 404 returned error can't find the container with id 78876188de9cb377798ed932c1e81d6ddd86e5a5c5190de373d71597dd3611e1
Nov 01 03:11:49.402012 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:49.401983    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:11:49.402075 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:49.402046    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:11:49.402114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:49.402074    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:11:50.167325 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:50.167284    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9" event={"ID":"7785e6a9-3334-4855-9744-23a8295e3116","Type":"ContainerStarted","Data":"190ed89f08dcb9a0c4831872dcdecb37d142d12e1ac68af6a8b632d9a6ee6a4d"}
Nov 01 03:11:50.167798 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:50.167347    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9" event={"ID":"7785e6a9-3334-4855-9744-23a8295e3116","Type":"ContainerStarted","Data":"78876188de9cb377798ed932c1e81d6ddd86e5a5c5190de373d71597dd3611e1"}
Nov 01 03:11:50.168980 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:50.168953    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-k7f54" event={"ID":"b06cb628-601b-44ce-9667-aa39fdbe1c01","Type":"ContainerStarted","Data":"c3084c147eea89f68d7b1a925e18bff8ab65f6bdabf8e6691df9e7e2ebd2c9f3"}
Nov 01 03:11:50.169088 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:50.169030    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-k7f54"
Nov 01 03:11:50.339413 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:50.339358    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9" podStartSLOduration=2.339344449 podStartE2EDuration="2.339344449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:11:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:11:50.337257792 +0000 UTC m=+314.860264381" watchObservedRunningTime="2026-11-01 03:11:50.339344449 +0000 UTC m=+314.862351034"
Nov 01 03:11:50.463754 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:50.463643    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-k7f54" podStartSLOduration=2.755724752 podStartE2EDuration="5.463630412s" totalImagesPullingTime="2.70790566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:11:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:11:50.46216169 +0000 UTC m=+314.985168290" watchObservedRunningTime="2026-11-01 03:11:50.463630412 +0000 UTC m=+314.986636999"
Nov 01 03:11:50.897358 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:50.897331    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9"
Nov 01 03:11:50.903450 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:50.903424    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9"
Nov 01 03:11:51.173916 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:51.173753    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9"
Nov 01 03:11:51.174979 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:51.174963    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9"
Nov 01 03:11:51.450180 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:51.450102    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:51.593197 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:51.593162    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:51.857790 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:51.857748    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:52.042655 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:52.042618    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:52.247626 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:52.247523    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:52.396319 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:52.396271    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:54.189779 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:54.189745    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b4sdg" event={"ID":"e4edaa96-e7f5-480f-9c85-b7189aedb11c","Type":"ContainerStarted","Data":"9d389f98887308f1f25fd47f24ba4ad9c22240696ad69ad51a8111fed9c43d7c"}
Nov 01 03:11:54.190403 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:54.189983    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b4sdg"
Nov 01 03:11:54.309690 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:54.309608    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b4sdg" podStartSLOduration=1.31016592 podStartE2EDuration="14.309595838s" totalImagesPullingTime="12.999429918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:11:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:11:54.307946278 +0000 UTC m=+318.830952867" watchObservedRunningTime="2026-11-01 03:11:54.309595838 +0000 UTC m=+318.832602425"
Nov 01 03:11:55.553076 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:55.553016    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:57.755203 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:57.755162    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:57.951967 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:57.951929    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:58.101674 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:58.101641    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:58.316409 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:58.316374    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:58.458621 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:58.458528    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:59.668471 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:59.668442    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-c5sng"]
Nov 01 03:11:59.671924 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:59.671906    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-c5sng"
Nov 01 03:11:59.724391 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:59.724366    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-47cjs\""
Nov 01 03:11:59.775507 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:59.775480    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-c5sng"]
Nov 01 03:11:59.818531 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:59.818501    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:11:59.835545 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:59.835520    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpwqg\" (UniqueName: \"kubernetes.io/projected/c1b8ab65-945d-42f6-9bce-d2919454c624-kube-api-access-qpwqg\") pod \"authorino-57f7f8659f-c5sng\" (UID: \"c1b8ab65-945d-42f6-9bce-d2919454c624\") " pod="kuadrant-system/authorino-57f7f8659f-c5sng"
Nov 01 03:11:59.936915 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:11:59.936840    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpwqg\" (UniqueName: \"kubernetes.io/projected/c1b8ab65-945d-42f6-9bce-d2919454c624-kube-api-access-qpwqg\") pod \"authorino-57f7f8659f-c5sng\" (UID: \"c1b8ab65-945d-42f6-9bce-d2919454c624\") " pod="kuadrant-system/authorino-57f7f8659f-c5sng"
Nov 01 03:12:00.010069 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:00.010042    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpwqg\" (UniqueName: \"kubernetes.io/projected/c1b8ab65-945d-42f6-9bce-d2919454c624-kube-api-access-qpwqg\") pod \"authorino-57f7f8659f-c5sng\" (UID: \"c1b8ab65-945d-42f6-9bce-d2919454c624\") " pod="kuadrant-system/authorino-57f7f8659f-c5sng"
Nov 01 03:12:00.041175 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:00.041145    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:12:00.293768 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:00.293700    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-c5sng"
Nov 01 03:12:00.668917 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:00.668891    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-c5sng"]
Nov 01 03:12:00.670653 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:12:00.670620    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1b8ab65_945d_42f6_9bce_d2919454c624.slice/crio-7d63e659aacb2ae6bdb46b89d5f6bd665be8c6993fe0429ec1f1996c15325106 WatchSource:0}: Error finding container 7d63e659aacb2ae6bdb46b89d5f6bd665be8c6993fe0429ec1f1996c15325106: Status 404 returned error can't find the container with id 7d63e659aacb2ae6bdb46b89d5f6bd665be8c6993fe0429ec1f1996c15325106
Nov 01 03:12:01.175738 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:01.175705    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-k7f54"
Nov 01 03:12:01.224419 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:01.224385    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-c5sng" event={"ID":"c1b8ab65-945d-42f6-9bce-d2919454c624","Type":"ContainerStarted","Data":"7d63e659aacb2ae6bdb46b89d5f6bd665be8c6993fe0429ec1f1996c15325106"}
Nov 01 03:12:02.520483 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:02.520445    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:12:02.803202 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:02.803112    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:12:04.241258 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:04.241216    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-c5sng" event={"ID":"c1b8ab65-945d-42f6-9bce-d2919454c624","Type":"ContainerStarted","Data":"3e04b19e34ed6105f1cdf0b92e7a608bffe1f246725999e788f10de32b51bfec"}
Nov 01 03:12:04.298974 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:04.298942    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:12:04.355878 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:04.355834    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-c5sng" podStartSLOduration=2.418930437 podStartE2EDuration="5.355823016s" totalImagesPullingTime="2.936892579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:11:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:12:04.355591111 +0000 UTC m=+328.878597700" watchObservedRunningTime="2026-11-01 03:12:04.355823016 +0000 UTC m=+328.878829603"
Nov 01 03:12:04.496885 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:04.496802    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:12:05.195988 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:05.195954    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b4sdg"
Nov 01 03:12:06.894678 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:06.894645    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:12:07.094193 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:07.094156    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:12:07.380573 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:07.380534    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-c5sng"]
Nov 01 03:12:07.380757 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:07.380734    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-c5sng" podUID="c1b8ab65-945d-42f6-9bce-d2919454c624" containerName="authorino" containerID="cri-o://3e04b19e34ed6105f1cdf0b92e7a608bffe1f246725999e788f10de32b51bfec" gracePeriod=30
Nov 01 03:12:07.641069 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:07.641012    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-c5sng"
Nov 01 03:12:07.804142 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:07.804113    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1b8ab65-945d-42f6-9bce-d2919454c624-kube-api-access-qpwqg\" (UniqueName: \"kubernetes.io/projected/c1b8ab65-945d-42f6-9bce-d2919454c624-kube-api-access-qpwqg\") pod \"c1b8ab65-945d-42f6-9bce-d2919454c624\" (UID: \"c1b8ab65-945d-42f6-9bce-d2919454c624\") "
Nov 01 03:12:07.806341 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:07.806316    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1b8ab65-945d-42f6-9bce-d2919454c624-kube-api-access-qpwqg" pod "c1b8ab65-945d-42f6-9bce-d2919454c624" (UID: "c1b8ab65-945d-42f6-9bce-d2919454c624"). InnerVolumeSpecName "kube-api-access-qpwqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:12:07.904868 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:07.904791    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpwqg\" (UniqueName: \"kubernetes.io/projected/c1b8ab65-945d-42f6-9bce-d2919454c624-kube-api-access-qpwqg\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:12:08.262351 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:08.262272    4930 generic.go:356] "Generic (PLEG): container finished" podID="c1b8ab65-945d-42f6-9bce-d2919454c624" containerID="3e04b19e34ed6105f1cdf0b92e7a608bffe1f246725999e788f10de32b51bfec" exitCode=0
Nov 01 03:12:08.262351 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:08.262323    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-c5sng"
Nov 01 03:12:08.262546 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:08.262350    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-c5sng" event={"ID":"c1b8ab65-945d-42f6-9bce-d2919454c624","Type":"ContainerDied","Data":"3e04b19e34ed6105f1cdf0b92e7a608bffe1f246725999e788f10de32b51bfec"}
Nov 01 03:12:08.262546 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:08.262392    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-c5sng" event={"ID":"c1b8ab65-945d-42f6-9bce-d2919454c624","Type":"ContainerDied","Data":"7d63e659aacb2ae6bdb46b89d5f6bd665be8c6993fe0429ec1f1996c15325106"}
Nov 01 03:12:08.262546 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:08.262409    4930 scope.go:122] "RemoveContainer" containerID="3e04b19e34ed6105f1cdf0b92e7a608bffe1f246725999e788f10de32b51bfec"
Nov 01 03:12:08.271304 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:08.271288    4930 scope.go:122] "RemoveContainer" containerID="3e04b19e34ed6105f1cdf0b92e7a608bffe1f246725999e788f10de32b51bfec"
Nov 01 03:12:08.271589 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:12:08.271550    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e04b19e34ed6105f1cdf0b92e7a608bffe1f246725999e788f10de32b51bfec\": container with ID starting with 3e04b19e34ed6105f1cdf0b92e7a608bffe1f246725999e788f10de32b51bfec not found: ID does not exist" containerID="3e04b19e34ed6105f1cdf0b92e7a608bffe1f246725999e788f10de32b51bfec"
Nov 01 03:12:08.271642 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:08.271602    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e04b19e34ed6105f1cdf0b92e7a608bffe1f246725999e788f10de32b51bfec"} err="failed to get container status \"3e04b19e34ed6105f1cdf0b92e7a608bffe1f246725999e788f10de32b51bfec\": rpc error: code = NotFound desc = could not find container \"3e04b19e34ed6105f1cdf0b92e7a608bffe1f246725999e788f10de32b51bfec\": container with ID starting with 3e04b19e34ed6105f1cdf0b92e7a608bffe1f246725999e788f10de32b51bfec not found: ID does not exist"
Nov 01 03:12:08.300318 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:08.300297    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:12:08.425830 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:08.425802    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-c5sng"]
Nov 01 03:12:08.440362 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:08.440339    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-c5sng"]
Nov 01 03:12:08.465825 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:08.465792    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:12:08.614311 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:08.614281    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:12:09.618525 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:09.618492    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1b8ab65-945d-42f6-9bce-d2919454c624" path="/var/lib/kubelet/pods/c1b8ab65-945d-42f6-9bce-d2919454c624/volumes"
Nov 01 03:12:09.787956 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:09.787921    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6794d9ffb8-c52ff"]
Nov 01 03:12:09.788342 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:09.788323    4930 cpu_manager.go:414] "containerMap: removing container" podUID="c1b8ab65-945d-42f6-9bce-d2919454c624" containerName="authorino"
Nov 01 03:12:09.788429 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:09.788343    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1b8ab65-945d-42f6-9bce-d2919454c624" containerName="authorino"
Nov 01 03:12:09.788484 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:09.788460    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1b8ab65-945d-42f6-9bce-d2919454c624" containerName="authorino"
Nov 01 03:12:09.794830 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:09.794808    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6794d9ffb8-c52ff"
Nov 01 03:12:09.850699 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:09.850665    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6794d9ffb8-c52ff"]
Nov 01 03:12:09.898548 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:09.898490    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Nov 01 03:12:09.898548 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:09.898526    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Nov 01 03:12:09.903060 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:09.903043    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 01 03:12:09.903060 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:09.903057    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-c8b6b\""
Nov 01 03:12:09.903147 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:09.903049    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Nov 01 03:12:09.903147 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:09.903059    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 01 03:12:09.918512 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:09.918493    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1000c3a1-251a-44b9-b24c-739477dace22-metrics-certs\") pod \"jobset-controller-manager-6794d9ffb8-c52ff\" (UID: \"1000c3a1-251a-44b9-b24c-739477dace22\") " pod="openshift-jobset-operator/jobset-controller-manager-6794d9ffb8-c52ff"
Nov 01 03:12:09.918598 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:09.918543    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1000c3a1-251a-44b9-b24c-739477dace22-cert\") pod \"jobset-controller-manager-6794d9ffb8-c52ff\" (UID: \"1000c3a1-251a-44b9-b24c-739477dace22\") " pod="openshift-jobset-operator/jobset-controller-manager-6794d9ffb8-c52ff"
Nov 01 03:12:09.918665 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:09.918627    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1000c3a1-251a-44b9-b24c-739477dace22-manager-config\") pod \"jobset-controller-manager-6794d9ffb8-c52ff\" (UID: \"1000c3a1-251a-44b9-b24c-739477dace22\") " pod="openshift-jobset-operator/jobset-controller-manager-6794d9ffb8-c52ff"
Nov 01 03:12:09.918744 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:09.918725    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8hqt\" (UniqueName: \"kubernetes.io/projected/1000c3a1-251a-44b9-b24c-739477dace22-kube-api-access-v8hqt\") pod \"jobset-controller-manager-6794d9ffb8-c52ff\" (UID: \"1000c3a1-251a-44b9-b24c-739477dace22\") " pod="openshift-jobset-operator/jobset-controller-manager-6794d9ffb8-c52ff"
Nov 01 03:12:10.019468 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:10.019442    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8hqt\" (UniqueName: \"kubernetes.io/projected/1000c3a1-251a-44b9-b24c-739477dace22-kube-api-access-v8hqt\") pod \"jobset-controller-manager-6794d9ffb8-c52ff\" (UID: \"1000c3a1-251a-44b9-b24c-739477dace22\") " pod="openshift-jobset-operator/jobset-controller-manager-6794d9ffb8-c52ff"
Nov 01 03:12:10.019607 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:10.019493    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1000c3a1-251a-44b9-b24c-739477dace22-metrics-certs\") pod \"jobset-controller-manager-6794d9ffb8-c52ff\" (UID: \"1000c3a1-251a-44b9-b24c-739477dace22\") " pod="openshift-jobset-operator/jobset-controller-manager-6794d9ffb8-c52ff"
Nov 01 03:12:10.019607 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:10.019530    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1000c3a1-251a-44b9-b24c-739477dace22-cert\") pod \"jobset-controller-manager-6794d9ffb8-c52ff\" (UID: \"1000c3a1-251a-44b9-b24c-739477dace22\") " pod="openshift-jobset-operator/jobset-controller-manager-6794d9ffb8-c52ff"
Nov 01 03:12:10.019607 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:10.019547    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1000c3a1-251a-44b9-b24c-739477dace22-manager-config\") pod \"jobset-controller-manager-6794d9ffb8-c52ff\" (UID: \"1000c3a1-251a-44b9-b24c-739477dace22\") " pod="openshift-jobset-operator/jobset-controller-manager-6794d9ffb8-c52ff"
Nov 01 03:12:10.020301 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:10.020279    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1000c3a1-251a-44b9-b24c-739477dace22-manager-config\") pod \"jobset-controller-manager-6794d9ffb8-c52ff\" (UID: \"1000c3a1-251a-44b9-b24c-739477dace22\") " pod="openshift-jobset-operator/jobset-controller-manager-6794d9ffb8-c52ff"
Nov 01 03:12:10.021963 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:10.021944    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1000c3a1-251a-44b9-b24c-739477dace22-metrics-certs\") pod \"jobset-controller-manager-6794d9ffb8-c52ff\" (UID: \"1000c3a1-251a-44b9-b24c-739477dace22\") " pod="openshift-jobset-operator/jobset-controller-manager-6794d9ffb8-c52ff"
Nov 01 03:12:10.022041 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:10.021974    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1000c3a1-251a-44b9-b24c-739477dace22-cert\") pod \"jobset-controller-manager-6794d9ffb8-c52ff\" (UID: \"1000c3a1-251a-44b9-b24c-739477dace22\") " pod="openshift-jobset-operator/jobset-controller-manager-6794d9ffb8-c52ff"
Nov 01 03:12:10.101039 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:10.101012    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8hqt\" (UniqueName: \"kubernetes.io/projected/1000c3a1-251a-44b9-b24c-739477dace22-kube-api-access-v8hqt\") pod \"jobset-controller-manager-6794d9ffb8-c52ff\" (UID: \"1000c3a1-251a-44b9-b24c-739477dace22\") " pod="openshift-jobset-operator/jobset-controller-manager-6794d9ffb8-c52ff"
Nov 01 03:12:10.106760 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:10.106732    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6794d9ffb8-c52ff"
Nov 01 03:12:10.404261 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:10.404188    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:12:10.481150 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:10.481129    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6794d9ffb8-c52ff"]
Nov 01 03:12:10.483067 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:12:10.483039    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1000c3a1_251a_44b9_b24c_739477dace22.slice/crio-b3135bfe7594bac8f5422c90923b7afb96007bcd3acfa2479b82d85e6f1eb814 WatchSource:0}: Error finding container b3135bfe7594bac8f5422c90923b7afb96007bcd3acfa2479b82d85e6f1eb814: Status 404 returned error can't find the container with id b3135bfe7594bac8f5422c90923b7afb96007bcd3acfa2479b82d85e6f1eb814
Nov 01 03:12:10.613732 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:10.613699    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:12:11.278318 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:11.278285    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6794d9ffb8-c52ff" event={"ID":"1000c3a1-251a-44b9-b24c-739477dace22","Type":"ContainerStarted","Data":"b3135bfe7594bac8f5422c90923b7afb96007bcd3acfa2479b82d85e6f1eb814"}
Nov 01 03:12:13.534547 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:13.534499    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:12:13.675411 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:13.675368    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:12:15.299497 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:15.299461    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6794d9ffb8-c52ff" event={"ID":"1000c3a1-251a-44b9-b24c-739477dace22","Type":"ContainerStarted","Data":"bb8d81a78dd3285d723a5e5103823fa7d494bca6ddecf3c6394bdcb32792f2c3"}
Nov 01 03:12:15.299976 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:15.299587    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-6794d9ffb8-c52ff"
Nov 01 03:12:15.415486 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:15.415430    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-6794d9ffb8-c52ff" podStartSLOduration=2.475634274 podStartE2EDuration="6.415414037s" totalImagesPullingTime="3.939779763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:12:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:12:15.415312933 +0000 UTC m=+339.938319521" watchObservedRunningTime="2026-11-01 03:12:15.415414037 +0000 UTC m=+339.938420627"
Nov 01 03:12:16.744671 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:16.744641    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:12:16.934035 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:16.933999    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:12:19.121495 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:12:19.121462    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:12:23.315988 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:23.315949    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:12:23.469645 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:23.469608    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:12:24.191187 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:24.191141    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:12:25.951101 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:25.951063    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:12:26.108746 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:26.108711    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:12:26.309973 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:26.309886    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-6794d9ffb8-c52ff"
Nov 01 03:12:27.433422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:27.433386    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:12:29.509500 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:29.509464    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:12:29.660404 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:29.660371    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:12:29.906770 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:29.906720    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:12:30.053138 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:30.053108    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:12:30.200765 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:30.200653    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:12:36.181990 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:36.181956    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:12:36.388199 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:36.388167    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:12:37.515480 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:37.515441    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:12:42.716500 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:42.716467    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:12:42.862671 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:42.862628    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:12:44.873582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:44.871629    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:12:45.014703 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:45.014665    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:12:45.217730 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:45.217648    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:12:45.374435 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:45.374404    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:12:45.521290 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:45.521191    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:12:49.121285 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:12:49.121239    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:12:50.018466 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:50.018429    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:12:50.164082 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:50.164030    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:12:50.346805 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:50.346776    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:12:50.495132 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:50.495096    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:12:53.155104 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:53.155067    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:12:53.435929 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:12:53.435845    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:13:18.673577 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:18.673453    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:13:18.823463 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:18.823417    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:13:19.124970 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:13:19.124937    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:13:20.730547 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:20.730509    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:13:20.927712 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:20.927649    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:13:21.075788 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:21.075712    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:13:21.220664 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:21.220632    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:13:21.369396 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:21.369359    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:13:21.518912 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:21.518868    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:13:21.677112 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:21.677019    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:13:21.890446 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:21.890417    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:13:23.106013 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:23.105976    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:13:23.903177 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:23.903124    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:13:24.054818 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:24.054785    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:13:24.205103 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:24.205007    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:13:24.351730 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:24.351696    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:13:27.782678 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:27.782638    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:13:28.384087 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:28.384043    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:13:28.538063 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:28.538027    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:13:28.742096 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:28.741989    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:13:28.893910 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:28.893876    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:13:29.052721 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:29.052628    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:13:29.241978 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:29.241934    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:13:29.417009 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:29.416975    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:13:29.585057 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:29.585015    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:13:34.849998 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:34.849966    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:13:35.073992 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:35.073959    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:13:35.193052 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:35.192975    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-c266n"]
Nov 01 03:13:35.196492 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:35.196471    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-c266n"
Nov 01 03:13:35.223465 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:35.223443    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k56rs\" (UniqueName: \"kubernetes.io/projected/9631d446-8b76-4d97-957c-827090ab69f6-kube-api-access-k56rs\") pod \"authorino-6f6944bcdd-c266n\" (UID: \"9631d446-8b76-4d97-957c-827090ab69f6\") " pod="kuadrant-system/authorino-6f6944bcdd-c266n"
Nov 01 03:13:35.249209 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:35.249186    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-47cjs\""
Nov 01 03:13:35.300088 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:35.300061    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-c266n"]
Nov 01 03:13:35.324404 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:35.324373    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k56rs\" (UniqueName: \"kubernetes.io/projected/9631d446-8b76-4d97-957c-827090ab69f6-kube-api-access-k56rs\") pod \"authorino-6f6944bcdd-c266n\" (UID: \"9631d446-8b76-4d97-957c-827090ab69f6\") " pod="kuadrant-system/authorino-6f6944bcdd-c266n"
Nov 01 03:13:35.393193 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:35.393159    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k56rs\" (UniqueName: \"kubernetes.io/projected/9631d446-8b76-4d97-957c-827090ab69f6-kube-api-access-k56rs\") pod \"authorino-6f6944bcdd-c266n\" (UID: \"9631d446-8b76-4d97-957c-827090ab69f6\") " pod="kuadrant-system/authorino-6f6944bcdd-c266n"
Nov 01 03:13:35.526181 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:35.526115    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-c266n"
Nov 01 03:13:35.564069 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:35.563996    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-c266n"]
Nov 01 03:13:35.602315 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:35.602285    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-cc764fd49-psr9v"]
Nov 01 03:13:35.605945 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:35.605926    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cc764fd49-psr9v"
Nov 01 03:13:35.627187 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:35.627161    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvscd\" (UniqueName: \"kubernetes.io/projected/a7350630-9345-4af0-a109-a2aeca497c8d-kube-api-access-qvscd\") pod \"authorino-cc764fd49-psr9v\" (UID: \"a7350630-9345-4af0-a109-a2aeca497c8d\") " pod="kuadrant-system/authorino-cc764fd49-psr9v"
Nov 01 03:13:35.698678 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:35.698647    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-cc764fd49-psr9v"]
Nov 01 03:13:35.728062 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:35.728026    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvscd\" (UniqueName: \"kubernetes.io/projected/a7350630-9345-4af0-a109-a2aeca497c8d-kube-api-access-qvscd\") pod \"authorino-cc764fd49-psr9v\" (UID: \"a7350630-9345-4af0-a109-a2aeca497c8d\") " pod="kuadrant-system/authorino-cc764fd49-psr9v"
Nov 01 03:13:35.791493 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:35.791439    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvscd\" (UniqueName: \"kubernetes.io/projected/a7350630-9345-4af0-a109-a2aeca497c8d-kube-api-access-qvscd\") pod \"authorino-cc764fd49-psr9v\" (UID: \"a7350630-9345-4af0-a109-a2aeca497c8d\") " pod="kuadrant-system/authorino-cc764fd49-psr9v"
Nov 01 03:13:35.916999 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:35.916974    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-c266n"]
Nov 01 03:13:35.919445 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:13:35.919419    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9631d446_8b76_4d97_957c_827090ab69f6.slice/crio-d341167bf5c7ac6d44265f3eacfcd5cf205012bd4b25e51847f84263dc2b657e WatchSource:0}: Error finding container d341167bf5c7ac6d44265f3eacfcd5cf205012bd4b25e51847f84263dc2b657e: Status 404 returned error can't find the container with id d341167bf5c7ac6d44265f3eacfcd5cf205012bd4b25e51847f84263dc2b657e
Nov 01 03:13:35.920577 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:35.920545    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cc764fd49-psr9v"
Nov 01 03:13:36.085629 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:36.085604    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-cc764fd49-psr9v"]
Nov 01 03:13:36.292181 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:36.292040    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-cc764fd49-psr9v"]
Nov 01 03:13:36.294698 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:13:36.294671    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7350630_9345_4af0_a109_a2aeca497c8d.slice/crio-daf676471d0b5df76c4d16d570b067748f02ced3e8765a017a4ef101e39313f1 WatchSource:0}: Error finding container daf676471d0b5df76c4d16d570b067748f02ced3e8765a017a4ef101e39313f1: Status 404 returned error can't find the container with id daf676471d0b5df76c4d16d570b067748f02ced3e8765a017a4ef101e39313f1
Nov 01 03:13:36.721264 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:36.721230    4930 generic.go:356] "Generic (PLEG): container finished" podID="24e9b838-6a66-49b0-9aec-c43bfd9221ae" containerID="bf4395ebac7bfe9834d0aa8d5038db9353479279937c9e6d106f6e515b87399f" exitCode=0
Nov 01 03:13:36.721480 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:36.721303    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5" event={"ID":"24e9b838-6a66-49b0-9aec-c43bfd9221ae","Type":"ContainerDied","Data":"bf4395ebac7bfe9834d0aa8d5038db9353479279937c9e6d106f6e515b87399f"}
Nov 01 03:13:36.721769 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:36.721745    4930 scope.go:122] "RemoveContainer" containerID="bf4395ebac7bfe9834d0aa8d5038db9353479279937c9e6d106f6e515b87399f"
Nov 01 03:13:36.722681 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:36.722547    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cc764fd49-psr9v" event={"ID":"a7350630-9345-4af0-a109-a2aeca497c8d","Type":"ContainerStarted","Data":"daf676471d0b5df76c4d16d570b067748f02ced3e8765a017a4ef101e39313f1"}
Nov 01 03:13:36.723025 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:36.722995    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:13:36.723126 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:36.723068    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:13:36.723126 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:36.723108    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:13:36.723827 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:36.723800    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-c266n" event={"ID":"9631d446-8b76-4d97-957c-827090ab69f6","Type":"ContainerStarted","Data":"d341167bf5c7ac6d44265f3eacfcd5cf205012bd4b25e51847f84263dc2b657e"}
Nov 01 03:13:37.730497 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:37.730459    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5" event={"ID":"24e9b838-6a66-49b0-9aec-c43bfd9221ae","Type":"ContainerStarted","Data":"7da7748e972490cf4d1f1b2791d55b107b7bcf9eb09d19ece15ddc709ae82f77"}
Nov 01 03:13:37.732020 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:37.731996    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cc764fd49-psr9v" event={"ID":"a7350630-9345-4af0-a109-a2aeca497c8d","Type":"ContainerStarted","Data":"2dde54d755df81cd3d55603f99f3fb36cd8aa10df6e06c4283fec4402b218e53"}
Nov 01 03:13:37.732140 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:37.732025    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-cc764fd49-psr9v" podUID="a7350630-9345-4af0-a109-a2aeca497c8d" containerName="authorino" containerID="cri-o://2dde54d755df81cd3d55603f99f3fb36cd8aa10df6e06c4283fec4402b218e53" gracePeriod=30
Nov 01 03:13:37.733605 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:37.733579    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-c266n" event={"ID":"9631d446-8b76-4d97-957c-827090ab69f6","Type":"ContainerStarted","Data":"fd7575cea32e81c27550c61f9b932e13e56e01d9b20bff34ba6e3a6d60626d70"}
Nov 01 03:13:37.733722 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:37.733651    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-c266n" podUID="9631d446-8b76-4d97-957c-827090ab69f6" containerName="authorino" containerID="cri-o://fd7575cea32e81c27550c61f9b932e13e56e01d9b20bff34ba6e3a6d60626d70" gracePeriod=30
Nov 01 03:13:37.846404 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:37.846366    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:13:37.987581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:37.985846    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-c266n" podStartSLOduration=1.400634527 podStartE2EDuration="2.985822356s" totalImagesPullingTime="1.585187829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:13:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:13:37.98494006 +0000 UTC m=+422.507946673" watchObservedRunningTime="2026-11-01 03:13:37.985822356 +0000 UTC m=+422.508828946"
Nov 01 03:13:38.036410 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:38.036386    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-c266n"
Nov 01 03:13:38.047150 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:38.047089    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9631d446-8b76-4d97-957c-827090ab69f6-kube-api-access-k56rs\" (UniqueName: \"kubernetes.io/projected/9631d446-8b76-4d97-957c-827090ab69f6-kube-api-access-k56rs\") pod \"9631d446-8b76-4d97-957c-827090ab69f6\" (UID: \"9631d446-8b76-4d97-957c-827090ab69f6\") "
Nov 01 03:13:38.049166 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:38.049142    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9631d446-8b76-4d97-957c-827090ab69f6-kube-api-access-k56rs" pod "9631d446-8b76-4d97-957c-827090ab69f6" (UID: "9631d446-8b76-4d97-957c-827090ab69f6"). InnerVolumeSpecName "kube-api-access-k56rs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:13:38.115429 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:38.115384    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-cc764fd49-psr9v" podStartSLOduration=1.856995333 podStartE2EDuration="3.1153662s" totalImagesPullingTime="1.258370867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:13:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:13:38.11386729 +0000 UTC m=+422.636873879" watchObservedRunningTime="2026-11-01 03:13:38.1153662 +0000 UTC m=+422.638372804"
Nov 01 03:13:38.122474 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:38.122449    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cc764fd49-psr9v"
Nov 01 03:13:38.147630 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:38.147607    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7350630-9345-4af0-a109-a2aeca497c8d-kube-api-access-qvscd\" (UniqueName: \"kubernetes.io/projected/a7350630-9345-4af0-a109-a2aeca497c8d-kube-api-access-qvscd\") pod \"a7350630-9345-4af0-a109-a2aeca497c8d\" (UID: \"a7350630-9345-4af0-a109-a2aeca497c8d\") "
Nov 01 03:13:38.147879 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:38.147861    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k56rs\" (UniqueName: \"kubernetes.io/projected/9631d446-8b76-4d97-957c-827090ab69f6-kube-api-access-k56rs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:13:38.149508 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:38.149487    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7350630-9345-4af0-a109-a2aeca497c8d-kube-api-access-qvscd" pod "a7350630-9345-4af0-a109-a2aeca497c8d" (UID: "a7350630-9345-4af0-a109-a2aeca497c8d"). InnerVolumeSpecName "kube-api-access-qvscd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:13:38.248456 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:38.248422    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvscd\" (UniqueName: \"kubernetes.io/projected/a7350630-9345-4af0-a109-a2aeca497c8d-kube-api-access-qvscd\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:13:38.358295 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:38.358270    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"
Nov 01 03:13:38.383579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:38.382078    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-zscr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 01 03:13:38.383579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:38.382227    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5" podUID="24e9b838-6a66-49b0-9aec-c43bfd9221ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 01 03:13:38.746111 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:38.746028    4930 generic.go:356] "Generic (PLEG): container finished" podID="a7350630-9345-4af0-a109-a2aeca497c8d" containerID="2dde54d755df81cd3d55603f99f3fb36cd8aa10df6e06c4283fec4402b218e53" exitCode=0
Nov 01 03:13:38.746111 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:38.746074    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cc764fd49-psr9v"
Nov 01 03:13:38.746576 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:38.746103    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cc764fd49-psr9v" event={"ID":"a7350630-9345-4af0-a109-a2aeca497c8d","Type":"ContainerDied","Data":"2dde54d755df81cd3d55603f99f3fb36cd8aa10df6e06c4283fec4402b218e53"}
Nov 01 03:13:38.746576 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:38.746147    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cc764fd49-psr9v" event={"ID":"a7350630-9345-4af0-a109-a2aeca497c8d","Type":"ContainerDied","Data":"daf676471d0b5df76c4d16d570b067748f02ced3e8765a017a4ef101e39313f1"}
Nov 01 03:13:38.746576 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:38.746172    4930 scope.go:122] "RemoveContainer" containerID="2dde54d755df81cd3d55603f99f3fb36cd8aa10df6e06c4283fec4402b218e53"
Nov 01 03:13:38.747851 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:38.747828    4930 generic.go:356] "Generic (PLEG): container finished" podID="9631d446-8b76-4d97-957c-827090ab69f6" containerID="fd7575cea32e81c27550c61f9b932e13e56e01d9b20bff34ba6e3a6d60626d70" exitCode=0
Nov 01 03:13:38.747943 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:38.747872    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-c266n"
Nov 01 03:13:38.747943 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:38.747913    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-c266n" event={"ID":"9631d446-8b76-4d97-957c-827090ab69f6","Type":"ContainerDied","Data":"fd7575cea32e81c27550c61f9b932e13e56e01d9b20bff34ba6e3a6d60626d70"}
Nov 01 03:13:38.748042 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:38.747954    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-c266n" event={"ID":"9631d446-8b76-4d97-957c-827090ab69f6","Type":"ContainerDied","Data":"d341167bf5c7ac6d44265f3eacfcd5cf205012bd4b25e51847f84263dc2b657e"}
Nov 01 03:13:38.758444 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:38.758428    4930 scope.go:122] "RemoveContainer" containerID="2dde54d755df81cd3d55603f99f3fb36cd8aa10df6e06c4283fec4402b218e53"
Nov 01 03:13:38.777575 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:13:38.765727    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dde54d755df81cd3d55603f99f3fb36cd8aa10df6e06c4283fec4402b218e53\": container with ID starting with 2dde54d755df81cd3d55603f99f3fb36cd8aa10df6e06c4283fec4402b218e53 not found: ID does not exist" containerID="2dde54d755df81cd3d55603f99f3fb36cd8aa10df6e06c4283fec4402b218e53"
Nov 01 03:13:38.777575 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:38.765767    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dde54d755df81cd3d55603f99f3fb36cd8aa10df6e06c4283fec4402b218e53"} err="failed to get container status \"2dde54d755df81cd3d55603f99f3fb36cd8aa10df6e06c4283fec4402b218e53\": rpc error: code = NotFound desc = could not find container \"2dde54d755df81cd3d55603f99f3fb36cd8aa10df6e06c4283fec4402b218e53\": container with ID starting with 2dde54d755df81cd3d55603f99f3fb36cd8aa10df6e06c4283fec4402b218e53 not found: ID does not exist"
Nov 01 03:13:38.777575 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:38.765791    4930 scope.go:122] "RemoveContainer" containerID="fd7575cea32e81c27550c61f9b932e13e56e01d9b20bff34ba6e3a6d60626d70"
Nov 01 03:13:38.794720 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:38.794697    4930 scope.go:122] "RemoveContainer" containerID="fd7575cea32e81c27550c61f9b932e13e56e01d9b20bff34ba6e3a6d60626d70"
Nov 01 03:13:38.795022 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:13:38.795003    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd7575cea32e81c27550c61f9b932e13e56e01d9b20bff34ba6e3a6d60626d70\": container with ID starting with fd7575cea32e81c27550c61f9b932e13e56e01d9b20bff34ba6e3a6d60626d70 not found: ID does not exist" containerID="fd7575cea32e81c27550c61f9b932e13e56e01d9b20bff34ba6e3a6d60626d70"
Nov 01 03:13:38.795077 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:38.795031    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd7575cea32e81c27550c61f9b932e13e56e01d9b20bff34ba6e3a6d60626d70"} err="failed to get container status \"fd7575cea32e81c27550c61f9b932e13e56e01d9b20bff34ba6e3a6d60626d70\": rpc error: code = NotFound desc = could not find container \"fd7575cea32e81c27550c61f9b932e13e56e01d9b20bff34ba6e3a6d60626d70\": container with ID starting with fd7575cea32e81c27550c61f9b932e13e56e01d9b20bff34ba6e3a6d60626d70 not found: ID does not exist"
Nov 01 03:13:38.947200 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:38.947158    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-cc764fd49-psr9v"]
Nov 01 03:13:38.952132 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:38.952110    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-cc764fd49-psr9v"]
Nov 01 03:13:39.083632 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:39.081617    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-c266n"]
Nov 01 03:13:39.085093 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:39.085072    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-c266n"]
Nov 01 03:13:39.358266 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:39.358232    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-zscr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 01 03:13:39.358451 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:39.358292    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5" podUID="24e9b838-6a66-49b0-9aec-c43bfd9221ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 01 03:13:39.618429 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:39.618350    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9631d446-8b76-4d97-957c-827090ab69f6" path="/var/lib/kubelet/pods/9631d446-8b76-4d97-957c-827090ab69f6/volumes"
Nov 01 03:13:39.618712 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:39.618698    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7350630-9345-4af0-a109-a2aeca497c8d" path="/var/lib/kubelet/pods/a7350630-9345-4af0-a109-a2aeca497c8d/volumes"
Nov 01 03:13:39.783710 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:39.783677    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"
Nov 01 03:13:40.358353 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:40.358319    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-zscr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 01 03:13:40.358534 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:40.358373    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5" podUID="24e9b838-6a66-49b0-9aec-c43bfd9221ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 01 03:13:41.358570 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:41.358532    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-zscr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 01 03:13:41.359096 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:41.358635    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5" podUID="24e9b838-6a66-49b0-9aec-c43bfd9221ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 01 03:13:42.358237 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:42.358198    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-zscr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 01 03:13:42.358421 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:42.358261    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5" podUID="24e9b838-6a66-49b0-9aec-c43bfd9221ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 01 03:13:43.362127 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:43.362093    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"
Nov 01 03:13:43.363156 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:43.363135    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"
Nov 01 03:13:44.146047 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:13:44.146008    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:13:49.121152 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:13:49.121111    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:14:19.120246 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:14:19.120210    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:14:28.132169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:14:28.132128    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:14:38.160919 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:14:38.160877    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:14:49.120412 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:14:49.120335    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:15:00.182104 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:00.182070    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891715-g442j"]
Nov 01 03:15:00.182949 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:00.182402    4930 cpu_manager.go:414] "containerMap: removing container" podUID="a7350630-9345-4af0-a109-a2aeca497c8d" containerName="authorino"
Nov 01 03:15:00.182949 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:00.182412    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7350630-9345-4af0-a109-a2aeca497c8d" containerName="authorino"
Nov 01 03:15:00.182949 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:00.182443    4930 cpu_manager.go:414] "containerMap: removing container" podUID="9631d446-8b76-4d97-957c-827090ab69f6" containerName="authorino"
Nov 01 03:15:00.182949 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:00.182448    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9631d446-8b76-4d97-957c-827090ab69f6" containerName="authorino"
Nov 01 03:15:00.182949 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:00.182505    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="9631d446-8b76-4d97-957c-827090ab69f6" containerName="authorino"
Nov 01 03:15:00.182949 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:00.182513    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7350630-9345-4af0-a109-a2aeca497c8d" containerName="authorino"
Nov 01 03:15:00.185590 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:00.185575    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891715-g442j"
Nov 01 03:15:00.203941 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:00.203918    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cc1a309-9843-4ce7-ad17-d6c72ade9425-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891715-g442j\" (UID: \"2cc1a309-9843-4ce7-ad17-d6c72ade9425\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891715-g442j"
Nov 01 03:15:00.204021 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:00.203949    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v4nl\" (UniqueName: \"kubernetes.io/projected/2cc1a309-9843-4ce7-ad17-d6c72ade9425-kube-api-access-6v4nl\") pod \"maas-api-key-cleanup-29891715-g442j\" (UID: \"2cc1a309-9843-4ce7-ad17-d6c72ade9425\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891715-g442j"
Nov 01 03:15:00.239129 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:00.239109    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:15:00.288748 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:00.288727    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:15:00.288748 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:00.288736    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:15:00.290540 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:00.290518    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891715-g442j"]
Nov 01 03:15:00.305145 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:00.305122    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cc1a309-9843-4ce7-ad17-d6c72ade9425-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891715-g442j\" (UID: \"2cc1a309-9843-4ce7-ad17-d6c72ade9425\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891715-g442j"
Nov 01 03:15:00.305230 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:00.305151    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v4nl\" (UniqueName: \"kubernetes.io/projected/2cc1a309-9843-4ce7-ad17-d6c72ade9425-kube-api-access-6v4nl\") pod \"maas-api-key-cleanup-29891715-g442j\" (UID: \"2cc1a309-9843-4ce7-ad17-d6c72ade9425\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891715-g442j"
Nov 01 03:15:00.305751 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:00.305734    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cc1a309-9843-4ce7-ad17-d6c72ade9425-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891715-g442j\" (UID: \"2cc1a309-9843-4ce7-ad17-d6c72ade9425\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891715-g442j"
Nov 01 03:15:00.367353 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:00.367332    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v4nl\" (UniqueName: \"kubernetes.io/projected/2cc1a309-9843-4ce7-ad17-d6c72ade9425-kube-api-access-6v4nl\") pod \"maas-api-key-cleanup-29891715-g442j\" (UID: \"2cc1a309-9843-4ce7-ad17-d6c72ade9425\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891715-g442j"
Nov 01 03:15:00.497945 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:00.497893    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891715-g442j"
Nov 01 03:15:00.866711 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:00.866689    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891715-g442j"]
Nov 01 03:15:00.868906 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:15:00.868881    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cc1a309_9843_4ce7_ad17_d6c72ade9425.slice/crio-c2c1e2f0e5de68f67bd2700dd3d6416912dfdda4c41619b1d23f3bdd459549b0 WatchSource:0}: Error finding container c2c1e2f0e5de68f67bd2700dd3d6416912dfdda4c41619b1d23f3bdd459549b0: Status 404 returned error can't find the container with id c2c1e2f0e5de68f67bd2700dd3d6416912dfdda4c41619b1d23f3bdd459549b0
Nov 01 03:15:01.161461 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:01.161365    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891715-g442j" event={"ID":"2cc1a309-9843-4ce7-ad17-d6c72ade9425","Type":"ContainerStarted","Data":"c2c1e2f0e5de68f67bd2700dd3d6416912dfdda4c41619b1d23f3bdd459549b0"}
Nov 01 03:15:06.188474 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:06.188443    4930 generic.go:356] "Generic (PLEG): container finished" podID="2cc1a309-9843-4ce7-ad17-d6c72ade9425" containerID="9bea41c0376c5db94b5296f994d92275b10fd644acec0ab9b48cebcc83260290" exitCode=60
Nov 01 03:15:06.188828 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:06.188520    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891715-g442j" event={"ID":"2cc1a309-9843-4ce7-ad17-d6c72ade9425","Type":"ContainerDied","Data":"9bea41c0376c5db94b5296f994d92275b10fd644acec0ab9b48cebcc83260290"}
Nov 01 03:15:06.188828 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:06.188704    4930 scope.go:122] "RemoveContainer" containerID="9bea41c0376c5db94b5296f994d92275b10fd644acec0ab9b48cebcc83260290"
Nov 01 03:15:07.195625 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:07.195537    4930 generic.go:356] "Generic (PLEG): container finished" podID="2cc1a309-9843-4ce7-ad17-d6c72ade9425" containerID="b48dd3fa9ac8bd51d0dbbd7b816ad18851e1327f5bce04434886c42fdcbcd265" exitCode=60
Nov 01 03:15:07.196023 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:07.195620    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891715-g442j" event={"ID":"2cc1a309-9843-4ce7-ad17-d6c72ade9425","Type":"ContainerDied","Data":"b48dd3fa9ac8bd51d0dbbd7b816ad18851e1327f5bce04434886c42fdcbcd265"}
Nov 01 03:15:07.196023 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:07.195697    4930 scope.go:122] "RemoveContainer" containerID="9bea41c0376c5db94b5296f994d92275b10fd644acec0ab9b48cebcc83260290"
Nov 01 03:15:07.196023 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:07.195917    4930 scope.go:122] "RemoveContainer" containerID="b48dd3fa9ac8bd51d0dbbd7b816ad18851e1327f5bce04434886c42fdcbcd265"
Nov 01 03:15:07.196125 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:15:07.196088    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891715-g442j_odh-ai-gateway-infra(2cc1a309-9843-4ce7-ad17-d6c72ade9425)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891715-g442j" podUID="2cc1a309-9843-4ce7-ad17-d6c72ade9425"
Nov 01 03:15:08.202455 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:08.202427    4930 scope.go:122] "RemoveContainer" containerID="b48dd3fa9ac8bd51d0dbbd7b816ad18851e1327f5bce04434886c42fdcbcd265"
Nov 01 03:15:08.202835 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:15:08.202636    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891715-g442j_odh-ai-gateway-infra(2cc1a309-9843-4ce7-ad17-d6c72ade9425)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891715-g442j" podUID="2cc1a309-9843-4ce7-ad17-d6c72ade9425"
Nov 01 03:15:14.503607 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:14.503574    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:15:14.716575 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:14.716517    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:15:19.120133 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:15:19.120094    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:15:20.613600 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:20.613550    4930 scope.go:122] "RemoveContainer" containerID="b48dd3fa9ac8bd51d0dbbd7b816ad18851e1327f5bce04434886c42fdcbcd265"
Nov 01 03:15:21.266713 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:21.266631    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891715-g442j" event={"ID":"2cc1a309-9843-4ce7-ad17-d6c72ade9425","Type":"ContainerStarted","Data":"5e8181cbf4d47cf30b8f41642b03e19ca00ca26c665478d3cabad8ef69d31bbd"}
Nov 01 03:15:21.383054 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:21.383005    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891715-g442j" podStartSLOduration=16.842053383 podStartE2EDuration="21.382991131s" totalImagesPullingTime="4.540937748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:15:21.381928541 +0000 UTC m=+525.904935132" watchObservedRunningTime="2026-11-01 03:15:21.382991131 +0000 UTC m=+525.905997719"
Nov 01 03:15:22.273649 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:22.273616    4930 generic.go:356] "Generic (PLEG): container finished" podID="2cc1a309-9843-4ce7-ad17-d6c72ade9425" containerID="5e8181cbf4d47cf30b8f41642b03e19ca00ca26c665478d3cabad8ef69d31bbd" exitCode=60
Nov 01 03:15:22.274130 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:22.273687    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891715-g442j" event={"ID":"2cc1a309-9843-4ce7-ad17-d6c72ade9425","Type":"ContainerDied","Data":"5e8181cbf4d47cf30b8f41642b03e19ca00ca26c665478d3cabad8ef69d31bbd"}
Nov 01 03:15:22.274130 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:22.273761    4930 scope.go:122] "RemoveContainer" containerID="b48dd3fa9ac8bd51d0dbbd7b816ad18851e1327f5bce04434886c42fdcbcd265"
Nov 01 03:15:22.274130 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:22.274002    4930 scope.go:122] "RemoveContainer" containerID="5e8181cbf4d47cf30b8f41642b03e19ca00ca26c665478d3cabad8ef69d31bbd"
Nov 01 03:15:22.274259 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:15:22.274225    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891715-g442j_odh-ai-gateway-infra(2cc1a309-9843-4ce7-ad17-d6c72ade9425)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891715-g442j" podUID="2cc1a309-9843-4ce7-ad17-d6c72ade9425"
Nov 01 03:15:22.403982 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:22.403946    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891715-g442j"]
Nov 01 03:15:23.409393 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:23.409368    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891715-g442j"
Nov 01 03:15:23.493124 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:23.493098    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cc1a309-9843-4ce7-ad17-d6c72ade9425-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cc1a309-9843-4ce7-ad17-d6c72ade9425-maas-api-service-ca\") pod \"2cc1a309-9843-4ce7-ad17-d6c72ade9425\" (UID: \"2cc1a309-9843-4ce7-ad17-d6c72ade9425\") "
Nov 01 03:15:23.493283 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:23.493190    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cc1a309-9843-4ce7-ad17-d6c72ade9425-kube-api-access-6v4nl\" (UniqueName: \"kubernetes.io/projected/2cc1a309-9843-4ce7-ad17-d6c72ade9425-kube-api-access-6v4nl\") pod \"2cc1a309-9843-4ce7-ad17-d6c72ade9425\" (UID: \"2cc1a309-9843-4ce7-ad17-d6c72ade9425\") "
Nov 01 03:15:23.493440 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:23.493410    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cc1a309-9843-4ce7-ad17-d6c72ade9425-maas-api-service-ca" pod "2cc1a309-9843-4ce7-ad17-d6c72ade9425" (UID: "2cc1a309-9843-4ce7-ad17-d6c72ade9425"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:15:23.495384 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:23.495363    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cc1a309-9843-4ce7-ad17-d6c72ade9425-kube-api-access-6v4nl" pod "2cc1a309-9843-4ce7-ad17-d6c72ade9425" (UID: "2cc1a309-9843-4ce7-ad17-d6c72ade9425"). InnerVolumeSpecName "kube-api-access-6v4nl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:15:23.594537 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:23.594507    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v4nl\" (UniqueName: \"kubernetes.io/projected/2cc1a309-9843-4ce7-ad17-d6c72ade9425-kube-api-access-6v4nl\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:15:23.594537 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:23.594534    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cc1a309-9843-4ce7-ad17-d6c72ade9425-maas-api-service-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:15:24.286312 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:24.286278    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891715-g442j" event={"ID":"2cc1a309-9843-4ce7-ad17-d6c72ade9425","Type":"ContainerDied","Data":"c2c1e2f0e5de68f67bd2700dd3d6416912dfdda4c41619b1d23f3bdd459549b0"}
Nov 01 03:15:24.286312 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:24.286298    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891715-g442j"
Nov 01 03:15:24.286601 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:24.286334    4930 scope.go:122] "RemoveContainer" containerID="5e8181cbf4d47cf30b8f41642b03e19ca00ca26c665478d3cabad8ef69d31bbd"
Nov 01 03:15:24.451551 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:24.451519    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891715-g442j"]
Nov 01 03:15:24.456061 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:24.456038    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891715-g442j"]
Nov 01 03:15:25.619568 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:25.619533    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cc1a309-9843-4ce7-ad17-d6c72ade9425" path="/var/lib/kubelet/pods/2cc1a309-9843-4ce7-ad17-d6c72ade9425/volumes"
Nov 01 03:15:28.469666 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:28.469629    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:15:37.207952 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:37.207915    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp"]
Nov 01 03:15:37.208500 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:37.208262    4930 cpu_manager.go:414] "containerMap: removing container" podUID="2cc1a309-9843-4ce7-ad17-d6c72ade9425" containerName="cleanup"
Nov 01 03:15:37.208500 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:37.208274    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cc1a309-9843-4ce7-ad17-d6c72ade9425" containerName="cleanup"
Nov 01 03:15:37.208500 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:37.208292    4930 cpu_manager.go:414] "containerMap: removing container" podUID="2cc1a309-9843-4ce7-ad17-d6c72ade9425" containerName="cleanup"
Nov 01 03:15:37.208500 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:37.208297    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cc1a309-9843-4ce7-ad17-d6c72ade9425" containerName="cleanup"
Nov 01 03:15:37.208500 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:37.208303    4930 cpu_manager.go:414] "containerMap: removing container" podUID="2cc1a309-9843-4ce7-ad17-d6c72ade9425" containerName="cleanup"
Nov 01 03:15:37.208500 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:37.208308    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cc1a309-9843-4ce7-ad17-d6c72ade9425" containerName="cleanup"
Nov 01 03:15:37.208500 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:37.208384    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cc1a309-9843-4ce7-ad17-d6c72ade9425" containerName="cleanup"
Nov 01 03:15:37.208500 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:37.208394    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cc1a309-9843-4ce7-ad17-d6c72ade9425" containerName="cleanup"
Nov 01 03:15:37.208500 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:37.208400    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cc1a309-9843-4ce7-ad17-d6c72ade9425" containerName="cleanup"
Nov 01 03:15:37.212700 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:37.212684    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp"
Nov 01 03:15:37.267116 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:37.267075    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Nov 01 03:15:37.287083 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:37.287050    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 01 03:15:37.287083 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:37.287054    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 01 03:15:37.296568 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:37.296523    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:15:37.299568 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:37.299540    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/667240b0-64a7-45cd-84fe-9cd7e0d8a301-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp\" (UID: \"667240b0-64a7-45cd-84fe-9cd7e0d8a301\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp"
Nov 01 03:15:37.299643 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:37.299581    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/667240b0-64a7-45cd-84fe-9cd7e0d8a301-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp\" (UID: \"667240b0-64a7-45cd-84fe-9cd7e0d8a301\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp"
Nov 01 03:15:37.299680 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:37.299656    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/667240b0-64a7-45cd-84fe-9cd7e0d8a301-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp\" (UID: \"667240b0-64a7-45cd-84fe-9cd7e0d8a301\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp"
Nov 01 03:15:37.299729 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:37.299712    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/667240b0-64a7-45cd-84fe-9cd7e0d8a301-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp\" (UID: \"667240b0-64a7-45cd-84fe-9cd7e0d8a301\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp"
Nov 01 03:15:37.299786 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:37.299773    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95c8v\" (UniqueName: \"kubernetes.io/projected/667240b0-64a7-45cd-84fe-9cd7e0d8a301-kube-api-access-95c8v\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp\" (UID: \"667240b0-64a7-45cd-84fe-9cd7e0d8a301\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp"
Nov 01 03:15:37.299825 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:37.299800    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/667240b0-64a7-45cd-84fe-9cd7e0d8a301-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp\" (UID: \"667240b0-64a7-45cd-84fe-9cd7e0d8a301\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp"
Nov 01 03:15:37.316667 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:37.316650    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5tqmw\""
Nov 01 03:15:37.318545 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:37.318527    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp"]
Nov 01 03:15:37.400653 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:37.400619    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/667240b0-64a7-45cd-84fe-9cd7e0d8a301-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp\" (UID: \"667240b0-64a7-45cd-84fe-9cd7e0d8a301\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp"
Nov 01 03:15:37.400653 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:37.400656    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/667240b0-64a7-45cd-84fe-9cd7e0d8a301-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp\" (UID: \"667240b0-64a7-45cd-84fe-9cd7e0d8a301\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp"
Nov 01 03:15:37.400891 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:37.400700    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/667240b0-64a7-45cd-84fe-9cd7e0d8a301-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp\" (UID: \"667240b0-64a7-45cd-84fe-9cd7e0d8a301\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp"
Nov 01 03:15:37.400891 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:37.400716    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/667240b0-64a7-45cd-84fe-9cd7e0d8a301-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp\" (UID: \"667240b0-64a7-45cd-84fe-9cd7e0d8a301\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp"
Nov 01 03:15:37.400891 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:37.400761    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95c8v\" (UniqueName: \"kubernetes.io/projected/667240b0-64a7-45cd-84fe-9cd7e0d8a301-kube-api-access-95c8v\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp\" (UID: \"667240b0-64a7-45cd-84fe-9cd7e0d8a301\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp"
Nov 01 03:15:37.400891 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:37.400791    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/667240b0-64a7-45cd-84fe-9cd7e0d8a301-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp\" (UID: \"667240b0-64a7-45cd-84fe-9cd7e0d8a301\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp"
Nov 01 03:15:37.401144 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:37.401119    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/667240b0-64a7-45cd-84fe-9cd7e0d8a301-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp\" (UID: \"667240b0-64a7-45cd-84fe-9cd7e0d8a301\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp"
Nov 01 03:15:37.401198 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:37.401150    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/667240b0-64a7-45cd-84fe-9cd7e0d8a301-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp\" (UID: \"667240b0-64a7-45cd-84fe-9cd7e0d8a301\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp"
Nov 01 03:15:37.401237 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:37.401198    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/667240b0-64a7-45cd-84fe-9cd7e0d8a301-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp\" (UID: \"667240b0-64a7-45cd-84fe-9cd7e0d8a301\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp"
Nov 01 03:15:37.402857 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:37.402834    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/667240b0-64a7-45cd-84fe-9cd7e0d8a301-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp\" (UID: \"667240b0-64a7-45cd-84fe-9cd7e0d8a301\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp"
Nov 01 03:15:37.403272 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:37.403251    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/667240b0-64a7-45cd-84fe-9cd7e0d8a301-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp\" (UID: \"667240b0-64a7-45cd-84fe-9cd7e0d8a301\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp"
Nov 01 03:15:37.465455 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:37.465389    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95c8v\" (UniqueName: \"kubernetes.io/projected/667240b0-64a7-45cd-84fe-9cd7e0d8a301-kube-api-access-95c8v\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp\" (UID: \"667240b0-64a7-45cd-84fe-9cd7e0d8a301\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp"
Nov 01 03:15:37.539105 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:37.539083    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp"
Nov 01 03:15:37.912273 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:37.912249    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp"]
Nov 01 03:15:37.914193 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:15:37.914169    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod667240b0_64a7_45cd_84fe_9cd7e0d8a301.slice/crio-857cf2f00c6232c7368540107b1a25109291f4b20d4c1829097363325ba0bb00 WatchSource:0}: Error finding container 857cf2f00c6232c7368540107b1a25109291f4b20d4c1829097363325ba0bb00: Status 404 returned error can't find the container with id 857cf2f00c6232c7368540107b1a25109291f4b20d4c1829097363325ba0bb00
Nov 01 03:15:38.361147 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:38.361110    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp" event={"ID":"667240b0-64a7-45cd-84fe-9cd7e0d8a301","Type":"ContainerStarted","Data":"857cf2f00c6232c7368540107b1a25109291f4b20d4c1829097363325ba0bb00"}
Nov 01 03:15:38.501345 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:38.501307    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:15:40.375305 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:40.375268    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp" event={"ID":"667240b0-64a7-45cd-84fe-9cd7e0d8a301","Type":"ContainerStarted","Data":"23c50e856c66e43f49a88855a95377f386486411bfa0f304c6b95a5d9446a7c6"}
Nov 01 03:15:40.495140 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:40.495088    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp" podStartSLOduration=1.153991394 podStartE2EDuration="3.495073186s" totalImagesPullingTime="2.341081792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:15:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:15:40.493146106 +0000 UTC m=+545.016152694" watchObservedRunningTime="2026-11-01 03:15:40.495073186 +0000 UTC m=+545.018079774"
Nov 01 03:15:40.539786 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:40.539730    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp"
Nov 01 03:15:41.770131 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:41.770094    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:15:42.606786 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:42.606746    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:15:47.539890 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:47.539854    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp"
Nov 01 03:15:47.552465 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:47.552435    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp"
Nov 01 03:15:48.426032 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:48.426005    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp"
Nov 01 03:15:49.120461 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:15:49.120429    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:15:49.315753 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:49.315722    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:15:53.730526 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:53.730491    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-64vp4"]
Nov 01 03:15:53.733902 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:53.733886    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-64vp4"
Nov 01 03:15:53.822952 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:53.822922    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:15:53.836046 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:53.836022    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Nov 01 03:15:53.838011 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:53.837990    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-64vp4"]
Nov 01 03:15:53.842821 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:53.842800    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16d41e60-8b45-447c-aec0-798c5712ece0-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-64vp4\" (UID: \"16d41e60-8b45-447c-aec0-798c5712ece0\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-64vp4"
Nov 01 03:15:53.842912 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:53.842835    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16d41e60-8b45-447c-aec0-798c5712ece0-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-64vp4\" (UID: \"16d41e60-8b45-447c-aec0-798c5712ece0\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-64vp4"
Nov 01 03:15:53.842985 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:53.842968    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16d41e60-8b45-447c-aec0-798c5712ece0-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-64vp4\" (UID: \"16d41e60-8b45-447c-aec0-798c5712ece0\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-64vp4"
Nov 01 03:15:53.843044 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:53.843014    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16d41e60-8b45-447c-aec0-798c5712ece0-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-64vp4\" (UID: \"16d41e60-8b45-447c-aec0-798c5712ece0\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-64vp4"
Nov 01 03:15:53.843101 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:53.843088    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16d41e60-8b45-447c-aec0-798c5712ece0-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-64vp4\" (UID: \"16d41e60-8b45-447c-aec0-798c5712ece0\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-64vp4"
Nov 01 03:15:53.843184 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:53.843120    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtcbw\" (UniqueName: \"kubernetes.io/projected/16d41e60-8b45-447c-aec0-798c5712ece0-kube-api-access-xtcbw\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-64vp4\" (UID: \"16d41e60-8b45-447c-aec0-798c5712ece0\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-64vp4"
Nov 01 03:15:53.943717 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:53.943687    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16d41e60-8b45-447c-aec0-798c5712ece0-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-64vp4\" (UID: \"16d41e60-8b45-447c-aec0-798c5712ece0\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-64vp4"
Nov 01 03:15:53.943869 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:53.943725    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16d41e60-8b45-447c-aec0-798c5712ece0-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-64vp4\" (UID: \"16d41e60-8b45-447c-aec0-798c5712ece0\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-64vp4"
Nov 01 03:15:53.943869 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:53.943747    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16d41e60-8b45-447c-aec0-798c5712ece0-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-64vp4\" (UID: \"16d41e60-8b45-447c-aec0-798c5712ece0\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-64vp4"
Nov 01 03:15:53.943869 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:53.943765    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtcbw\" (UniqueName: \"kubernetes.io/projected/16d41e60-8b45-447c-aec0-798c5712ece0-kube-api-access-xtcbw\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-64vp4\" (UID: \"16d41e60-8b45-447c-aec0-798c5712ece0\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-64vp4"
Nov 01 03:15:53.943869 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:53.943814    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16d41e60-8b45-447c-aec0-798c5712ece0-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-64vp4\" (UID: \"16d41e60-8b45-447c-aec0-798c5712ece0\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-64vp4"
Nov 01 03:15:53.943869 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:53.943831    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16d41e60-8b45-447c-aec0-798c5712ece0-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-64vp4\" (UID: \"16d41e60-8b45-447c-aec0-798c5712ece0\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-64vp4"
Nov 01 03:15:53.944195 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:53.944170    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16d41e60-8b45-447c-aec0-798c5712ece0-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-64vp4\" (UID: \"16d41e60-8b45-447c-aec0-798c5712ece0\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-64vp4"
Nov 01 03:15:53.944257 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:53.944204    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16d41e60-8b45-447c-aec0-798c5712ece0-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-64vp4\" (UID: \"16d41e60-8b45-447c-aec0-798c5712ece0\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-64vp4"
Nov 01 03:15:53.944257 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:53.944244    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16d41e60-8b45-447c-aec0-798c5712ece0-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-64vp4\" (UID: \"16d41e60-8b45-447c-aec0-798c5712ece0\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-64vp4"
Nov 01 03:15:53.946017 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:53.945993    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16d41e60-8b45-447c-aec0-798c5712ece0-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-64vp4\" (UID: \"16d41e60-8b45-447c-aec0-798c5712ece0\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-64vp4"
Nov 01 03:15:53.946189 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:53.946171    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16d41e60-8b45-447c-aec0-798c5712ece0-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-64vp4\" (UID: \"16d41e60-8b45-447c-aec0-798c5712ece0\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-64vp4"
Nov 01 03:15:54.005596 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:54.005539    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtcbw\" (UniqueName: \"kubernetes.io/projected/16d41e60-8b45-447c-aec0-798c5712ece0-kube-api-access-xtcbw\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-64vp4\" (UID: \"16d41e60-8b45-447c-aec0-798c5712ece0\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-64vp4"
Nov 01 03:15:54.046242 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:54.046220    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-64vp4"
Nov 01 03:15:54.419131 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:54.419097    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-64vp4"]
Nov 01 03:15:54.420158 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:15:54.420123    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16d41e60_8b45_447c_aec0_798c5712ece0.slice/crio-a96d886d3f12e142cfc03160da0adcb8ad315dff76de0fccb9c3460823f62975 WatchSource:0}: Error finding container a96d886d3f12e142cfc03160da0adcb8ad315dff76de0fccb9c3460823f62975: Status 404 returned error can't find the container with id a96d886d3f12e142cfc03160da0adcb8ad315dff76de0fccb9c3460823f62975
Nov 01 03:15:54.447786 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:54.447763    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-64vp4" event={"ID":"16d41e60-8b45-447c-aec0-798c5712ece0","Type":"ContainerStarted","Data":"a96d886d3f12e142cfc03160da0adcb8ad315dff76de0fccb9c3460823f62975"}
Nov 01 03:15:55.454357 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:55.454273    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-64vp4" event={"ID":"16d41e60-8b45-447c-aec0-798c5712ece0","Type":"ContainerStarted","Data":"71e4aa5e3f4c0b6c4df4722bf632e549768cbc7c5b74aef510d3428a6722d4fa"}
Nov 01 03:15:55.572473 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:55.572423    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-64vp4" podStartSLOduration=1.8164609440000001 podStartE2EDuration="2.57241004s" totalImagesPullingTime="755.949096ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:15:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:15:55.570994315 +0000 UTC m=+560.094000909" watchObservedRunningTime="2026-11-01 03:15:55.57241004 +0000 UTC m=+560.095416627"
Nov 01 03:15:56.046637 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:56.046609    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-64vp4"
Nov 01 03:15:57.918530 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:57.918489    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-5rwkr"]
Nov 01 03:15:57.922759 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:57.922734    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5rwkr"
Nov 01 03:15:57.976594 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:57.976546    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53a7d5be-9872-49c5-9f21-f1e2f55d444a-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5rwkr\" (UID: \"53a7d5be-9872-49c5-9f21-f1e2f55d444a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5rwkr"
Nov 01 03:15:57.976703 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:57.976644    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53a7d5be-9872-49c5-9f21-f1e2f55d444a-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5rwkr\" (UID: \"53a7d5be-9872-49c5-9f21-f1e2f55d444a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5rwkr"
Nov 01 03:15:57.976703 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:57.976672    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htgmc\" (UniqueName: \"kubernetes.io/projected/53a7d5be-9872-49c5-9f21-f1e2f55d444a-kube-api-access-htgmc\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5rwkr\" (UID: \"53a7d5be-9872-49c5-9f21-f1e2f55d444a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5rwkr"
Nov 01 03:15:57.976816 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:57.976713    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53a7d5be-9872-49c5-9f21-f1e2f55d444a-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5rwkr\" (UID: \"53a7d5be-9872-49c5-9f21-f1e2f55d444a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5rwkr"
Nov 01 03:15:57.976909 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:57.976888    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53a7d5be-9872-49c5-9f21-f1e2f55d444a-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5rwkr\" (UID: \"53a7d5be-9872-49c5-9f21-f1e2f55d444a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5rwkr"
Nov 01 03:15:57.976949 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:57.976926    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53a7d5be-9872-49c5-9f21-f1e2f55d444a-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5rwkr\" (UID: \"53a7d5be-9872-49c5-9f21-f1e2f55d444a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5rwkr"
Nov 01 03:15:58.016002 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:58.015974    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:15:58.024319 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:58.024294    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Nov 01 03:15:58.027692 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:58.027673    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-5rwkr"]
Nov 01 03:15:58.078109 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:58.078082    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53a7d5be-9872-49c5-9f21-f1e2f55d444a-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5rwkr\" (UID: \"53a7d5be-9872-49c5-9f21-f1e2f55d444a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5rwkr"
Nov 01 03:15:58.078109 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:58.078110    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htgmc\" (UniqueName: \"kubernetes.io/projected/53a7d5be-9872-49c5-9f21-f1e2f55d444a-kube-api-access-htgmc\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5rwkr\" (UID: \"53a7d5be-9872-49c5-9f21-f1e2f55d444a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5rwkr"
Nov 01 03:15:58.078304 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:58.078148    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53a7d5be-9872-49c5-9f21-f1e2f55d444a-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5rwkr\" (UID: \"53a7d5be-9872-49c5-9f21-f1e2f55d444a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5rwkr"
Nov 01 03:15:58.078304 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:58.078183    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53a7d5be-9872-49c5-9f21-f1e2f55d444a-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5rwkr\" (UID: \"53a7d5be-9872-49c5-9f21-f1e2f55d444a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5rwkr"
Nov 01 03:15:58.078304 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:58.078197    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53a7d5be-9872-49c5-9f21-f1e2f55d444a-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5rwkr\" (UID: \"53a7d5be-9872-49c5-9f21-f1e2f55d444a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5rwkr"
Nov 01 03:15:58.078304 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:58.078235    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53a7d5be-9872-49c5-9f21-f1e2f55d444a-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5rwkr\" (UID: \"53a7d5be-9872-49c5-9f21-f1e2f55d444a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5rwkr"
Nov 01 03:15:58.078721 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:58.078692    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53a7d5be-9872-49c5-9f21-f1e2f55d444a-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5rwkr\" (UID: \"53a7d5be-9872-49c5-9f21-f1e2f55d444a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5rwkr"
Nov 01 03:15:58.078833 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:58.078703    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53a7d5be-9872-49c5-9f21-f1e2f55d444a-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5rwkr\" (UID: \"53a7d5be-9872-49c5-9f21-f1e2f55d444a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5rwkr"
Nov 01 03:15:58.078833 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:58.078819    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53a7d5be-9872-49c5-9f21-f1e2f55d444a-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5rwkr\" (UID: \"53a7d5be-9872-49c5-9f21-f1e2f55d444a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5rwkr"
Nov 01 03:15:58.080465 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:58.080437    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53a7d5be-9872-49c5-9f21-f1e2f55d444a-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5rwkr\" (UID: \"53a7d5be-9872-49c5-9f21-f1e2f55d444a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5rwkr"
Nov 01 03:15:58.080665 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:58.080647    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53a7d5be-9872-49c5-9f21-f1e2f55d444a-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5rwkr\" (UID: \"53a7d5be-9872-49c5-9f21-f1e2f55d444a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5rwkr"
Nov 01 03:15:58.139860 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:58.139834    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htgmc\" (UniqueName: \"kubernetes.io/projected/53a7d5be-9872-49c5-9f21-f1e2f55d444a-kube-api-access-htgmc\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-5rwkr\" (UID: \"53a7d5be-9872-49c5-9f21-f1e2f55d444a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5rwkr"
Nov 01 03:15:58.235934 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:58.235868    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5rwkr"
Nov 01 03:15:58.620971 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:58.620928    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-5rwkr"]
Nov 01 03:15:58.622944 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:15:58.622905    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53a7d5be_9872_49c5_9f21_f1e2f55d444a.slice/crio-bf46e43445ddfb4872a6e38c29611282484c73b35ef016c6ba390636e31d6ea0 WatchSource:0}: Error finding container bf46e43445ddfb4872a6e38c29611282484c73b35ef016c6ba390636e31d6ea0: Status 404 returned error can't find the container with id bf46e43445ddfb4872a6e38c29611282484c73b35ef016c6ba390636e31d6ea0
Nov 01 03:15:59.476080 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:59.476047    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5rwkr" event={"ID":"53a7d5be-9872-49c5-9f21-f1e2f55d444a","Type":"ContainerStarted","Data":"74e9f5b0216cd9edc5106281f1d89ec53d047be5b563d1b1f9eebbc4dedefd64"}
Nov 01 03:15:59.476411 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:59.476100    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5rwkr" event={"ID":"53a7d5be-9872-49c5-9f21-f1e2f55d444a","Type":"ContainerStarted","Data":"bf46e43445ddfb4872a6e38c29611282484c73b35ef016c6ba390636e31d6ea0"}
Nov 01 03:15:59.593692 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:15:59.593642    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5rwkr" podStartSLOduration=1.8549876699999999 podStartE2EDuration="2.593627363s" totalImagesPullingTime="738.639693ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:15:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:15:59.592892846 +0000 UTC m=+564.115899513" watchObservedRunningTime="2026-11-01 03:15:59.593627363 +0000 UTC m=+564.116633951"
Nov 01 03:16:00.236306 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:00.236269    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5rwkr"
Nov 01 03:16:01.545664 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:01.545627    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:16:04.047375 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:04.047341    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-64vp4"
Nov 01 03:16:04.059829 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:04.059803    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-64vp4"
Nov 01 03:16:04.517643 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:04.517609    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-64vp4"
Nov 01 03:16:05.186217 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:05.186183    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:16:07.647404 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:07.647367    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:16:07.843866 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:07.843837    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:16:08.235996 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:08.235965    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5rwkr"
Nov 01 03:16:08.248678 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:08.248651    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5rwkr"
Nov 01 03:16:08.533858 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:08.533775    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5rwkr"
Nov 01 03:16:09.129161 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:09.129122    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-qhchk"]
Nov 01 03:16:09.135418 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:09.135385    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-qhchk"
Nov 01 03:16:09.169853 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:09.169822    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98c81f0f-a540-4758-a3fa-e46f639e89d6-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-qhchk\" (UID: \"98c81f0f-a540-4758-a3fa-e46f639e89d6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qhchk"
Nov 01 03:16:09.170013 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:09.169870    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98c81f0f-a540-4758-a3fa-e46f639e89d6-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-qhchk\" (UID: \"98c81f0f-a540-4758-a3fa-e46f639e89d6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qhchk"
Nov 01 03:16:09.170013 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:09.169956    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98c81f0f-a540-4758-a3fa-e46f639e89d6-home\") pod \"echo-simulator-kserve-64b6f8f96c-qhchk\" (UID: \"98c81f0f-a540-4758-a3fa-e46f639e89d6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qhchk"
Nov 01 03:16:09.170139 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:09.170011    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbh8q\" (UniqueName: \"kubernetes.io/projected/98c81f0f-a540-4758-a3fa-e46f639e89d6-kube-api-access-xbh8q\") pod \"echo-simulator-kserve-64b6f8f96c-qhchk\" (UID: \"98c81f0f-a540-4758-a3fa-e46f639e89d6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qhchk"
Nov 01 03:16:09.170139 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:09.170045    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98c81f0f-a540-4758-a3fa-e46f639e89d6-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-qhchk\" (UID: \"98c81f0f-a540-4758-a3fa-e46f639e89d6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qhchk"
Nov 01 03:16:09.170240 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:09.170149    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98c81f0f-a540-4758-a3fa-e46f639e89d6-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-qhchk\" (UID: \"98c81f0f-a540-4758-a3fa-e46f639e89d6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qhchk"
Nov 01 03:16:09.197388 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:09.197322    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Nov 01 03:16:09.224537 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:09.224506    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:16:09.248836 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:09.248801    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-qhchk"]
Nov 01 03:16:09.271068 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:09.271041    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98c81f0f-a540-4758-a3fa-e46f639e89d6-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-qhchk\" (UID: \"98c81f0f-a540-4758-a3fa-e46f639e89d6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qhchk"
Nov 01 03:16:09.271235 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:09.271080    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98c81f0f-a540-4758-a3fa-e46f639e89d6-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-qhchk\" (UID: \"98c81f0f-a540-4758-a3fa-e46f639e89d6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qhchk"
Nov 01 03:16:09.271235 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:09.271112    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98c81f0f-a540-4758-a3fa-e46f639e89d6-home\") pod \"echo-simulator-kserve-64b6f8f96c-qhchk\" (UID: \"98c81f0f-a540-4758-a3fa-e46f639e89d6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qhchk"
Nov 01 03:16:09.271235 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:09.271148    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbh8q\" (UniqueName: \"kubernetes.io/projected/98c81f0f-a540-4758-a3fa-e46f639e89d6-kube-api-access-xbh8q\") pod \"echo-simulator-kserve-64b6f8f96c-qhchk\" (UID: \"98c81f0f-a540-4758-a3fa-e46f639e89d6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qhchk"
Nov 01 03:16:09.271235 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:09.271173    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98c81f0f-a540-4758-a3fa-e46f639e89d6-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-qhchk\" (UID: \"98c81f0f-a540-4758-a3fa-e46f639e89d6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qhchk"
Nov 01 03:16:09.271235 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:09.271210    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98c81f0f-a540-4758-a3fa-e46f639e89d6-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-qhchk\" (UID: \"98c81f0f-a540-4758-a3fa-e46f639e89d6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qhchk"
Nov 01 03:16:09.271509 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:09.271495    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98c81f0f-a540-4758-a3fa-e46f639e89d6-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-qhchk\" (UID: \"98c81f0f-a540-4758-a3fa-e46f639e89d6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qhchk"
Nov 01 03:16:09.271594 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:09.271507    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98c81f0f-a540-4758-a3fa-e46f639e89d6-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-qhchk\" (UID: \"98c81f0f-a540-4758-a3fa-e46f639e89d6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qhchk"
Nov 01 03:16:09.271813 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:09.271788    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98c81f0f-a540-4758-a3fa-e46f639e89d6-home\") pod \"echo-simulator-kserve-64b6f8f96c-qhchk\" (UID: \"98c81f0f-a540-4758-a3fa-e46f639e89d6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qhchk"
Nov 01 03:16:09.273762 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:09.273740    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98c81f0f-a540-4758-a3fa-e46f639e89d6-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-qhchk\" (UID: \"98c81f0f-a540-4758-a3fa-e46f639e89d6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qhchk"
Nov 01 03:16:09.274261 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:09.274240    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98c81f0f-a540-4758-a3fa-e46f639e89d6-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-qhchk\" (UID: \"98c81f0f-a540-4758-a3fa-e46f639e89d6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qhchk"
Nov 01 03:16:09.332878 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:09.332851    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbh8q\" (UniqueName: \"kubernetes.io/projected/98c81f0f-a540-4758-a3fa-e46f639e89d6-kube-api-access-xbh8q\") pod \"echo-simulator-kserve-64b6f8f96c-qhchk\" (UID: \"98c81f0f-a540-4758-a3fa-e46f639e89d6\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qhchk"
Nov 01 03:16:09.449227 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:09.449143    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-qhchk"
Nov 01 03:16:09.837958 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:09.837931    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-qhchk"]
Nov 01 03:16:09.839264 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:16:09.839236    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98c81f0f_a540_4758_a3fa_e46f639e89d6.slice/crio-2fa398613f001cc2fed3ed0bc38ff21b2c622ac1b556a57d08f93949fe965b15 WatchSource:0}: Error finding container 2fa398613f001cc2fed3ed0bc38ff21b2c622ac1b556a57d08f93949fe965b15: Status 404 returned error can't find the container with id 2fa398613f001cc2fed3ed0bc38ff21b2c622ac1b556a57d08f93949fe965b15
Nov 01 03:16:10.535328 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:10.535295    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-qhchk" event={"ID":"98c81f0f-a540-4758-a3fa-e46f639e89d6","Type":"ContainerStarted","Data":"2fa398613f001cc2fed3ed0bc38ff21b2c622ac1b556a57d08f93949fe965b15"}
Nov 01 03:16:13.551719 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:13.551678    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-qhchk" event={"ID":"98c81f0f-a540-4758-a3fa-e46f639e89d6","Type":"ContainerStarted","Data":"b1ddd6a01851bc6b373ac636e9a1cacfa476dbf90d905f95bc82da8fcbf4b680"}
Nov 01 03:16:13.670949 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:13.670892    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-qhchk" podStartSLOduration=1.866302989 podStartE2EDuration="4.670877329s" totalImagesPullingTime="2.80457434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:16:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:16:13.669180386 +0000 UTC m=+578.192186985" watchObservedRunningTime="2026-11-01 03:16:13.670877329 +0000 UTC m=+578.193883917"
Nov 01 03:16:14.450253 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:14.450219    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-qhchk"
Nov 01 03:16:15.778751 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:15.778717    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n"]
Nov 01 03:16:15.783034 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:15.783011    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n"
Nov 01 03:16:15.826689 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:15.826662    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35b89d8d-ab16-41ea-8a99-bcc80e212838-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-4sm7n\" (UID: \"35b89d8d-ab16-41ea-8a99-bcc80e212838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n"
Nov 01 03:16:15.826835 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:15.826693    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35b89d8d-ab16-41ea-8a99-bcc80e212838-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-4sm7n\" (UID: \"35b89d8d-ab16-41ea-8a99-bcc80e212838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n"
Nov 01 03:16:15.826835 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:15.826714    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35b89d8d-ab16-41ea-8a99-bcc80e212838-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-4sm7n\" (UID: \"35b89d8d-ab16-41ea-8a99-bcc80e212838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n"
Nov 01 03:16:15.826835 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:15.826789    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35b89d8d-ab16-41ea-8a99-bcc80e212838-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-4sm7n\" (UID: \"35b89d8d-ab16-41ea-8a99-bcc80e212838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n"
Nov 01 03:16:15.826991 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:15.826878    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35b89d8d-ab16-41ea-8a99-bcc80e212838-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-4sm7n\" (UID: \"35b89d8d-ab16-41ea-8a99-bcc80e212838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n"
Nov 01 03:16:15.826991 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:15.826962    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35b89d8d-ab16-41ea-8a99-bcc80e212838-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-4sm7n\" (UID: \"35b89d8d-ab16-41ea-8a99-bcc80e212838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n"
Nov 01 03:16:15.827075 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:15.826992    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm6sz\" (UniqueName: \"kubernetes.io/projected/35b89d8d-ab16-41ea-8a99-bcc80e212838-kube-api-access-jm6sz\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-4sm7n\" (UID: \"35b89d8d-ab16-41ea-8a99-bcc80e212838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n"
Nov 01 03:16:15.827075 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:15.827010    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35b89d8d-ab16-41ea-8a99-bcc80e212838-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-4sm7n\" (UID: \"35b89d8d-ab16-41ea-8a99-bcc80e212838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n"
Nov 01 03:16:15.827151 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:15.827134    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35b89d8d-ab16-41ea-8a99-bcc80e212838-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-4sm7n\" (UID: \"35b89d8d-ab16-41ea-8a99-bcc80e212838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n"
Nov 01 03:16:15.875244 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:15.875199    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n"]
Nov 01 03:16:15.927842 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:15.927809    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35b89d8d-ab16-41ea-8a99-bcc80e212838-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-4sm7n\" (UID: \"35b89d8d-ab16-41ea-8a99-bcc80e212838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n"
Nov 01 03:16:15.928026 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:15.927850    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35b89d8d-ab16-41ea-8a99-bcc80e212838-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-4sm7n\" (UID: \"35b89d8d-ab16-41ea-8a99-bcc80e212838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n"
Nov 01 03:16:15.928026 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:15.927868    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35b89d8d-ab16-41ea-8a99-bcc80e212838-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-4sm7n\" (UID: \"35b89d8d-ab16-41ea-8a99-bcc80e212838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n"
Nov 01 03:16:15.928026 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:15.927892    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35b89d8d-ab16-41ea-8a99-bcc80e212838-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-4sm7n\" (UID: \"35b89d8d-ab16-41ea-8a99-bcc80e212838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n"
Nov 01 03:16:15.928026 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:15.927929    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35b89d8d-ab16-41ea-8a99-bcc80e212838-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-4sm7n\" (UID: \"35b89d8d-ab16-41ea-8a99-bcc80e212838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n"
Nov 01 03:16:15.928026 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:15.927972    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35b89d8d-ab16-41ea-8a99-bcc80e212838-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-4sm7n\" (UID: \"35b89d8d-ab16-41ea-8a99-bcc80e212838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n"
Nov 01 03:16:15.928026 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:15.928022    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35b89d8d-ab16-41ea-8a99-bcc80e212838-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-4sm7n\" (UID: \"35b89d8d-ab16-41ea-8a99-bcc80e212838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n"
Nov 01 03:16:15.928348 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:15.928053    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm6sz\" (UniqueName: \"kubernetes.io/projected/35b89d8d-ab16-41ea-8a99-bcc80e212838-kube-api-access-jm6sz\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-4sm7n\" (UID: \"35b89d8d-ab16-41ea-8a99-bcc80e212838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n"
Nov 01 03:16:15.928348 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:15.928069    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35b89d8d-ab16-41ea-8a99-bcc80e212838-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-4sm7n\" (UID: \"35b89d8d-ab16-41ea-8a99-bcc80e212838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n"
Nov 01 03:16:15.928452 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:15.928346    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35b89d8d-ab16-41ea-8a99-bcc80e212838-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-4sm7n\" (UID: \"35b89d8d-ab16-41ea-8a99-bcc80e212838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n"
Nov 01 03:16:15.928452 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:15.928422    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35b89d8d-ab16-41ea-8a99-bcc80e212838-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-4sm7n\" (UID: \"35b89d8d-ab16-41ea-8a99-bcc80e212838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n"
Nov 01 03:16:15.928639 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:15.928615    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35b89d8d-ab16-41ea-8a99-bcc80e212838-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-4sm7n\" (UID: \"35b89d8d-ab16-41ea-8a99-bcc80e212838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n"
Nov 01 03:16:15.928895 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:15.928650    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35b89d8d-ab16-41ea-8a99-bcc80e212838-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-4sm7n\" (UID: \"35b89d8d-ab16-41ea-8a99-bcc80e212838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n"
Nov 01 03:16:15.929028 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:15.929012    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35b89d8d-ab16-41ea-8a99-bcc80e212838-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-4sm7n\" (UID: \"35b89d8d-ab16-41ea-8a99-bcc80e212838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n"
Nov 01 03:16:15.930285 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:15.930266    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35b89d8d-ab16-41ea-8a99-bcc80e212838-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-4sm7n\" (UID: \"35b89d8d-ab16-41ea-8a99-bcc80e212838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n"
Nov 01 03:16:15.930570 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:15.930535    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35b89d8d-ab16-41ea-8a99-bcc80e212838-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-4sm7n\" (UID: \"35b89d8d-ab16-41ea-8a99-bcc80e212838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n"
Nov 01 03:16:15.988918 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:15.988881    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35b89d8d-ab16-41ea-8a99-bcc80e212838-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-4sm7n\" (UID: \"35b89d8d-ab16-41ea-8a99-bcc80e212838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n"
Nov 01 03:16:15.989103 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:15.989082    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm6sz\" (UniqueName: \"kubernetes.io/projected/35b89d8d-ab16-41ea-8a99-bcc80e212838-kube-api-access-jm6sz\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-4sm7n\" (UID: \"35b89d8d-ab16-41ea-8a99-bcc80e212838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n"
Nov 01 03:16:16.096375 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:16.096341    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n"
Nov 01 03:16:16.534119 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:16.534086    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n"]
Nov 01 03:16:16.535900 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:16:16.535876    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35b89d8d_ab16_41ea_8a99_bcc80e212838.slice/crio-f18cf66c59f9a44cfbdacedd06ed893a8dbc949eb4fa7b0153377e59b936f876 WatchSource:0}: Error finding container f18cf66c59f9a44cfbdacedd06ed893a8dbc949eb4fa7b0153377e59b936f876: Status 404 returned error can't find the container with id f18cf66c59f9a44cfbdacedd06ed893a8dbc949eb4fa7b0153377e59b936f876
Nov 01 03:16:16.538002 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:16.537970    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:16:16.538083 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:16.538029    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:16:16.538083 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:16.538060    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:16:16.579842 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:16.579814    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n" event={"ID":"35b89d8d-ab16-41ea-8a99-bcc80e212838","Type":"ContainerStarted","Data":"f18cf66c59f9a44cfbdacedd06ed893a8dbc949eb4fa7b0153377e59b936f876"}
Nov 01 03:16:17.589276 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:17.589238    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n" event={"ID":"35b89d8d-ab16-41ea-8a99-bcc80e212838","Type":"ContainerStarted","Data":"1f5d54f9f7fbe2e03833c6dddedd330b6aba8f0cee43ba61d529a0683894b586"}
Nov 01 03:16:17.813944 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:17.813889    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n" podStartSLOduration=2.813873128 podStartE2EDuration="2.813873128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:16:17.763058154 +0000 UTC m=+582.286064742" watchObservedRunningTime="2026-11-01 03:16:17.813873128 +0000 UTC m=+582.336879791"
Nov 01 03:16:18.097172 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:18.097139    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n"
Nov 01 03:16:18.098497 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:18.098476    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 01 03:16:18.098570 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:18.098530    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n" podUID="35b89d8d-ab16-41ea-8a99-bcc80e212838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 01 03:16:19.097781 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:19.097743    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 01 03:16:19.098265 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:19.097825    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n" podUID="35b89d8d-ab16-41ea-8a99-bcc80e212838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 01 03:16:19.120258 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:16:19.120222    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:16:19.449572 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:19.449463    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-qhchk"
Nov 01 03:16:19.471857 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:19.471827    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-qhchk"
Nov 01 03:16:19.620076 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:19.620046    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-qhchk"
Nov 01 03:16:20.097704 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:20.097672    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 01 03:16:20.097855 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:20.097724    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n" podUID="35b89d8d-ab16-41ea-8a99-bcc80e212838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 01 03:16:21.097203 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:21.097166    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 01 03:16:21.097381 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:21.097224    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n" podUID="35b89d8d-ab16-41ea-8a99-bcc80e212838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 01 03:16:22.097491 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:22.097455    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 01 03:16:22.097895 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:22.097509    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n" podUID="35b89d8d-ab16-41ea-8a99-bcc80e212838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 01 03:16:23.096863 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:23.096832    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 01 03:16:23.097028 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:23.096880    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n" podUID="35b89d8d-ab16-41ea-8a99-bcc80e212838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 01 03:16:24.809152 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:24.809121    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n"
Nov 01 03:16:24.809665 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:24.809488    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n"
Nov 01 03:16:24.810252 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:24.810230    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n"
Nov 01 03:16:28.805040 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:28.804956    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:16:30.895764 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:30.895727    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7"]
Nov 01 03:16:30.899509 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:30.899486    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7"
Nov 01 03:16:30.982006 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:30.981968    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e325fcbf-d504-4923-b0fd-809005447f3e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-jmwc7\" (UID: \"e325fcbf-d504-4923-b0fd-809005447f3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7"
Nov 01 03:16:30.982170 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:30.982034    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e325fcbf-d504-4923-b0fd-809005447f3e-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-jmwc7\" (UID: \"e325fcbf-d504-4923-b0fd-809005447f3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7"
Nov 01 03:16:30.982170 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:30.982118    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e325fcbf-d504-4923-b0fd-809005447f3e-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-jmwc7\" (UID: \"e325fcbf-d504-4923-b0fd-809005447f3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7"
Nov 01 03:16:30.982299 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:30.982167    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e325fcbf-d504-4923-b0fd-809005447f3e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-jmwc7\" (UID: \"e325fcbf-d504-4923-b0fd-809005447f3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7"
Nov 01 03:16:30.982299 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:30.982273    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e325fcbf-d504-4923-b0fd-809005447f3e-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-jmwc7\" (UID: \"e325fcbf-d504-4923-b0fd-809005447f3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7"
Nov 01 03:16:30.982412 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:30.982317    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djtps\" (UniqueName: \"kubernetes.io/projected/e325fcbf-d504-4923-b0fd-809005447f3e-kube-api-access-djtps\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-jmwc7\" (UID: \"e325fcbf-d504-4923-b0fd-809005447f3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7"
Nov 01 03:16:30.982469 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:30.982428    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e325fcbf-d504-4923-b0fd-809005447f3e-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-jmwc7\" (UID: \"e325fcbf-d504-4923-b0fd-809005447f3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7"
Nov 01 03:16:30.982534 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:30.982466    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e325fcbf-d504-4923-b0fd-809005447f3e-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-jmwc7\" (UID: \"e325fcbf-d504-4923-b0fd-809005447f3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7"
Nov 01 03:16:30.982623 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:30.982580    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e325fcbf-d504-4923-b0fd-809005447f3e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-jmwc7\" (UID: \"e325fcbf-d504-4923-b0fd-809005447f3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7"
Nov 01 03:16:31.083686 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:31.083654    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e325fcbf-d504-4923-b0fd-809005447f3e-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-jmwc7\" (UID: \"e325fcbf-d504-4923-b0fd-809005447f3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7"
Nov 01 03:16:31.083841 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:31.083694    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djtps\" (UniqueName: \"kubernetes.io/projected/e325fcbf-d504-4923-b0fd-809005447f3e-kube-api-access-djtps\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-jmwc7\" (UID: \"e325fcbf-d504-4923-b0fd-809005447f3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7"
Nov 01 03:16:31.083841 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:31.083723    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e325fcbf-d504-4923-b0fd-809005447f3e-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-jmwc7\" (UID: \"e325fcbf-d504-4923-b0fd-809005447f3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7"
Nov 01 03:16:31.083841 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:31.083754    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e325fcbf-d504-4923-b0fd-809005447f3e-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-jmwc7\" (UID: \"e325fcbf-d504-4923-b0fd-809005447f3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7"
Nov 01 03:16:31.083841 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:31.083790    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e325fcbf-d504-4923-b0fd-809005447f3e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-jmwc7\" (UID: \"e325fcbf-d504-4923-b0fd-809005447f3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7"
Nov 01 03:16:31.083841 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:31.083835    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e325fcbf-d504-4923-b0fd-809005447f3e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-jmwc7\" (UID: \"e325fcbf-d504-4923-b0fd-809005447f3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7"
Nov 01 03:16:31.084124 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:31.083895    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e325fcbf-d504-4923-b0fd-809005447f3e-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-jmwc7\" (UID: \"e325fcbf-d504-4923-b0fd-809005447f3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7"
Nov 01 03:16:31.084124 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:31.083927    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e325fcbf-d504-4923-b0fd-809005447f3e-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-jmwc7\" (UID: \"e325fcbf-d504-4923-b0fd-809005447f3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7"
Nov 01 03:16:31.084124 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:31.083973    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e325fcbf-d504-4923-b0fd-809005447f3e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-jmwc7\" (UID: \"e325fcbf-d504-4923-b0fd-809005447f3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7"
Nov 01 03:16:31.084320 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:31.084292    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e325fcbf-d504-4923-b0fd-809005447f3e-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-jmwc7\" (UID: \"e325fcbf-d504-4923-b0fd-809005447f3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7"
Nov 01 03:16:31.084446 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:31.084329    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e325fcbf-d504-4923-b0fd-809005447f3e-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-jmwc7\" (UID: \"e325fcbf-d504-4923-b0fd-809005447f3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7"
Nov 01 03:16:31.084505 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:31.084371    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e325fcbf-d504-4923-b0fd-809005447f3e-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-jmwc7\" (UID: \"e325fcbf-d504-4923-b0fd-809005447f3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7"
Nov 01 03:16:31.084686 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:31.084662    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e325fcbf-d504-4923-b0fd-809005447f3e-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-jmwc7\" (UID: \"e325fcbf-d504-4923-b0fd-809005447f3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7"
Nov 01 03:16:31.084962 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:31.084945    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e325fcbf-d504-4923-b0fd-809005447f3e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-jmwc7\" (UID: \"e325fcbf-d504-4923-b0fd-809005447f3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7"
Nov 01 03:16:31.086207 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:31.086189    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e325fcbf-d504-4923-b0fd-809005447f3e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-jmwc7\" (UID: \"e325fcbf-d504-4923-b0fd-809005447f3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7"
Nov 01 03:16:31.086416 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:31.086393    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e325fcbf-d504-4923-b0fd-809005447f3e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-jmwc7\" (UID: \"e325fcbf-d504-4923-b0fd-809005447f3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7"
Nov 01 03:16:31.113423 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:31.113396    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7"]
Nov 01 03:16:31.151955 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:31.151898    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djtps\" (UniqueName: \"kubernetes.io/projected/e325fcbf-d504-4923-b0fd-809005447f3e-kube-api-access-djtps\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-jmwc7\" (UID: \"e325fcbf-d504-4923-b0fd-809005447f3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7"
Nov 01 03:16:31.158742 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:31.158713    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e325fcbf-d504-4923-b0fd-809005447f3e-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-jmwc7\" (UID: \"e325fcbf-d504-4923-b0fd-809005447f3e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7"
Nov 01 03:16:31.212252 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:31.212226    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7"
Nov 01 03:16:31.638322 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:31.638299    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7"]
Nov 01 03:16:31.639849 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:16:31.639821    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode325fcbf_d504_4923_b0fd_809005447f3e.slice/crio-8e413152a817d3ada4a843bd3d358795dae38c02fd99191119538cde21d25c6f WatchSource:0}: Error finding container 8e413152a817d3ada4a843bd3d358795dae38c02fd99191119538cde21d25c6f: Status 404 returned error can't find the container with id 8e413152a817d3ada4a843bd3d358795dae38c02fd99191119538cde21d25c6f
Nov 01 03:16:31.641868 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:31.641835    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:16:31.641954 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:31.641898    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:16:31.641954 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:31.641937    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:16:31.661825 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:31.661797    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7" event={"ID":"e325fcbf-d504-4923-b0fd-809005447f3e","Type":"ContainerStarted","Data":"8e413152a817d3ada4a843bd3d358795dae38c02fd99191119538cde21d25c6f"}
Nov 01 03:16:32.688931 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:32.688894    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7" event={"ID":"e325fcbf-d504-4923-b0fd-809005447f3e","Type":"ContainerStarted","Data":"8fc9c28599b028d464ecb875484b3d5d8c6db7cb616e7c96ba2fd6d8e019dd75"}
Nov 01 03:16:32.884426 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:32.884357    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7" podStartSLOduration=2.884336904 podStartE2EDuration="2.884336904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:16:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:16:32.881877444 +0000 UTC m=+597.404884032" watchObservedRunningTime="2026-11-01 03:16:32.884336904 +0000 UTC m=+597.407343494"
Nov 01 03:16:33.212790 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:33.212754    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7"
Nov 01 03:16:33.214157 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:33.214138    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 01 03:16:33.214211 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:33.214183    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7" podUID="e325fcbf-d504-4923-b0fd-809005447f3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 01 03:16:33.842130 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:33.842097    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8549fcc7fb-s5trb"]
Nov 01 03:16:33.845912 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:33.845893    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8549fcc7fb-s5trb"
Nov 01 03:16:33.947822 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:33.947793    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-47cjs\""
Nov 01 03:16:33.947822 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:33.947795    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 01 03:16:33.950218 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:33.950198    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8549fcc7fb-s5trb"]
Nov 01 03:16:33.985763 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:33.984107    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:16:34.011851 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:34.011825    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpd9l\" (UniqueName: \"kubernetes.io/projected/66e26411-cab9-4f6a-8814-53165e790a45-kube-api-access-rpd9l\") pod \"authorino-8549fcc7fb-s5trb\" (UID: \"66e26411-cab9-4f6a-8814-53165e790a45\") " pod="kuadrant-system/authorino-8549fcc7fb-s5trb"
Nov 01 03:16:34.012013 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:34.011883    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/66e26411-cab9-4f6a-8814-53165e790a45-tls-cert\") pod \"authorino-8549fcc7fb-s5trb\" (UID: \"66e26411-cab9-4f6a-8814-53165e790a45\") " pod="kuadrant-system/authorino-8549fcc7fb-s5trb"
Nov 01 03:16:34.112963 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:34.112874    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/66e26411-cab9-4f6a-8814-53165e790a45-tls-cert\") pod \"authorino-8549fcc7fb-s5trb\" (UID: \"66e26411-cab9-4f6a-8814-53165e790a45\") " pod="kuadrant-system/authorino-8549fcc7fb-s5trb"
Nov 01 03:16:34.113112 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:34.112971    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpd9l\" (UniqueName: \"kubernetes.io/projected/66e26411-cab9-4f6a-8814-53165e790a45-kube-api-access-rpd9l\") pod \"authorino-8549fcc7fb-s5trb\" (UID: \"66e26411-cab9-4f6a-8814-53165e790a45\") " pod="kuadrant-system/authorino-8549fcc7fb-s5trb"
Nov 01 03:16:34.115384 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:34.115354    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/66e26411-cab9-4f6a-8814-53165e790a45-tls-cert\") pod \"authorino-8549fcc7fb-s5trb\" (UID: \"66e26411-cab9-4f6a-8814-53165e790a45\") " pod="kuadrant-system/authorino-8549fcc7fb-s5trb"
Nov 01 03:16:34.177988 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:34.177957    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpd9l\" (UniqueName: \"kubernetes.io/projected/66e26411-cab9-4f6a-8814-53165e790a45-kube-api-access-rpd9l\") pod \"authorino-8549fcc7fb-s5trb\" (UID: \"66e26411-cab9-4f6a-8814-53165e790a45\") " pod="kuadrant-system/authorino-8549fcc7fb-s5trb"
Nov 01 03:16:34.212841 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:34.212811    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 01 03:16:34.212995 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:34.212858    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7" podUID="e325fcbf-d504-4923-b0fd-809005447f3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 01 03:16:34.224894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:34.224869    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:16:34.457901 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:34.457810    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8549fcc7fb-s5trb"
Nov 01 03:16:34.852299 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:34.852271    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8549fcc7fb-s5trb"]
Nov 01 03:16:34.854390 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:16:34.854350    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66e26411_cab9_4f6a_8814_53165e790a45.slice/crio-2a115757606a6b66c89ee163ae03bc62dc5b93a6a0c15918f2b7bc762fb884b4 WatchSource:0}: Error finding container 2a115757606a6b66c89ee163ae03bc62dc5b93a6a0c15918f2b7bc762fb884b4: Status 404 returned error can't find the container with id 2a115757606a6b66c89ee163ae03bc62dc5b93a6a0c15918f2b7bc762fb884b4
Nov 01 03:16:35.212769 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:35.212682    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 01 03:16:35.212769 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:35.212749    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7" podUID="e325fcbf-d504-4923-b0fd-809005447f3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 01 03:16:35.700486 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:35.700451    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jh955_f29ea61e-2a78-4c11-9566-f52497e135ff/console-operator/2.log"
Nov 01 03:16:35.703022 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:35.702999    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jh955_f29ea61e-2a78-4c11-9566-f52497e135ff/console-operator/2.log"
Nov 01 03:16:35.707931 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:35.707901    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8549fcc7fb-s5trb" event={"ID":"66e26411-cab9-4f6a-8814-53165e790a45","Type":"ContainerStarted","Data":"2a115757606a6b66c89ee163ae03bc62dc5b93a6a0c15918f2b7bc762fb884b4"}
Nov 01 03:16:36.213345 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:36.213311    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 01 03:16:36.213723 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:36.213363    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7" podUID="e325fcbf-d504-4923-b0fd-809005447f3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 01 03:16:36.714813 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:36.714776    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8549fcc7fb-s5trb" event={"ID":"66e26411-cab9-4f6a-8814-53165e790a45","Type":"ContainerStarted","Data":"911d3b0e837387348f978b626f2386a844b360d77172333572c9a7d61291e93c"}
Nov 01 03:16:36.849302 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:36.849253    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-8549fcc7fb-s5trb" podStartSLOduration=2.298367866 podStartE2EDuration="3.849240358s" totalImagesPullingTime="1.550872492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:16:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:16:36.848861058 +0000 UTC m=+601.371867646" watchObservedRunningTime="2026-11-01 03:16:36.849240358 +0000 UTC m=+601.372246943"
Nov 01 03:16:37.213253 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:37.213171    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 01 03:16:37.213253 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:37.213226    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7" podUID="e325fcbf-d504-4923-b0fd-809005447f3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 01 03:16:38.213415 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:38.213377    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Nov 01 03:16:38.213948 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:38.213480    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7" podUID="e325fcbf-d504-4923-b0fd-809005447f3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Nov 01 03:16:38.906815 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:38.906766    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:16:39.944406 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:39.944364    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7"
Nov 01 03:16:39.945015 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:39.944992    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7"
Nov 01 03:16:39.945715 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:39.945695    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7"
Nov 01 03:16:43.460758 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:43.460724    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:16:43.633722 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:43.633691    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:16:49.121004 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:16:49.120959    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:16:53.782420 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:53.782380    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:16:53.975602 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:16:53.975571    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:07.222258 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:07.222222    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:07.719698 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:07.719653    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:09.283847 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:09.283814    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:09.436446 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:09.436413    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:10.806069 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:10.806033    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:10.970458 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:10.970423    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:13.900306 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:13.900272    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:14.045476 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:14.045445    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:16.249453 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:16.249420    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh"]
Nov 01 03:17:16.254140 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:16.254123    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh"
Nov 01 03:17:16.301906 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:16.301878    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f3268e-176f-46ef-8431-0b3e985caa0f-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xm5sh\" (UID: \"97f3268e-176f-46ef-8431-0b3e985caa0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh"
Nov 01 03:17:16.302014 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:16.301914    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97f3268e-176f-46ef-8431-0b3e985caa0f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xm5sh\" (UID: \"97f3268e-176f-46ef-8431-0b3e985caa0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh"
Nov 01 03:17:16.302014 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:16.301935    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgx5g\" (UniqueName: \"kubernetes.io/projected/97f3268e-176f-46ef-8431-0b3e985caa0f-kube-api-access-zgx5g\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xm5sh\" (UID: \"97f3268e-176f-46ef-8431-0b3e985caa0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh"
Nov 01 03:17:16.302086 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:16.302061    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97f3268e-176f-46ef-8431-0b3e985caa0f-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xm5sh\" (UID: \"97f3268e-176f-46ef-8431-0b3e985caa0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh"
Nov 01 03:17:16.302122 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:16.302085    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97f3268e-176f-46ef-8431-0b3e985caa0f-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xm5sh\" (UID: \"97f3268e-176f-46ef-8431-0b3e985caa0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh"
Nov 01 03:17:16.302122 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:16.302108    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f3268e-176f-46ef-8431-0b3e985caa0f-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xm5sh\" (UID: \"97f3268e-176f-46ef-8431-0b3e985caa0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh"
Nov 01 03:17:16.302189 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:16.302151    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97f3268e-176f-46ef-8431-0b3e985caa0f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xm5sh\" (UID: \"97f3268e-176f-46ef-8431-0b3e985caa0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh"
Nov 01 03:17:16.302222 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:16.302206    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97f3268e-176f-46ef-8431-0b3e985caa0f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xm5sh\" (UID: \"97f3268e-176f-46ef-8431-0b3e985caa0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh"
Nov 01 03:17:16.302256 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:16.302232    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f3268e-176f-46ef-8431-0b3e985caa0f-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xm5sh\" (UID: \"97f3268e-176f-46ef-8431-0b3e985caa0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh"
Nov 01 03:17:16.403352 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:16.403320    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97f3268e-176f-46ef-8431-0b3e985caa0f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xm5sh\" (UID: \"97f3268e-176f-46ef-8431-0b3e985caa0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh"
Nov 01 03:17:16.403485 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:16.403360    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f3268e-176f-46ef-8431-0b3e985caa0f-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xm5sh\" (UID: \"97f3268e-176f-46ef-8431-0b3e985caa0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh"
Nov 01 03:17:16.403485 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:16.403446    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f3268e-176f-46ef-8431-0b3e985caa0f-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xm5sh\" (UID: \"97f3268e-176f-46ef-8431-0b3e985caa0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh"
Nov 01 03:17:16.403485 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:16.403477    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97f3268e-176f-46ef-8431-0b3e985caa0f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xm5sh\" (UID: \"97f3268e-176f-46ef-8431-0b3e985caa0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh"
Nov 01 03:17:16.403667 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:16.403509    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgx5g\" (UniqueName: \"kubernetes.io/projected/97f3268e-176f-46ef-8431-0b3e985caa0f-kube-api-access-zgx5g\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xm5sh\" (UID: \"97f3268e-176f-46ef-8431-0b3e985caa0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh"
Nov 01 03:17:16.403667 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:16.403578    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97f3268e-176f-46ef-8431-0b3e985caa0f-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xm5sh\" (UID: \"97f3268e-176f-46ef-8431-0b3e985caa0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh"
Nov 01 03:17:16.403667 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:16.403607    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97f3268e-176f-46ef-8431-0b3e985caa0f-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xm5sh\" (UID: \"97f3268e-176f-46ef-8431-0b3e985caa0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh"
Nov 01 03:17:16.403667 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:16.403634    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f3268e-176f-46ef-8431-0b3e985caa0f-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xm5sh\" (UID: \"97f3268e-176f-46ef-8431-0b3e985caa0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh"
Nov 01 03:17:16.403889 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:16.403671    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97f3268e-176f-46ef-8431-0b3e985caa0f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xm5sh\" (UID: \"97f3268e-176f-46ef-8431-0b3e985caa0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh"
Nov 01 03:17:16.403988 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:16.403948    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f3268e-176f-46ef-8431-0b3e985caa0f-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xm5sh\" (UID: \"97f3268e-176f-46ef-8431-0b3e985caa0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh"
Nov 01 03:17:16.404046 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:16.404001    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97f3268e-176f-46ef-8431-0b3e985caa0f-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xm5sh\" (UID: \"97f3268e-176f-46ef-8431-0b3e985caa0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh"
Nov 01 03:17:16.404099 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:16.404081    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f3268e-176f-46ef-8431-0b3e985caa0f-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xm5sh\" (UID: \"97f3268e-176f-46ef-8431-0b3e985caa0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh"
Nov 01 03:17:16.404189 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:16.404172    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f3268e-176f-46ef-8431-0b3e985caa0f-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xm5sh\" (UID: \"97f3268e-176f-46ef-8431-0b3e985caa0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh"
Nov 01 03:17:16.404431 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:16.404405    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97f3268e-176f-46ef-8431-0b3e985caa0f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xm5sh\" (UID: \"97f3268e-176f-46ef-8431-0b3e985caa0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh"
Nov 01 03:17:16.405799 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:16.405768    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97f3268e-176f-46ef-8431-0b3e985caa0f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xm5sh\" (UID: \"97f3268e-176f-46ef-8431-0b3e985caa0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh"
Nov 01 03:17:16.406134 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:16.406112    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97f3268e-176f-46ef-8431-0b3e985caa0f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xm5sh\" (UID: \"97f3268e-176f-46ef-8431-0b3e985caa0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh"
Nov 01 03:17:16.461980 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:16.461951    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh"]
Nov 01 03:17:16.473259 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:16.473235    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97f3268e-176f-46ef-8431-0b3e985caa0f-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xm5sh\" (UID: \"97f3268e-176f-46ef-8431-0b3e985caa0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh"
Nov 01 03:17:16.473599 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:16.473580    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgx5g\" (UniqueName: \"kubernetes.io/projected/97f3268e-176f-46ef-8431-0b3e985caa0f-kube-api-access-zgx5g\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xm5sh\" (UID: \"97f3268e-176f-46ef-8431-0b3e985caa0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh"
Nov 01 03:17:16.566724 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:16.566650    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh"
Nov 01 03:17:16.990736 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:16.990706    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh"]
Nov 01 03:17:16.992631 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:17:16.992605    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97f3268e_176f_46ef_8431_0b3e985caa0f.slice/crio-daa2bc383bb9dd49191ec3edbdae198f1f5d89b741f3827d76c4a30af6efb91e WatchSource:0}: Error finding container daa2bc383bb9dd49191ec3edbdae198f1f5d89b741f3827d76c4a30af6efb91e: Status 404 returned error can't find the container with id daa2bc383bb9dd49191ec3edbdae198f1f5d89b741f3827d76c4a30af6efb91e
Nov 01 03:17:16.994473 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:16.994440    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:17:16.994583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:16.994507    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:17:16.994583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:16.994540    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:17:17.930583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:17.930524    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh" event={"ID":"97f3268e-176f-46ef-8431-0b3e985caa0f","Type":"ContainerStarted","Data":"71075b263b0356ade2cb8a16ed245ad3bf547c27d53d4691287cb9f8ab038f30"}
Nov 01 03:17:17.931078 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:17.930603    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh" event={"ID":"97f3268e-176f-46ef-8431-0b3e985caa0f","Type":"ContainerStarted","Data":"daa2bc383bb9dd49191ec3edbdae198f1f5d89b741f3827d76c4a30af6efb91e"}
Nov 01 03:17:18.098474 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:18.098414    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh" podStartSLOduration=2.098400232 podStartE2EDuration="2.098400232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:17:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:17:18.09636283 +0000 UTC m=+642.619369420" watchObservedRunningTime="2026-11-01 03:17:18.098400232 +0000 UTC m=+642.621406820"
Nov 01 03:17:18.579581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:18.573825    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh"
Nov 01 03:17:18.583680 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:18.583653    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 01 03:17:18.583802 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:18.583711    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh" podUID="97f3268e-176f-46ef-8431-0b3e985caa0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 01 03:17:19.120139 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:17:19.120100    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:17:19.578633 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:19.577891    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 01 03:17:19.578633 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:19.577953    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh" podUID="97f3268e-176f-46ef-8431-0b3e985caa0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 01 03:17:20.577581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:20.571878    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 01 03:17:20.577581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:20.571955    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh" podUID="97f3268e-176f-46ef-8431-0b3e985caa0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 01 03:17:21.570593 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:21.569926    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 01 03:17:21.570593 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:21.569987    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh" podUID="97f3268e-176f-46ef-8431-0b3e985caa0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 01 03:17:22.574953 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:22.574914    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 01 03:17:22.575576 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:22.575523    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh" podUID="97f3268e-176f-46ef-8431-0b3e985caa0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 01 03:17:23.567372 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:23.567336    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Nov 01 03:17:23.567540 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:23.567389    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh" podUID="97f3268e-176f-46ef-8431-0b3e985caa0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Nov 01 03:17:25.242069 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:25.242033    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh"
Nov 01 03:17:25.242687 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:25.242653    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh"
Nov 01 03:17:25.243527 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:25.243505    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh"
Nov 01 03:17:29.151568 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:29.151517    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:35.449131 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:35.449093    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:35.651187 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:35.651151    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:36.097776 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.097736    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"]
Nov 01 03:17:36.106073 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.106043    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"
Nov 01 03:17:36.159982 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.159950    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-b12199-openshift-default-dockercfg-lc8zw\""
Nov 01 03:17:36.193587 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.193541    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:36.211898 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.211869    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"]
Nov 01 03:17:36.215249 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.215225    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f8e6d8-aafa-4a82-b1e4-63c358380207-workload-socket\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm\" (UID: \"58f8e6d8-aafa-4a82-b1e4-63c358380207\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"
Nov 01 03:17:36.215355 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.215275    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58f8e6d8-aafa-4a82-b1e4-63c358380207-istio-podinfo\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm\" (UID: \"58f8e6d8-aafa-4a82-b1e4-63c358380207\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"
Nov 01 03:17:36.215424 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.215398    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58f8e6d8-aafa-4a82-b1e4-63c358380207-istiod-ca-cert\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm\" (UID: \"58f8e6d8-aafa-4a82-b1e4-63c358380207\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"
Nov 01 03:17:36.215468 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.215441    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58f8e6d8-aafa-4a82-b1e4-63c358380207-istio-envoy\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm\" (UID: \"58f8e6d8-aafa-4a82-b1e4-63c358380207\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"
Nov 01 03:17:36.215590 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.215569    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f8e6d8-aafa-4a82-b1e4-63c358380207-credential-socket\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm\" (UID: \"58f8e6d8-aafa-4a82-b1e4-63c358380207\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"
Nov 01 03:17:36.215665 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.215644    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58f8e6d8-aafa-4a82-b1e4-63c358380207-workload-certs\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm\" (UID: \"58f8e6d8-aafa-4a82-b1e4-63c358380207\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"
Nov 01 03:17:36.215728 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.215691    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58f8e6d8-aafa-4a82-b1e4-63c358380207-istio-token\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm\" (UID: \"58f8e6d8-aafa-4a82-b1e4-63c358380207\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"
Nov 01 03:17:36.215803 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.215789    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bghzq\" (UniqueName: \"kubernetes.io/projected/58f8e6d8-aafa-4a82-b1e4-63c358380207-kube-api-access-bghzq\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm\" (UID: \"58f8e6d8-aafa-4a82-b1e4-63c358380207\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"
Nov 01 03:17:36.215869 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.215854    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58f8e6d8-aafa-4a82-b1e4-63c358380207-istio-data\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm\" (UID: \"58f8e6d8-aafa-4a82-b1e4-63c358380207\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"
Nov 01 03:17:36.316790 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.316755    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58f8e6d8-aafa-4a82-b1e4-63c358380207-istiod-ca-cert\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm\" (UID: \"58f8e6d8-aafa-4a82-b1e4-63c358380207\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"
Nov 01 03:17:36.316790 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.316793    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58f8e6d8-aafa-4a82-b1e4-63c358380207-istio-envoy\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm\" (UID: \"58f8e6d8-aafa-4a82-b1e4-63c358380207\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"
Nov 01 03:17:36.317051 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.316821    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f8e6d8-aafa-4a82-b1e4-63c358380207-credential-socket\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm\" (UID: \"58f8e6d8-aafa-4a82-b1e4-63c358380207\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"
Nov 01 03:17:36.317051 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.316939    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58f8e6d8-aafa-4a82-b1e4-63c358380207-workload-certs\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm\" (UID: \"58f8e6d8-aafa-4a82-b1e4-63c358380207\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"
Nov 01 03:17:36.317051 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.316988    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58f8e6d8-aafa-4a82-b1e4-63c358380207-istio-token\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm\" (UID: \"58f8e6d8-aafa-4a82-b1e4-63c358380207\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"
Nov 01 03:17:36.317051 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.317025    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bghzq\" (UniqueName: \"kubernetes.io/projected/58f8e6d8-aafa-4a82-b1e4-63c358380207-kube-api-access-bghzq\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm\" (UID: \"58f8e6d8-aafa-4a82-b1e4-63c358380207\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"
Nov 01 03:17:36.317277 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.317095    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58f8e6d8-aafa-4a82-b1e4-63c358380207-istio-data\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm\" (UID: \"58f8e6d8-aafa-4a82-b1e4-63c358380207\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"
Nov 01 03:17:36.317277 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.317159    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f8e6d8-aafa-4a82-b1e4-63c358380207-workload-socket\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm\" (UID: \"58f8e6d8-aafa-4a82-b1e4-63c358380207\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"
Nov 01 03:17:36.317277 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.317178    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f8e6d8-aafa-4a82-b1e4-63c358380207-credential-socket\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm\" (UID: \"58f8e6d8-aafa-4a82-b1e4-63c358380207\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"
Nov 01 03:17:36.317277 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.317199    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58f8e6d8-aafa-4a82-b1e4-63c358380207-istio-podinfo\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm\" (UID: \"58f8e6d8-aafa-4a82-b1e4-63c358380207\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"
Nov 01 03:17:36.317495 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.317347    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58f8e6d8-aafa-4a82-b1e4-63c358380207-workload-certs\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm\" (UID: \"58f8e6d8-aafa-4a82-b1e4-63c358380207\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"
Nov 01 03:17:36.317495 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.317452    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58f8e6d8-aafa-4a82-b1e4-63c358380207-istio-data\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm\" (UID: \"58f8e6d8-aafa-4a82-b1e4-63c358380207\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"
Nov 01 03:17:36.317627 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.317598    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f8e6d8-aafa-4a82-b1e4-63c358380207-workload-socket\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm\" (UID: \"58f8e6d8-aafa-4a82-b1e4-63c358380207\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"
Nov 01 03:17:36.317696 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.317673    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58f8e6d8-aafa-4a82-b1e4-63c358380207-istiod-ca-cert\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm\" (UID: \"58f8e6d8-aafa-4a82-b1e4-63c358380207\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"
Nov 01 03:17:36.319061 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.319043    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58f8e6d8-aafa-4a82-b1e4-63c358380207-istio-envoy\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm\" (UID: \"58f8e6d8-aafa-4a82-b1e4-63c358380207\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"
Nov 01 03:17:36.319470 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.319454    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58f8e6d8-aafa-4a82-b1e4-63c358380207-istio-podinfo\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm\" (UID: \"58f8e6d8-aafa-4a82-b1e4-63c358380207\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"
Nov 01 03:17:36.378568 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.378494    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58f8e6d8-aafa-4a82-b1e4-63c358380207-istio-token\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm\" (UID: \"58f8e6d8-aafa-4a82-b1e4-63c358380207\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"
Nov 01 03:17:36.378666 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.378621    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bghzq\" (UniqueName: \"kubernetes.io/projected/58f8e6d8-aafa-4a82-b1e4-63c358380207-kube-api-access-bghzq\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm\" (UID: \"58f8e6d8-aafa-4a82-b1e4-63c358380207\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"
Nov 01 03:17:36.403757 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.403731    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"]
Nov 01 03:17:36.408477 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.408462    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"
Nov 01 03:17:36.423529 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.423511    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"
Nov 01 03:17:36.441974 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.441898    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:36.479867 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.478304    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"]
Nov 01 03:17:36.483999 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.483968    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-19faa5-openshift-default-dockercfg-tk569\""
Nov 01 03:17:36.520030 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.520003    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-istio-envoy\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j\" (UID: \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"
Nov 01 03:17:36.520170 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.520037    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-istiod-ca-cert\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j\" (UID: \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"
Nov 01 03:17:36.520170 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.520059    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-istio-data\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j\" (UID: \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"
Nov 01 03:17:36.520170 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.520160    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-istio-token\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j\" (UID: \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"
Nov 01 03:17:36.520286 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.520270    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzsp4\" (UniqueName: \"kubernetes.io/projected/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-kube-api-access-mzsp4\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j\" (UID: \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"
Nov 01 03:17:36.520335 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.520304    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-workload-certs\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j\" (UID: \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"
Nov 01 03:17:36.520385 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.520333    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-credential-socket\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j\" (UID: \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"
Nov 01 03:17:36.520462 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.520442    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-workload-socket\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j\" (UID: \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"
Nov 01 03:17:36.520543 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.520527    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-istio-podinfo\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j\" (UID: \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"
Nov 01 03:17:36.594543 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.594499    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:36.621628 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.621596    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-istio-token\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j\" (UID: \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"
Nov 01 03:17:36.621793 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.621635    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzsp4\" (UniqueName: \"kubernetes.io/projected/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-kube-api-access-mzsp4\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j\" (UID: \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"
Nov 01 03:17:36.621793 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.621660    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-workload-certs\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j\" (UID: \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"
Nov 01 03:17:36.621793 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.621765    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-credential-socket\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j\" (UID: \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"
Nov 01 03:17:36.621912 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.621824    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-workload-socket\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j\" (UID: \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"
Nov 01 03:17:36.621912 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.621880    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-istio-podinfo\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j\" (UID: \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"
Nov 01 03:17:36.622003 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.621973    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-istio-envoy\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j\" (UID: \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"
Nov 01 03:17:36.622057 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.622000    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-istiod-ca-cert\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j\" (UID: \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"
Nov 01 03:17:36.622057 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.622028    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-istio-data\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j\" (UID: \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"
Nov 01 03:17:36.622158 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.622094    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-workload-certs\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j\" (UID: \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"
Nov 01 03:17:36.622218 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.622172    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-credential-socket\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j\" (UID: \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"
Nov 01 03:17:36.622271 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.622244    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-workload-socket\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j\" (UID: \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"
Nov 01 03:17:36.622511 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.622479    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-istio-data\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j\" (UID: \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"
Nov 01 03:17:36.622784 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.622763    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-istiod-ca-cert\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j\" (UID: \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"
Nov 01 03:17:36.624317 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.624293    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-istio-envoy\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j\" (UID: \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"
Nov 01 03:17:36.624621 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.624600    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-istio-podinfo\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j\" (UID: \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"
Nov 01 03:17:36.690162 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.690104    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-istio-token\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j\" (UID: \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"
Nov 01 03:17:36.690365 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.690348    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzsp4\" (UniqueName: \"kubernetes.io/projected/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-kube-api-access-mzsp4\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j\" (UID: \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"
Nov 01 03:17:36.722149 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.722117    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"
Nov 01 03:17:36.743598 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.743574    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"]
Nov 01 03:17:36.747237 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.747218    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"
Nov 01 03:17:36.802453 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.802425    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ded2d7-openshift-default-dockercfg-glqxh\""
Nov 01 03:17:36.814194 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.814125    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"]
Nov 01 03:17:36.837486 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.837448    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:36.857003 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.856980    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"]
Nov 01 03:17:36.857234 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:17:36.857213    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58f8e6d8_aafa_4a82_b1e4_63c358380207.slice/crio-0d9cd85c21544fdbe215eca8128d6f38c122be48785f9bd1d1b1615e87f5c705 WatchSource:0}: Error finding container 0d9cd85c21544fdbe215eca8128d6f38c122be48785f9bd1d1b1615e87f5c705: Status 404 returned error can't find the container with id 0d9cd85c21544fdbe215eca8128d6f38c122be48785f9bd1d1b1615e87f5c705
Nov 01 03:17:36.859684 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.859656    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:17:36.859751 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.859717    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:17:36.859787 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.859752    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:17:36.925220 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.925193    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a115577-6d0a-4047-ab8b-8eb78f51acbe-workload-certs\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r\" (UID: \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"
Nov 01 03:17:36.925361 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.925229    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a115577-6d0a-4047-ab8b-8eb78f51acbe-istiod-ca-cert\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r\" (UID: \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"
Nov 01 03:17:36.925361 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.925250    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a115577-6d0a-4047-ab8b-8eb78f51acbe-istio-data\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r\" (UID: \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"
Nov 01 03:17:36.925459 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.925362    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a115577-6d0a-4047-ab8b-8eb78f51acbe-workload-socket\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r\" (UID: \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"
Nov 01 03:17:36.925459 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.925392    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a115577-6d0a-4047-ab8b-8eb78f51acbe-istio-envoy\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r\" (UID: \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"
Nov 01 03:17:36.925633 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.925468    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a115577-6d0a-4047-ab8b-8eb78f51acbe-credential-socket\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r\" (UID: \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"
Nov 01 03:17:36.925633 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.925491    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a115577-6d0a-4047-ab8b-8eb78f51acbe-istio-token\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r\" (UID: \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"
Nov 01 03:17:36.925633 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.925618    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a115577-6d0a-4047-ab8b-8eb78f51acbe-istio-podinfo\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r\" (UID: \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"
Nov 01 03:17:36.925797 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.925669    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmmsz\" (UniqueName: \"kubernetes.io/projected/5a115577-6d0a-4047-ab8b-8eb78f51acbe-kube-api-access-jmmsz\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r\" (UID: \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"
Nov 01 03:17:36.981911 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:36.981790    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:37.026741 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.026702    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a115577-6d0a-4047-ab8b-8eb78f51acbe-credential-socket\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r\" (UID: \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"
Nov 01 03:17:37.026741 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.026742    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a115577-6d0a-4047-ab8b-8eb78f51acbe-istio-token\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r\" (UID: \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"
Nov 01 03:17:37.027000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.026775    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a115577-6d0a-4047-ab8b-8eb78f51acbe-istio-podinfo\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r\" (UID: \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"
Nov 01 03:17:37.027000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.026808    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmmsz\" (UniqueName: \"kubernetes.io/projected/5a115577-6d0a-4047-ab8b-8eb78f51acbe-kube-api-access-jmmsz\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r\" (UID: \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"
Nov 01 03:17:37.027000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.026856    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a115577-6d0a-4047-ab8b-8eb78f51acbe-workload-certs\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r\" (UID: \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"
Nov 01 03:17:37.027000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.026884    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a115577-6d0a-4047-ab8b-8eb78f51acbe-istiod-ca-cert\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r\" (UID: \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"
Nov 01 03:17:37.027000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.026907    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a115577-6d0a-4047-ab8b-8eb78f51acbe-istio-data\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r\" (UID: \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"
Nov 01 03:17:37.027000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.026947    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a115577-6d0a-4047-ab8b-8eb78f51acbe-workload-socket\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r\" (UID: \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"
Nov 01 03:17:37.027000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.026975    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a115577-6d0a-4047-ab8b-8eb78f51acbe-istio-envoy\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r\" (UID: \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"
Nov 01 03:17:37.027370 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.027127    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a115577-6d0a-4047-ab8b-8eb78f51acbe-credential-socket\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r\" (UID: \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"
Nov 01 03:17:37.027431 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.027396    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a115577-6d0a-4047-ab8b-8eb78f51acbe-workload-certs\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r\" (UID: \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"
Nov 01 03:17:37.029115 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.028199    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a115577-6d0a-4047-ab8b-8eb78f51acbe-istio-data\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r\" (UID: \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"
Nov 01 03:17:37.029115 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.028406    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a115577-6d0a-4047-ab8b-8eb78f51acbe-workload-socket\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r\" (UID: \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"
Nov 01 03:17:37.029115 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.029069    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a115577-6d0a-4047-ab8b-8eb78f51acbe-istiod-ca-cert\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r\" (UID: \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"
Nov 01 03:17:37.030719 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.030632    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a115577-6d0a-4047-ab8b-8eb78f51acbe-istio-envoy\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r\" (UID: \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"
Nov 01 03:17:37.030920 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.030900    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a115577-6d0a-4047-ab8b-8eb78f51acbe-istio-podinfo\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r\" (UID: \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"
Nov 01 03:17:37.034601 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.034542    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm" event={"ID":"58f8e6d8-aafa-4a82-b1e4-63c358380207","Type":"ContainerStarted","Data":"aafc35f35a3dde77e5b573be2be98eaa2018cd07ef430dbbfdcc84e2359e07a1"}
Nov 01 03:17:37.034703 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.034637    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm" event={"ID":"58f8e6d8-aafa-4a82-b1e4-63c358380207","Type":"ContainerStarted","Data":"0d9cd85c21544fdbe215eca8128d6f38c122be48785f9bd1d1b1615e87f5c705"}
Nov 01 03:17:37.106615 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.103819    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a115577-6d0a-4047-ab8b-8eb78f51acbe-istio-token\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r\" (UID: \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"
Nov 01 03:17:37.106615 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.104376    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmmsz\" (UniqueName: \"kubernetes.io/projected/5a115577-6d0a-4047-ab8b-8eb78f51acbe-kube-api-access-jmmsz\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r\" (UID: \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"
Nov 01 03:17:37.115213 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.113814    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"]
Nov 01 03:17:37.116268 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:17:37.116235    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3c4cdb7_e686_4b71_9ab9_3c82f4c24648.slice/crio-fe4aa40ed07a33a49e7ea585a8bed77550ef71e0afae2082b93c5a34dcbc451d WatchSource:0}: Error finding container fe4aa40ed07a33a49e7ea585a8bed77550ef71e0afae2082b93c5a34dcbc451d: Status 404 returned error can't find the container with id fe4aa40ed07a33a49e7ea585a8bed77550ef71e0afae2082b93c5a34dcbc451d
Nov 01 03:17:37.119301 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.119261    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:17:37.119405 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.119349    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:17:37.119405 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.119392    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:17:37.165416 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.165366    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm" podStartSLOduration=1.165350778 podStartE2EDuration="1.165350778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:17:37.163977322 +0000 UTC m=+661.686983908" watchObservedRunningTime="2026-11-01 03:17:37.165350778 +0000 UTC m=+661.688357366"
Nov 01 03:17:37.175539 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.175514    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:37.327748 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.327613    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:37.362806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.362780    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"
Nov 01 03:17:37.427165 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.427065    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"
Nov 01 03:17:37.453109 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.452893    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"
Nov 01 03:17:37.490455 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.490407    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:37.697699 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.697666    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:37.720812 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.720778    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"]
Nov 01 03:17:37.725299 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.725281    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"
Nov 01 03:17:37.753767 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.752700    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/267415f5-02bf-4571-914a-45aa36ae41c9-istio-token\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr\" (UID: \"267415f5-02bf-4571-914a-45aa36ae41c9\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"
Nov 01 03:17:37.753767 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.752743    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/267415f5-02bf-4571-914a-45aa36ae41c9-istio-envoy\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr\" (UID: \"267415f5-02bf-4571-914a-45aa36ae41c9\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"
Nov 01 03:17:37.753767 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.752776    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/267415f5-02bf-4571-914a-45aa36ae41c9-workload-socket\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr\" (UID: \"267415f5-02bf-4571-914a-45aa36ae41c9\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"
Nov 01 03:17:37.753767 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.752803    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/267415f5-02bf-4571-914a-45aa36ae41c9-credential-socket\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr\" (UID: \"267415f5-02bf-4571-914a-45aa36ae41c9\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"
Nov 01 03:17:37.753767 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.752855    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxg92\" (UniqueName: \"kubernetes.io/projected/267415f5-02bf-4571-914a-45aa36ae41c9-kube-api-access-wxg92\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr\" (UID: \"267415f5-02bf-4571-914a-45aa36ae41c9\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"
Nov 01 03:17:37.753767 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.752923    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/267415f5-02bf-4571-914a-45aa36ae41c9-workload-certs\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr\" (UID: \"267415f5-02bf-4571-914a-45aa36ae41c9\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"
Nov 01 03:17:37.753767 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.752978    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/267415f5-02bf-4571-914a-45aa36ae41c9-istio-data\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr\" (UID: \"267415f5-02bf-4571-914a-45aa36ae41c9\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"
Nov 01 03:17:37.753767 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.753011    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/267415f5-02bf-4571-914a-45aa36ae41c9-istio-podinfo\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr\" (UID: \"267415f5-02bf-4571-914a-45aa36ae41c9\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"
Nov 01 03:17:37.753767 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.753076    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/267415f5-02bf-4571-914a-45aa36ae41c9-istiod-ca-cert\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr\" (UID: \"267415f5-02bf-4571-914a-45aa36ae41c9\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"
Nov 01 03:17:37.759168 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.758372    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"]
Nov 01 03:17:37.764619 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:17:37.764590    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a115577_6d0a_4047_ab8b_8eb78f51acbe.slice/crio-b5814351a3c6aa46164405f9564bb6cde12ad22dc35004db84710fe560333bd6 WatchSource:0}: Error finding container b5814351a3c6aa46164405f9564bb6cde12ad22dc35004db84710fe560333bd6: Status 404 returned error can't find the container with id b5814351a3c6aa46164405f9564bb6cde12ad22dc35004db84710fe560333bd6
Nov 01 03:17:37.768780 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.768744    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:17:37.768924 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.768855    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:17:37.768924 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.768898    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:17:37.781601 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.781544    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-c8cb9a-openshift-default-dockercfg-r4cfc\""
Nov 01 03:17:37.802506 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.802481    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"]
Nov 01 03:17:37.841451 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.841422    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:37.854031 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.854010    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/267415f5-02bf-4571-914a-45aa36ae41c9-workload-certs\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr\" (UID: \"267415f5-02bf-4571-914a-45aa36ae41c9\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"
Nov 01 03:17:37.854151 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.854043    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/267415f5-02bf-4571-914a-45aa36ae41c9-istio-data\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr\" (UID: \"267415f5-02bf-4571-914a-45aa36ae41c9\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"
Nov 01 03:17:37.854151 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.854061    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/267415f5-02bf-4571-914a-45aa36ae41c9-istio-podinfo\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr\" (UID: \"267415f5-02bf-4571-914a-45aa36ae41c9\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"
Nov 01 03:17:37.854151 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.854106    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/267415f5-02bf-4571-914a-45aa36ae41c9-istiod-ca-cert\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr\" (UID: \"267415f5-02bf-4571-914a-45aa36ae41c9\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"
Nov 01 03:17:37.854322 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.854167    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/267415f5-02bf-4571-914a-45aa36ae41c9-istio-token\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr\" (UID: \"267415f5-02bf-4571-914a-45aa36ae41c9\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"
Nov 01 03:17:37.854322 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.854185    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/267415f5-02bf-4571-914a-45aa36ae41c9-istio-envoy\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr\" (UID: \"267415f5-02bf-4571-914a-45aa36ae41c9\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"
Nov 01 03:17:37.854322 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.854202    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/267415f5-02bf-4571-914a-45aa36ae41c9-workload-socket\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr\" (UID: \"267415f5-02bf-4571-914a-45aa36ae41c9\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"
Nov 01 03:17:37.854322 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.854228    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/267415f5-02bf-4571-914a-45aa36ae41c9-credential-socket\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr\" (UID: \"267415f5-02bf-4571-914a-45aa36ae41c9\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"
Nov 01 03:17:37.854322 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.854282    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxg92\" (UniqueName: \"kubernetes.io/projected/267415f5-02bf-4571-914a-45aa36ae41c9-kube-api-access-wxg92\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr\" (UID: \"267415f5-02bf-4571-914a-45aa36ae41c9\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"
Nov 01 03:17:37.854658 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.854458    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/267415f5-02bf-4571-914a-45aa36ae41c9-istio-data\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr\" (UID: \"267415f5-02bf-4571-914a-45aa36ae41c9\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"
Nov 01 03:17:37.854734 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.854716    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/267415f5-02bf-4571-914a-45aa36ae41c9-workload-socket\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr\" (UID: \"267415f5-02bf-4571-914a-45aa36ae41c9\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"
Nov 01 03:17:37.854949 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.854931    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/267415f5-02bf-4571-914a-45aa36ae41c9-workload-certs\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr\" (UID: \"267415f5-02bf-4571-914a-45aa36ae41c9\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"
Nov 01 03:17:37.854949 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.854940    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/267415f5-02bf-4571-914a-45aa36ae41c9-credential-socket\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr\" (UID: \"267415f5-02bf-4571-914a-45aa36ae41c9\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"
Nov 01 03:17:37.855251 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.855227    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/267415f5-02bf-4571-914a-45aa36ae41c9-istiod-ca-cert\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr\" (UID: \"267415f5-02bf-4571-914a-45aa36ae41c9\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"
Nov 01 03:17:37.856617 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.856598    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/267415f5-02bf-4571-914a-45aa36ae41c9-istio-podinfo\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr\" (UID: \"267415f5-02bf-4571-914a-45aa36ae41c9\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"
Nov 01 03:17:37.856685 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.856620    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/267415f5-02bf-4571-914a-45aa36ae41c9-istio-envoy\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr\" (UID: \"267415f5-02bf-4571-914a-45aa36ae41c9\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"
Nov 01 03:17:37.915486 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.915444    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/267415f5-02bf-4571-914a-45aa36ae41c9-istio-token\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr\" (UID: \"267415f5-02bf-4571-914a-45aa36ae41c9\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"
Nov 01 03:17:37.915685 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:37.915668    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxg92\" (UniqueName: \"kubernetes.io/projected/267415f5-02bf-4571-914a-45aa36ae41c9-kube-api-access-wxg92\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr\" (UID: \"267415f5-02bf-4571-914a-45aa36ae41c9\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"
Nov 01 03:17:38.038723 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.038551    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"
Nov 01 03:17:38.041611 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.041532    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r" event={"ID":"5a115577-6d0a-4047-ab8b-8eb78f51acbe","Type":"ContainerStarted","Data":"9094fc4b7319e4e96f85bb5a6d8e218344d2f40afb91f12c36ab6e1417424ca8"}
Nov 01 03:17:38.041761 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.041665    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r" event={"ID":"5a115577-6d0a-4047-ab8b-8eb78f51acbe","Type":"ContainerStarted","Data":"b5814351a3c6aa46164405f9564bb6cde12ad22dc35004db84710fe560333bd6"}
Nov 01 03:17:38.043378 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.043350    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j" event={"ID":"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648","Type":"ContainerStarted","Data":"50e10515f8b8c5685a5aeb267f96fb5cbda7782d25a15cfad5613df7ba5078b2"}
Nov 01 03:17:38.043472 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.043407    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j" event={"ID":"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648","Type":"ContainerStarted","Data":"fe4aa40ed07a33a49e7ea585a8bed77550ef71e0afae2082b93c5a34dcbc451d"}
Nov 01 03:17:38.043692 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.043637    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"
Nov 01 03:17:38.044458 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.044436    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"
Nov 01 03:17:38.062722 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.062692    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"]
Nov 01 03:17:38.068946 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.068922    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"
Nov 01 03:17:38.079942 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.079847    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:38.137894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.137837    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"]
Nov 01 03:17:38.157267 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.157236    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c515c2c3-7e65-4512-9209-7994209c12e5-istio-podinfo\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst\" (UID: \"c515c2c3-7e65-4512-9209-7994209c12e5\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"
Nov 01 03:17:38.157419 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.157307    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c515c2c3-7e65-4512-9209-7994209c12e5-istio-data\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst\" (UID: \"c515c2c3-7e65-4512-9209-7994209c12e5\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"
Nov 01 03:17:38.157419 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.157375    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c515c2c3-7e65-4512-9209-7994209c12e5-credential-socket\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst\" (UID: \"c515c2c3-7e65-4512-9209-7994209c12e5\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"
Nov 01 03:17:38.157419 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.157415    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c515c2c3-7e65-4512-9209-7994209c12e5-istio-envoy\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst\" (UID: \"c515c2c3-7e65-4512-9209-7994209c12e5\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"
Nov 01 03:17:38.157585 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.157440    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c515c2c3-7e65-4512-9209-7994209c12e5-workload-socket\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst\" (UID: \"c515c2c3-7e65-4512-9209-7994209c12e5\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"
Nov 01 03:17:38.157585 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.157478    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c515c2c3-7e65-4512-9209-7994209c12e5-workload-certs\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst\" (UID: \"c515c2c3-7e65-4512-9209-7994209c12e5\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"
Nov 01 03:17:38.157585 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.157505    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncqld\" (UniqueName: \"kubernetes.io/projected/c515c2c3-7e65-4512-9209-7994209c12e5-kube-api-access-ncqld\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst\" (UID: \"c515c2c3-7e65-4512-9209-7994209c12e5\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"
Nov 01 03:17:38.157585 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.157539    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c515c2c3-7e65-4512-9209-7994209c12e5-istio-token\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst\" (UID: \"c515c2c3-7e65-4512-9209-7994209c12e5\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"
Nov 01 03:17:38.157779 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.157665    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c515c2c3-7e65-4512-9209-7994209c12e5-istiod-ca-cert\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst\" (UID: \"c515c2c3-7e65-4512-9209-7994209c12e5\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"
Nov 01 03:17:38.163774 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.163731    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r" podStartSLOduration=2.163718414 podStartE2EDuration="2.163718414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:17:38.163230763 +0000 UTC m=+662.686237351" watchObservedRunningTime="2026-11-01 03:17:38.163718414 +0000 UTC m=+662.686725001"
Nov 01 03:17:38.209087 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.209060    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-e8c789-openshift-default-dockercfg-vpp4p\""
Nov 01 03:17:38.222716 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.222685    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:38.258357 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.258334    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c515c2c3-7e65-4512-9209-7994209c12e5-credential-socket\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst\" (UID: \"c515c2c3-7e65-4512-9209-7994209c12e5\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"
Nov 01 03:17:38.258551 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.258369    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c515c2c3-7e65-4512-9209-7994209c12e5-istio-envoy\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst\" (UID: \"c515c2c3-7e65-4512-9209-7994209c12e5\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"
Nov 01 03:17:38.258551 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.258387    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c515c2c3-7e65-4512-9209-7994209c12e5-workload-socket\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst\" (UID: \"c515c2c3-7e65-4512-9209-7994209c12e5\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"
Nov 01 03:17:38.258551 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.258410    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c515c2c3-7e65-4512-9209-7994209c12e5-workload-certs\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst\" (UID: \"c515c2c3-7e65-4512-9209-7994209c12e5\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"
Nov 01 03:17:38.258551 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.258431    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncqld\" (UniqueName: \"kubernetes.io/projected/c515c2c3-7e65-4512-9209-7994209c12e5-kube-api-access-ncqld\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst\" (UID: \"c515c2c3-7e65-4512-9209-7994209c12e5\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"
Nov 01 03:17:38.258551 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.258465    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c515c2c3-7e65-4512-9209-7994209c12e5-istio-token\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst\" (UID: \"c515c2c3-7e65-4512-9209-7994209c12e5\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"
Nov 01 03:17:38.258551 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.258537    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c515c2c3-7e65-4512-9209-7994209c12e5-istiod-ca-cert\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst\" (UID: \"c515c2c3-7e65-4512-9209-7994209c12e5\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"
Nov 01 03:17:38.258871 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.258601    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c515c2c3-7e65-4512-9209-7994209c12e5-istio-podinfo\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst\" (UID: \"c515c2c3-7e65-4512-9209-7994209c12e5\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"
Nov 01 03:17:38.258871 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.258650    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c515c2c3-7e65-4512-9209-7994209c12e5-istio-data\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst\" (UID: \"c515c2c3-7e65-4512-9209-7994209c12e5\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"
Nov 01 03:17:38.258871 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.258793    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c515c2c3-7e65-4512-9209-7994209c12e5-credential-socket\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst\" (UID: \"c515c2c3-7e65-4512-9209-7994209c12e5\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"
Nov 01 03:17:38.259004 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.258881    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c515c2c3-7e65-4512-9209-7994209c12e5-workload-socket\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst\" (UID: \"c515c2c3-7e65-4512-9209-7994209c12e5\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"
Nov 01 03:17:38.259079 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.259057    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c515c2c3-7e65-4512-9209-7994209c12e5-istio-data\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst\" (UID: \"c515c2c3-7e65-4512-9209-7994209c12e5\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"
Nov 01 03:17:38.259219 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.259203    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c515c2c3-7e65-4512-9209-7994209c12e5-workload-certs\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst\" (UID: \"c515c2c3-7e65-4512-9209-7994209c12e5\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"
Nov 01 03:17:38.259330 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.259305    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c515c2c3-7e65-4512-9209-7994209c12e5-istiod-ca-cert\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst\" (UID: \"c515c2c3-7e65-4512-9209-7994209c12e5\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"
Nov 01 03:17:38.260817 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.260797    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c515c2c3-7e65-4512-9209-7994209c12e5-istio-envoy\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst\" (UID: \"c515c2c3-7e65-4512-9209-7994209c12e5\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"
Nov 01 03:17:38.261044 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.261025    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c515c2c3-7e65-4512-9209-7994209c12e5-istio-podinfo\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst\" (UID: \"c515c2c3-7e65-4512-9209-7994209c12e5\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"
Nov 01 03:17:38.320395 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.320322    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c515c2c3-7e65-4512-9209-7994209c12e5-istio-token\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst\" (UID: \"c515c2c3-7e65-4512-9209-7994209c12e5\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"
Nov 01 03:17:38.321637 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.321613    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncqld\" (UniqueName: \"kubernetes.io/projected/c515c2c3-7e65-4512-9209-7994209c12e5-kube-api-access-ncqld\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst\" (UID: \"c515c2c3-7e65-4512-9209-7994209c12e5\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"
Nov 01 03:17:38.363337 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.363312    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"
Nov 01 03:17:38.368118 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.368094    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"
Nov 01 03:17:38.369290 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.369265    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:38.386127 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.386102    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"
Nov 01 03:17:38.446176 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.446125    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j" podStartSLOduration=2.446108273 podStartE2EDuration="2.446108273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:17:38.429875691 +0000 UTC m=+662.952882291" watchObservedRunningTime="2026-11-01 03:17:38.446108273 +0000 UTC m=+662.969114860"
Nov 01 03:17:38.446413 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.446395    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"]
Nov 01 03:17:38.447313 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:17:38.447286    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod267415f5_02bf_4571_914a_45aa36ae41c9.slice/crio-8bdb2dcda1f847538c19a2d1b802fb4573e7cd87b90d3c4b7703b21f8e68614b WatchSource:0}: Error finding container 8bdb2dcda1f847538c19a2d1b802fb4573e7cd87b90d3c4b7703b21f8e68614b: Status 404 returned error can't find the container with id 8bdb2dcda1f847538c19a2d1b802fb4573e7cd87b90d3c4b7703b21f8e68614b
Nov 01 03:17:38.450083 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.450046    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:17:38.450167 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.450127    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:17:38.450219 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.450202    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:17:38.723191 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.723159    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"
Nov 01 03:17:38.728754 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.728730    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"
Nov 01 03:17:38.792968 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.792945    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"]
Nov 01 03:17:38.795211 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:17:38.795178    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc515c2c3_7e65_4512_9209_7994209c12e5.slice/crio-e4fd51851cdab6903c4f54bb181348debbebc1552ae14974c384c7bf36b07230 WatchSource:0}: Error finding container e4fd51851cdab6903c4f54bb181348debbebc1552ae14974c384c7bf36b07230: Status 404 returned error can't find the container with id e4fd51851cdab6903c4f54bb181348debbebc1552ae14974c384c7bf36b07230
Nov 01 03:17:38.797457 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.797426    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:17:38.797546 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.797492    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:17:38.797546 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:38.797520    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:17:39.050434 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:39.050342    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst" event={"ID":"c515c2c3-7e65-4512-9209-7994209c12e5","Type":"ContainerStarted","Data":"24e2bb6cb44527449d7d19395c72ed888c39b92171bbf9fc5f5fe1ea2ddab61c"}
Nov 01 03:17:39.050434 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:39.050403    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst" event={"ID":"c515c2c3-7e65-4512-9209-7994209c12e5","Type":"ContainerStarted","Data":"e4fd51851cdab6903c4f54bb181348debbebc1552ae14974c384c7bf36b07230"}
Nov 01 03:17:39.052183 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:39.052159    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr" event={"ID":"267415f5-02bf-4571-914a-45aa36ae41c9","Type":"ContainerStarted","Data":"414127804b6b9477bd68c04db236c25a65d858c5da049a77a83d81fa391fd91b"}
Nov 01 03:17:39.052300 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:39.052204    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr" event={"ID":"267415f5-02bf-4571-914a-45aa36ae41c9","Type":"ContainerStarted","Data":"8bdb2dcda1f847538c19a2d1b802fb4573e7cd87b90d3c4b7703b21f8e68614b"}
Nov 01 03:17:39.053137 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:39.052671    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"
Nov 01 03:17:39.053137 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:39.052805    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"
Nov 01 03:17:39.053491 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:39.053471    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"
Nov 01 03:17:39.053606 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:39.053551    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"
Nov 01 03:17:39.246252 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:39.246219    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:39.250604 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:39.250541    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst" podStartSLOduration=1.250525439 podStartE2EDuration="1.250525439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:17:39.249326765 +0000 UTC m=+663.772333354" watchObservedRunningTime="2026-11-01 03:17:39.250525439 +0000 UTC m=+663.773532028"
Nov 01 03:17:39.389031 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:39.388951    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"
Nov 01 03:17:39.400722 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:39.400693    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"
Nov 01 03:17:39.624321 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:39.624265    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr" podStartSLOduration=2.624249064 podStartE2EDuration="2.624249064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:17:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:17:39.622919176 +0000 UTC m=+664.145925765" watchObservedRunningTime="2026-11-01 03:17:39.624249064 +0000 UTC m=+664.147255660"
Nov 01 03:17:40.038754 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:40.038717    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"
Nov 01 03:17:40.043854 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:40.043829    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"
Nov 01 03:17:40.058938 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:40.058888    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"
Nov 01 03:17:40.058938 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:40.058935    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"
Nov 01 03:17:40.059307 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:40.059280    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"
Nov 01 03:17:40.059622 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:40.059606    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"
Nov 01 03:17:40.349455 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:40.349417    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:40.522257 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:40.522224    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:40.687778 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:40.687663    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:41.813010 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:41.812946    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:42.096433 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:42.096403    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:42.244901 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:42.244837    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:43.563373 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:43.563346    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:43.744942 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:43.744909    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:43.892693 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:43.892665    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:44.145404 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:44.145304    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:44.301930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:44.301894    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:44.459822 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:44.459747    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:44.799165 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:44.799070    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:45.043034 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:45.043003    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:45.406022 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:45.405992    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:45.591365 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:45.591331    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:45.739447 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:45.739361    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:46.017077 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:46.016999    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:46.290680 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:46.290590    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:46.442952 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:46.442918    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:46.875779 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:46.875695    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:47.056094 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:47.056060    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:47.204825 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:47.204740    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:49.122159 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:17:49.122127    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:17:51.388331 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:51.388302    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:51.582725 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:51.582691    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:52.152250 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:52.152210    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:53.907896 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:53.907864    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:54.103405 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:54.103370    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:56.441302 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:56.441264    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:56.636093 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:56.636061    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:57.386107 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:57.386065    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:57.555113 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:57.555074    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:17:57.831441 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:17:57.831345    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:18:02.494168 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:02.494138    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:18:02.698127 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:02.698083    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:18:02.869935 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:02.869894    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:18:03.598669 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:03.598630    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:18:03.868852 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:03.868766    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:18:04.170131 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:04.170046    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e8c789-5b848f6fc5-9s298"]
Nov 01 03:18:04.173982 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:04.173958    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-e8c789-5b848f6fc5-9s298"
Nov 01 03:18:04.189357 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:04.189318    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxmwn\" (UniqueName: \"kubernetes.io/projected/1f72415b-cf6a-40c8-b9ab-33763ebbf66a-kube-api-access-fxmwn\") pod \"payload-processing-e2e-worker-w4-e8c789-5b848f6fc5-9s298\" (UID: \"1f72415b-cf6a-40c8-b9ab-33763ebbf66a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e8c789-5b848f6fc5-9s298"
Nov 01 03:18:04.189512 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:04.189488    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f72415b-cf6a-40c8-b9ab-33763ebbf66a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-e8c789-5b848f6fc5-9s298\" (UID: \"1f72415b-cf6a-40c8-b9ab-33763ebbf66a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e8c789-5b848f6fc5-9s298"
Nov 01 03:18:04.310844 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:04.310810    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-e8c789-dockercfg-cq2f5\""
Nov 01 03:18:04.311266 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:04.311213    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-e8c789\""
Nov 01 03:18:04.312114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:04.312071    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f72415b-cf6a-40c8-b9ab-33763ebbf66a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-e8c789-5b848f6fc5-9s298\" (UID: \"1f72415b-cf6a-40c8-b9ab-33763ebbf66a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e8c789-5b848f6fc5-9s298"
Nov 01 03:18:04.312940 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:04.312917    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f72415b-cf6a-40c8-b9ab-33763ebbf66a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-e8c789-5b848f6fc5-9s298\" (UID: \"1f72415b-cf6a-40c8-b9ab-33763ebbf66a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e8c789-5b848f6fc5-9s298"
Nov 01 03:18:04.313102 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:04.313085    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxmwn\" (UniqueName: \"kubernetes.io/projected/1f72415b-cf6a-40c8-b9ab-33763ebbf66a-kube-api-access-fxmwn\") pod \"payload-processing-e2e-worker-w4-e8c789-5b848f6fc5-9s298\" (UID: \"1f72415b-cf6a-40c8-b9ab-33763ebbf66a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e8c789-5b848f6fc5-9s298"
Nov 01 03:18:04.319457 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:04.319432    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e8c789-5b848f6fc5-9s298"]
Nov 01 03:18:04.376166 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:04.376128    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxmwn\" (UniqueName: \"kubernetes.io/projected/1f72415b-cf6a-40c8-b9ab-33763ebbf66a-kube-api-access-fxmwn\") pod \"payload-processing-e2e-worker-w4-e8c789-5b848f6fc5-9s298\" (UID: \"1f72415b-cf6a-40c8-b9ab-33763ebbf66a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e8c789-5b848f6fc5-9s298"
Nov 01 03:18:04.491731 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:04.491703    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-e8c789-5b848f6fc5-9s298"
Nov 01 03:18:04.887603 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:04.887577    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e8c789-5b848f6fc5-9s298"]
Nov 01 03:18:04.889058 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:18:04.889030    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f72415b_cf6a_40c8_b9ab_33763ebbf66a.slice/crio-6d33f77984dbf9af35ae99d225dddabe4f52581a0e2fc3771d319229e8f628c9 WatchSource:0}: Error finding container 6d33f77984dbf9af35ae99d225dddabe4f52581a0e2fc3771d319229e8f628c9: Status 404 returned error can't find the container with id 6d33f77984dbf9af35ae99d225dddabe4f52581a0e2fc3771d319229e8f628c9
Nov 01 03:18:04.891093 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:04.891076    4930 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 03:18:05.234894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:05.234814    4930 generic.go:356] "Generic (PLEG): container finished" podID="58f8e6d8-aafa-4a82-b1e4-63c358380207" containerID="aafc35f35a3dde77e5b573be2be98eaa2018cd07ef430dbbfdcc84e2359e07a1" exitCode=0
Nov 01 03:18:05.234894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:05.234881    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm" event={"ID":"58f8e6d8-aafa-4a82-b1e4-63c358380207","Type":"ContainerDied","Data":"aafc35f35a3dde77e5b573be2be98eaa2018cd07ef430dbbfdcc84e2359e07a1"}
Nov 01 03:18:05.235257 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:05.235239    4930 scope.go:122] "RemoveContainer" containerID="aafc35f35a3dde77e5b573be2be98eaa2018cd07ef430dbbfdcc84e2359e07a1"
Nov 01 03:18:05.236302 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:05.236271    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-e8c789-5b848f6fc5-9s298" event={"ID":"1f72415b-cf6a-40c8-b9ab-33763ebbf66a","Type":"ContainerStarted","Data":"6d33f77984dbf9af35ae99d225dddabe4f52581a0e2fc3771d319229e8f628c9"}
Nov 01 03:18:05.236458 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:05.236434    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:18:05.236505 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:05.236498    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:18:05.236539 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:05.236527    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:18:06.113131 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:06.113095    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:18:06.306581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:06.306454    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:18:06.312584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:06.306908    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm" event={"ID":"58f8e6d8-aafa-4a82-b1e4-63c358380207","Type":"ContainerStarted","Data":"a2a769f870a5917c452b00a2196b52e027797797f5ac49010ec4fec40e450dc6"}
Nov 01 03:18:06.425866 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:06.424872    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"
Nov 01 03:18:06.447634 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:06.430671    4930 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 01 03:18:06.447634 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:06.430739    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm" podUID="58f8e6d8-aafa-4a82-b1e4-63c358380207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 01 03:18:07.425220 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:07.424783    4930 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 01 03:18:07.425220 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:07.424839    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm" podUID="58f8e6d8-aafa-4a82-b1e4-63c358380207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 01 03:18:08.044402 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:08.044371    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"
Nov 01 03:18:08.107527 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:08.107484    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:18:08.424818 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:08.424781    4930 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 01 03:18:08.424992 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:08.424836    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm" podUID="58f8e6d8-aafa-4a82-b1e4-63c358380207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 01 03:18:09.424425 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:09.423969    4930 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 01 03:18:09.424425 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:09.424028    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm" podUID="58f8e6d8-aafa-4a82-b1e4-63c358380207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 01 03:18:10.339195 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:10.339158    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-e8c789-5b848f6fc5-9s298" event={"ID":"1f72415b-cf6a-40c8-b9ab-33763ebbf66a","Type":"ContainerStarted","Data":"386ac520879aefb3585d501e641f74490a85ed70c0e2cf5c08a68d442a1b8ef4"}
Nov 01 03:18:10.339381 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:10.339286    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-e8c789-5b848f6fc5-9s298"
Nov 01 03:18:10.424663 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:10.424630    4930 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 01 03:18:10.425097 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:10.424685    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm" podUID="58f8e6d8-aafa-4a82-b1e4-63c358380207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 01 03:18:10.478704 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:10.478631    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-e8c789-5b848f6fc5-9s298" podStartSLOduration=1.429768887 podStartE2EDuration="6.478610716s" totalImagesPullingTime="5.048841829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:18:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:18:10.458596926 +0000 UTC m=+694.981603515" watchObservedRunningTime="2026-11-01 03:18:10.478610716 +0000 UTC m=+695.001617306"
Nov 01 03:18:11.024529 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:11.024492    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:18:11.216334 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:11.216293    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:18:11.424865 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:11.424837    4930 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 01 03:18:11.425239 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:11.424886    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm" podUID="58f8e6d8-aafa-4a82-b1e4-63c358380207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 01 03:18:12.428680 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:12.428650    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"
Nov 01 03:18:12.429735 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:12.429714    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"
Nov 01 03:18:13.279197 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:13.279161    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:18:13.488914 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:13.488872    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:18:13.638793 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:13.638761    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:18:14.365547 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:14.365509    4930 generic.go:356] "Generic (PLEG): container finished" podID="f3c4cdb7-e686-4b71-9ab9-3c82f4c24648" containerID="50e10515f8b8c5685a5aeb267f96fb5cbda7782d25a15cfad5613df7ba5078b2" exitCode=0
Nov 01 03:18:14.365765 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:14.365577    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j" event={"ID":"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648","Type":"ContainerDied","Data":"50e10515f8b8c5685a5aeb267f96fb5cbda7782d25a15cfad5613df7ba5078b2"}
Nov 01 03:18:14.365960 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:14.365943    4930 scope.go:122] "RemoveContainer" containerID="50e10515f8b8c5685a5aeb267f96fb5cbda7782d25a15cfad5613df7ba5078b2"
Nov 01 03:18:14.367281 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:14.367253    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:18:14.367369 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:14.367333    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:18:14.367369 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:14.367365    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:18:15.178580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:15.177686    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:18:15.402273 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:15.402241    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j" event={"ID":"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648","Type":"ContainerStarted","Data":"143d2317b405875bd15fb6fcbcc10bc0a02c0104fb4b8569a986b10ddc70ec71"}
Nov 01 03:18:15.723005 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:15.722971    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"
Nov 01 03:18:15.723282 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:15.723258    4930 patch_prober.go:29] interesting pod/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 01 03:18:15.723340 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:15.723316    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j" podUID="f3c4cdb7-e686-4b71-9ab9-3c82f4c24648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 01 03:18:15.756304 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:15.756276    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:18:16.081596 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:16.081489    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:18:16.723569 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:16.723517    4930 patch_prober.go:29] interesting pod/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 01 03:18:16.724193 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:16.724156    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j" podUID="f3c4cdb7-e686-4b71-9ab9-3c82f4c24648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 01 03:18:17.723207 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:17.723170    4930 patch_prober.go:29] interesting pod/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 01 03:18:17.723409 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:17.723231    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j" podUID="f3c4cdb7-e686-4b71-9ab9-3c82f4c24648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 01 03:18:18.695247 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:18.695215    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"]
Nov 01 03:18:18.700274 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:18.700243    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:18:18.707971 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:18.707613    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:18:18.723369 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:18.723347    4930 patch_prober.go:29] interesting pod/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 01 03:18:18.723451 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:18.723397    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j" podUID="f3c4cdb7-e686-4b71-9ab9-3c82f4c24648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 01 03:18:18.734386 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:18.734361    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"]
Nov 01 03:18:18.787839 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:18.787806    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b12199\"/\"default-dockercfg-hx678\""
Nov 01 03:18:18.788297 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:18.788275    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b12199\"/\"openshift-service-ca.crt\""
Nov 01 03:18:18.789004 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:18.788505    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b12199\"/\"kube-root-ca.crt\""
Nov 01 03:18:18.789149 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:18.788605    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b12199\"/\"simulator-w1-b12199-kserve-self-signed-certs\""
Nov 01 03:18:18.796576 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:18.796538    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0e32cb3-ac18-4c1f-819c-f1139a88bfea-home\") pod \"simulator-w1-b12199-kserve-7cfcb76798-g5hnj\" (UID: \"b0e32cb3-ac18-4c1f-819c-f1139a88bfea\") " pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:18:18.796771 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:18.796751    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0e32cb3-ac18-4c1f-819c-f1139a88bfea-dshm\") pod \"simulator-w1-b12199-kserve-7cfcb76798-g5hnj\" (UID: \"b0e32cb3-ac18-4c1f-819c-f1139a88bfea\") " pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:18:18.796857 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:18.796812    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrp6z\" (UniqueName: \"kubernetes.io/projected/b0e32cb3-ac18-4c1f-819c-f1139a88bfea-kube-api-access-xrp6z\") pod \"simulator-w1-b12199-kserve-7cfcb76798-g5hnj\" (UID: \"b0e32cb3-ac18-4c1f-819c-f1139a88bfea\") " pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:18:18.796917 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:18.796868    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0e32cb3-ac18-4c1f-819c-f1139a88bfea-tls-certs\") pod \"simulator-w1-b12199-kserve-7cfcb76798-g5hnj\" (UID: \"b0e32cb3-ac18-4c1f-819c-f1139a88bfea\") " pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:18:18.796995 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:18.796976    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0e32cb3-ac18-4c1f-819c-f1139a88bfea-model-cache\") pod \"simulator-w1-b12199-kserve-7cfcb76798-g5hnj\" (UID: \"b0e32cb3-ac18-4c1f-819c-f1139a88bfea\") " pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:18:18.797229 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:18.797209    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0e32cb3-ac18-4c1f-819c-f1139a88bfea-tmp-dir\") pod \"simulator-w1-b12199-kserve-7cfcb76798-g5hnj\" (UID: \"b0e32cb3-ac18-4c1f-819c-f1139a88bfea\") " pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:18:18.883678 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:18.883628    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:18:18.898787 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:18.898762    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0e32cb3-ac18-4c1f-819c-f1139a88bfea-model-cache\") pod \"simulator-w1-b12199-kserve-7cfcb76798-g5hnj\" (UID: \"b0e32cb3-ac18-4c1f-819c-f1139a88bfea\") " pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:18:18.898901 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:18.898801    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0e32cb3-ac18-4c1f-819c-f1139a88bfea-tmp-dir\") pod \"simulator-w1-b12199-kserve-7cfcb76798-g5hnj\" (UID: \"b0e32cb3-ac18-4c1f-819c-f1139a88bfea\") " pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:18:18.898901 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:18.898838    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0e32cb3-ac18-4c1f-819c-f1139a88bfea-home\") pod \"simulator-w1-b12199-kserve-7cfcb76798-g5hnj\" (UID: \"b0e32cb3-ac18-4c1f-819c-f1139a88bfea\") " pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:18:18.898901 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:18.898877    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0e32cb3-ac18-4c1f-819c-f1139a88bfea-dshm\") pod \"simulator-w1-b12199-kserve-7cfcb76798-g5hnj\" (UID: \"b0e32cb3-ac18-4c1f-819c-f1139a88bfea\") " pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:18:18.899057 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:18.898907    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrp6z\" (UniqueName: \"kubernetes.io/projected/b0e32cb3-ac18-4c1f-819c-f1139a88bfea-kube-api-access-xrp6z\") pod \"simulator-w1-b12199-kserve-7cfcb76798-g5hnj\" (UID: \"b0e32cb3-ac18-4c1f-819c-f1139a88bfea\") " pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:18:18.899057 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:18.898944    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0e32cb3-ac18-4c1f-819c-f1139a88bfea-tls-certs\") pod \"simulator-w1-b12199-kserve-7cfcb76798-g5hnj\" (UID: \"b0e32cb3-ac18-4c1f-819c-f1139a88bfea\") " pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:18:18.899256 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:18.899230    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0e32cb3-ac18-4c1f-819c-f1139a88bfea-tmp-dir\") pod \"simulator-w1-b12199-kserve-7cfcb76798-g5hnj\" (UID: \"b0e32cb3-ac18-4c1f-819c-f1139a88bfea\") " pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:18:18.899332 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:18.899275    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0e32cb3-ac18-4c1f-819c-f1139a88bfea-model-cache\") pod \"simulator-w1-b12199-kserve-7cfcb76798-g5hnj\" (UID: \"b0e32cb3-ac18-4c1f-819c-f1139a88bfea\") " pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:18:18.899405 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:18.899380    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0e32cb3-ac18-4c1f-819c-f1139a88bfea-home\") pod \"simulator-w1-b12199-kserve-7cfcb76798-g5hnj\" (UID: \"b0e32cb3-ac18-4c1f-819c-f1139a88bfea\") " pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:18:18.901219 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:18.901194    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0e32cb3-ac18-4c1f-819c-f1139a88bfea-dshm\") pod \"simulator-w1-b12199-kserve-7cfcb76798-g5hnj\" (UID: \"b0e32cb3-ac18-4c1f-819c-f1139a88bfea\") " pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:18:18.901512 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:18.901491    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0e32cb3-ac18-4c1f-819c-f1139a88bfea-tls-certs\") pod \"simulator-w1-b12199-kserve-7cfcb76798-g5hnj\" (UID: \"b0e32cb3-ac18-4c1f-819c-f1139a88bfea\") " pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:18:18.978319 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:18.978248    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrp6z\" (UniqueName: \"kubernetes.io/projected/b0e32cb3-ac18-4c1f-819c-f1139a88bfea-kube-api-access-xrp6z\") pod \"simulator-w1-b12199-kserve-7cfcb76798-g5hnj\" (UID: \"b0e32cb3-ac18-4c1f-819c-f1139a88bfea\") " pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:18:19.016628 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:19.016581    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:18:19.033395 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:19.033364    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:18:19.195940 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:18:19.195900    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:18:19.416766 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:19.416741    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"]
Nov 01 03:18:19.419148 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:18:19.419114    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0e32cb3_ac18_4c1f_819c_f1139a88bfea.slice/crio-1f307b357bbee507e31b256c0fbb1a61a07d4c123c5ae3b27bb8b7fd9ad26b11 WatchSource:0}: Error finding container 1f307b357bbee507e31b256c0fbb1a61a07d4c123c5ae3b27bb8b7fd9ad26b11: Status 404 returned error can't find the container with id 1f307b357bbee507e31b256c0fbb1a61a07d4c123c5ae3b27bb8b7fd9ad26b11
Nov 01 03:18:19.437862 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:19.437837    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj" event={"ID":"b0e32cb3-ac18-4c1f-819c-f1139a88bfea","Type":"ContainerStarted","Data":"1f307b357bbee507e31b256c0fbb1a61a07d4c123c5ae3b27bb8b7fd9ad26b11"}
Nov 01 03:18:19.723327 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:19.723245    4930 patch_prober.go:29] interesting pod/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 01 03:18:19.723327 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:19.723295    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j" podUID="f3c4cdb7-e686-4b71-9ab9-3c82f4c24648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 01 03:18:20.446366 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:20.446326    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj" event={"ID":"b0e32cb3-ac18-4c1f-819c-f1139a88bfea","Type":"ContainerStarted","Data":"c0bccd55d992db25325dd41c86a53c6948929d08e109184a2ba8a47a651901ca"}
Nov 01 03:18:20.576180 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:20.576111    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj" podStartSLOduration=2.576094222 podStartE2EDuration="2.576094222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:18:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:18:20.575106597 +0000 UTC m=+705.098113184" watchObservedRunningTime="2026-11-01 03:18:20.576094222 +0000 UTC m=+705.099100810"
Nov 01 03:18:20.727542 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:20.727464    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"
Nov 01 03:18:20.727990 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:20.727763    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"
Nov 01 03:18:20.728432 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:20.728416    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"
Nov 01 03:18:21.016967 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:21.016884    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:18:21.349612 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:21.349588    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-e8c789-5b848f6fc5-9s298"
Nov 01 03:18:23.616703 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:23.616668    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:18:23.784641 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:23.782146    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:18:23.990295 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:23.990189    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:18:28.605809 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:28.605741    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:18:28.823168 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:28.821787    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:18:29.017605 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:29.017505    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:18:29.029993 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:29.029967    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:18:29.510672 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:29.510636    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:18:29.540313 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:29.540276    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:18:32.552596 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:32.552532    4930 generic.go:356] "Generic (PLEG): container finished" podID="5a115577-6d0a-4047-ab8b-8eb78f51acbe" containerID="9094fc4b7319e4e96f85bb5a6d8e218344d2f40afb91f12c36ab6e1417424ca8" exitCode=0
Nov 01 03:18:32.552596 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:32.552593    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r" event={"ID":"5a115577-6d0a-4047-ab8b-8eb78f51acbe","Type":"ContainerDied","Data":"9094fc4b7319e4e96f85bb5a6d8e218344d2f40afb91f12c36ab6e1417424ca8"}
Nov 01 03:18:32.553096 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:32.553011    4930 scope.go:122] "RemoveContainer" containerID="9094fc4b7319e4e96f85bb5a6d8e218344d2f40afb91f12c36ab6e1417424ca8"
Nov 01 03:18:32.554542 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:32.554500    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:18:32.554710 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:32.554620    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:18:32.554710 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:32.554674    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:18:33.545775 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:33.545740    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:18:33.576579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:33.575020    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r" event={"ID":"5a115577-6d0a-4047-ab8b-8eb78f51acbe","Type":"ContainerStarted","Data":"20b525519fe9a48c409c20991b41e9576708c11064082a14690f7b6aa020727b"}
Nov 01 03:18:34.370150 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:34.369699    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"
Nov 01 03:18:34.370150 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:34.369824    4930 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 01 03:18:34.370150 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:34.369877    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r" podUID="5a115577-6d0a-4047-ab8b-8eb78f51acbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 01 03:18:34.916916 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:34.916889    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:18:35.382663 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:35.382617    4930 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 01 03:18:35.382951 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:35.382924    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r" podUID="5a115577-6d0a-4047-ab8b-8eb78f51acbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 01 03:18:36.378006 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:36.377965    4930 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 01 03:18:36.378668 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:36.378635    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r" podUID="5a115577-6d0a-4047-ab8b-8eb78f51acbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 01 03:18:37.363961 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:37.363919    4930 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 01 03:18:37.364250 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:37.364218    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r" podUID="5a115577-6d0a-4047-ab8b-8eb78f51acbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 01 03:18:38.363918 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:38.363414    4930 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Nov 01 03:18:38.363918 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:38.363480    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r" podUID="5a115577-6d0a-4047-ab8b-8eb78f51acbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Nov 01 03:18:38.493594 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:38.493542    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:18:39.052302 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:39.052264    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"
Nov 01 03:18:39.387579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:39.386854    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"
Nov 01 03:18:39.682283 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:39.682199    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:18:39.711786 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:39.711756    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"
Nov 01 03:18:43.344330 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:43.344284    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:18:44.743877 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:44.743847    4930 generic.go:356] "Generic (PLEG): container finished" podID="267415f5-02bf-4571-914a-45aa36ae41c9" containerID="414127804b6b9477bd68c04db236c25a65d858c5da049a77a83d81fa391fd91b" exitCode=0
Nov 01 03:18:44.744295 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:44.743872    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr" event={"ID":"267415f5-02bf-4571-914a-45aa36ae41c9","Type":"ContainerDied","Data":"414127804b6b9477bd68c04db236c25a65d858c5da049a77a83d81fa391fd91b"}
Nov 01 03:18:44.744295 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:44.744267    4930 scope.go:122] "RemoveContainer" containerID="414127804b6b9477bd68c04db236c25a65d858c5da049a77a83d81fa391fd91b"
Nov 01 03:18:44.745764 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:44.745736    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:18:44.745877 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:44.745774    4930 generic.go:356] "Generic (PLEG): container finished" podID="c515c2c3-7e65-4512-9209-7994209c12e5" containerID="24e2bb6cb44527449d7d19395c72ed888c39b92171bbf9fc5f5fe1ea2ddab61c" exitCode=0
Nov 01 03:18:44.745877 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:44.745804    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:18:44.745877 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:44.745837    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:18:44.745877 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:44.745851    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst" event={"ID":"c515c2c3-7e65-4512-9209-7994209c12e5","Type":"ContainerDied","Data":"24e2bb6cb44527449d7d19395c72ed888c39b92171bbf9fc5f5fe1ea2ddab61c"}
Nov 01 03:18:44.746278 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:44.746233    4930 scope.go:122] "RemoveContainer" containerID="24e2bb6cb44527449d7d19395c72ed888c39b92171bbf9fc5f5fe1ea2ddab61c"
Nov 01 03:18:44.747520 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:44.747484    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:18:44.747619 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:44.747590    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:18:44.747666 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:44.747643    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:18:45.754382 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:45.754342    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr" event={"ID":"267415f5-02bf-4571-914a-45aa36ae41c9","Type":"ContainerStarted","Data":"e79fab7e2557d937d8010fea46cbac0a838d03f17abb66abf00e348bec67aa5d"}
Nov 01 03:18:45.757247 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:45.757216    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst" event={"ID":"c515c2c3-7e65-4512-9209-7994209c12e5","Type":"ContainerStarted","Data":"e884a1d81198148e1d9e64c0b5d0028c48c9ea3898519ef56be32ab7a302dac6"}
Nov 01 03:18:46.040120 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:46.040023    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"
Nov 01 03:18:46.040490 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:46.040462    4930 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 01 03:18:46.040729 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:46.040700    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr" podUID="267415f5-02bf-4571-914a-45aa36ae41c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 01 03:18:46.394010 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:46.393968    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"
Nov 01 03:18:46.394416 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:46.394324    4930 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 01 03:18:46.394637 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:46.394609    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst" podUID="c515c2c3-7e65-4512-9209-7994209c12e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 01 03:18:47.039783 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:47.039748    4930 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 01 03:18:47.040165 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:47.039804    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr" podUID="267415f5-02bf-4571-914a-45aa36ae41c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 01 03:18:47.377581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:47.372800    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:18:47.387945 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:47.387905    4930 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 01 03:18:47.388223 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:47.388185    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst" podUID="c515c2c3-7e65-4512-9209-7994209c12e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 01 03:18:47.951959 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:47.951928    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:18:48.039163 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:48.039127    4930 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 01 03:18:48.039328 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:48.039192    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr" podUID="267415f5-02bf-4571-914a-45aa36ae41c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 01 03:18:48.110583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:48.110514    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:18:48.281637 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:48.272148    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:18:48.387168 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:48.387132    4930 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 01 03:18:48.387400 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:48.387372    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst" podUID="c515c2c3-7e65-4512-9209-7994209c12e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 01 03:18:48.850481 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:48.850453    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:18:49.039225 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:49.039190    4930 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 01 03:18:49.039427 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:49.039248    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr" podUID="267415f5-02bf-4571-914a-45aa36ae41c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 01 03:18:49.130482 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:18:49.130397    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:18:49.387463 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:49.386940    4930 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 01 03:18:49.387463 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:49.387016    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst" podUID="c515c2c3-7e65-4512-9209-7994209c12e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 01 03:18:49.503279 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:49.503245    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:18:50.039330 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:50.039296    4930 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 01 03:18:50.039493 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:50.039352    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr" podUID="267415f5-02bf-4571-914a-45aa36ae41c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 01 03:18:50.395981 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:50.395943    4930 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 01 03:18:50.396629 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:50.396595    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst" podUID="c515c2c3-7e65-4512-9209-7994209c12e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 01 03:18:51.039070 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:51.039027    4930 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 01 03:18:51.039252 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:51.039089    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr" podUID="267415f5-02bf-4571-914a-45aa36ae41c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 01 03:18:51.390808 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:51.390780    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"
Nov 01 03:18:51.391201 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:51.391182    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"
Nov 01 03:18:51.391951 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:51.391930    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"
Nov 01 03:18:52.043880 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:52.043850    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"
Nov 01 03:18:52.044327 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:52.044280    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"
Nov 01 03:18:52.045039 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:52.045021    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"
Nov 01 03:18:53.530924 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:53.530879    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:18:54.147126 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:54.147078    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:18:54.308009 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:54.307973    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:18:59.453363 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:18:59.453328    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:19:02.768124 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:02.768066    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:19:03.360530 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:03.360493    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:19:03.527137 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:03.527103    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:19:04.753997 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:04.753959    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:19:07.961659 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:07.961617    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:19:08.611645 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:08.611607    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:19:08.759130 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:08.759100    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:19:09.855231 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:09.855189    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:19:13.790755 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:13.790724    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:19:14.383665 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:14.383624    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:19:14.531760 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:14.531727    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:19:14.681866 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:14.681790    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"]
Nov 01 03:19:14.684433 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:14.684417    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:19:14.760035 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:14.760005    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzmjc\" (UniqueName: \"kubernetes.io/projected/13930c1e-2200-4412-b8d3-ae3e4126a851-kube-api-access-pzmjc\") pod \"simulator-w6-19faa5-kserve-5b5796d994-khm8t\" (UID: \"13930c1e-2200-4412-b8d3-ae3e4126a851\") " pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:19:14.760184 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:14.760055    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13930c1e-2200-4412-b8d3-ae3e4126a851-model-cache\") pod \"simulator-w6-19faa5-kserve-5b5796d994-khm8t\" (UID: \"13930c1e-2200-4412-b8d3-ae3e4126a851\") " pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:19:14.760184 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:14.760172    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13930c1e-2200-4412-b8d3-ae3e4126a851-tmp-dir\") pod \"simulator-w6-19faa5-kserve-5b5796d994-khm8t\" (UID: \"13930c1e-2200-4412-b8d3-ae3e4126a851\") " pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:19:14.760317 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:14.760199    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13930c1e-2200-4412-b8d3-ae3e4126a851-dshm\") pod \"simulator-w6-19faa5-kserve-5b5796d994-khm8t\" (UID: \"13930c1e-2200-4412-b8d3-ae3e4126a851\") " pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:19:14.760382 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:14.760338    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13930c1e-2200-4412-b8d3-ae3e4126a851-tls-certs\") pod \"simulator-w6-19faa5-kserve-5b5796d994-khm8t\" (UID: \"13930c1e-2200-4412-b8d3-ae3e4126a851\") " pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:19:14.760436 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:14.760425    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13930c1e-2200-4412-b8d3-ae3e4126a851-home\") pod \"simulator-w6-19faa5-kserve-5b5796d994-khm8t\" (UID: \"13930c1e-2200-4412-b8d3-ae3e4126a851\") " pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:19:14.778504 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:14.778478    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-19faa5\"/\"openshift-service-ca.crt\""
Nov 01 03:19:14.778660 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:14.778512    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-19faa5\"/\"kube-root-ca.crt\""
Nov 01 03:19:14.780627 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:14.780603    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:19:14.781580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:14.781547    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"]
Nov 01 03:19:14.803331 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:14.803314    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-19faa5\"/\"default-dockercfg-8s7vl\""
Nov 01 03:19:14.803693 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:14.803375    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-19faa5\"/\"simulator-w6-19faa5-kserve-self-signed-certs\""
Nov 01 03:19:14.861317 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:14.861293    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13930c1e-2200-4412-b8d3-ae3e4126a851-model-cache\") pod \"simulator-w6-19faa5-kserve-5b5796d994-khm8t\" (UID: \"13930c1e-2200-4412-b8d3-ae3e4126a851\") " pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:19:14.861458 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:14.861338    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13930c1e-2200-4412-b8d3-ae3e4126a851-tmp-dir\") pod \"simulator-w6-19faa5-kserve-5b5796d994-khm8t\" (UID: \"13930c1e-2200-4412-b8d3-ae3e4126a851\") " pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:19:14.861458 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:14.861391    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13930c1e-2200-4412-b8d3-ae3e4126a851-dshm\") pod \"simulator-w6-19faa5-kserve-5b5796d994-khm8t\" (UID: \"13930c1e-2200-4412-b8d3-ae3e4126a851\") " pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:19:14.861586 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:14.861461    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13930c1e-2200-4412-b8d3-ae3e4126a851-tls-certs\") pod \"simulator-w6-19faa5-kserve-5b5796d994-khm8t\" (UID: \"13930c1e-2200-4412-b8d3-ae3e4126a851\") " pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:19:14.861586 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:14.861507    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13930c1e-2200-4412-b8d3-ae3e4126a851-home\") pod \"simulator-w6-19faa5-kserve-5b5796d994-khm8t\" (UID: \"13930c1e-2200-4412-b8d3-ae3e4126a851\") " pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:19:14.861706 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:14.861589    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzmjc\" (UniqueName: \"kubernetes.io/projected/13930c1e-2200-4412-b8d3-ae3e4126a851-kube-api-access-pzmjc\") pod \"simulator-w6-19faa5-kserve-5b5796d994-khm8t\" (UID: \"13930c1e-2200-4412-b8d3-ae3e4126a851\") " pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:19:14.861769 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:14.861723    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13930c1e-2200-4412-b8d3-ae3e4126a851-model-cache\") pod \"simulator-w6-19faa5-kserve-5b5796d994-khm8t\" (UID: \"13930c1e-2200-4412-b8d3-ae3e4126a851\") " pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:19:14.861821 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:14.861772    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13930c1e-2200-4412-b8d3-ae3e4126a851-tmp-dir\") pod \"simulator-w6-19faa5-kserve-5b5796d994-khm8t\" (UID: \"13930c1e-2200-4412-b8d3-ae3e4126a851\") " pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:19:14.861946 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:14.861929    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13930c1e-2200-4412-b8d3-ae3e4126a851-home\") pod \"simulator-w6-19faa5-kserve-5b5796d994-khm8t\" (UID: \"13930c1e-2200-4412-b8d3-ae3e4126a851\") " pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:19:14.863469 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:14.863453    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13930c1e-2200-4412-b8d3-ae3e4126a851-dshm\") pod \"simulator-w6-19faa5-kserve-5b5796d994-khm8t\" (UID: \"13930c1e-2200-4412-b8d3-ae3e4126a851\") " pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:19:14.863911 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:14.863892    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13930c1e-2200-4412-b8d3-ae3e4126a851-tls-certs\") pod \"simulator-w6-19faa5-kserve-5b5796d994-khm8t\" (UID: \"13930c1e-2200-4412-b8d3-ae3e4126a851\") " pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:19:14.921985 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:14.921958    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzmjc\" (UniqueName: \"kubernetes.io/projected/13930c1e-2200-4412-b8d3-ae3e4126a851-kube-api-access-pzmjc\") pod \"simulator-w6-19faa5-kserve-5b5796d994-khm8t\" (UID: \"13930c1e-2200-4412-b8d3-ae3e4126a851\") " pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:19:14.926687 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:14.926664    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:19:14.995597 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:14.995508    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:19:15.374467 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:15.374287    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"]
Nov 01 03:19:15.377450 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:19:15.377412    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13930c1e_2200_4412_b8d3_ae3e4126a851.slice/crio-bfcef07b400ce8a43b7987fedd0b20488a34a2daa38c55f54fc4e4c2263be184 WatchSource:0}: Error finding container bfcef07b400ce8a43b7987fedd0b20488a34a2daa38c55f54fc4e4c2263be184: Status 404 returned error can't find the container with id bfcef07b400ce8a43b7987fedd0b20488a34a2daa38c55f54fc4e4c2263be184
Nov 01 03:19:15.983690 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:15.983652    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t" event={"ID":"13930c1e-2200-4412-b8d3-ae3e4126a851","Type":"ContainerStarted","Data":"2df24b3e6888fcf107df563d3fe900e7eb77cf7c9623071339d8bff59698c040"}
Nov 01 03:19:15.983690 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:15.983696    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t" event={"ID":"13930c1e-2200-4412-b8d3-ae3e4126a851","Type":"ContainerStarted","Data":"bfcef07b400ce8a43b7987fedd0b20488a34a2daa38c55f54fc4e4c2263be184"}
Nov 01 03:19:15.995637 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:15.995613    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:19:16.102167 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:16.102112    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t" podStartSLOduration=2.102098509 podStartE2EDuration="2.102098509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:19:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:19:16.099997759 +0000 UTC m=+760.623004348" watchObservedRunningTime="2026-11-01 03:19:16.102098509 +0000 UTC m=+760.625105098"
Nov 01 03:19:19.121489 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:19:19.121405    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:19:19.340652 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:19.340614    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:19:19.970433 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:19.970396    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:19:20.128719 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:20.128684    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:19:20.294280 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:20.294181    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:19:23.772333 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:23.772293    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"]
Nov 01 03:19:23.775023 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:23.775004    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:19:23.865982 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:23.865949    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6d688b8-b911-4a6a-a842-2b0fb7a75e83-model-cache\") pod \"simulator-w5-c22dd8-kserve-66547dfc77-8bhmw\" (UID: \"b6d688b8-b911-4a6a-a842-2b0fb7a75e83\") " pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:19:23.865982 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:23.865985    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xltfx\" (UniqueName: \"kubernetes.io/projected/b6d688b8-b911-4a6a-a842-2b0fb7a75e83-kube-api-access-xltfx\") pod \"simulator-w5-c22dd8-kserve-66547dfc77-8bhmw\" (UID: \"b6d688b8-b911-4a6a-a842-2b0fb7a75e83\") " pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:19:23.866195 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:23.866004    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6d688b8-b911-4a6a-a842-2b0fb7a75e83-dshm\") pod \"simulator-w5-c22dd8-kserve-66547dfc77-8bhmw\" (UID: \"b6d688b8-b911-4a6a-a842-2b0fb7a75e83\") " pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:19:23.866195 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:23.866045    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6d688b8-b911-4a6a-a842-2b0fb7a75e83-home\") pod \"simulator-w5-c22dd8-kserve-66547dfc77-8bhmw\" (UID: \"b6d688b8-b911-4a6a-a842-2b0fb7a75e83\") " pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:19:23.866195 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:23.866162    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6d688b8-b911-4a6a-a842-2b0fb7a75e83-tmp-dir\") pod \"simulator-w5-c22dd8-kserve-66547dfc77-8bhmw\" (UID: \"b6d688b8-b911-4a6a-a842-2b0fb7a75e83\") " pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:19:23.866308 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:23.866210    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6d688b8-b911-4a6a-a842-2b0fb7a75e83-tls-certs\") pod \"simulator-w5-c22dd8-kserve-66547dfc77-8bhmw\" (UID: \"b6d688b8-b911-4a6a-a842-2b0fb7a75e83\") " pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:19:23.876685 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:23.876660    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-c22dd8\"/\"openshift-service-ca.crt\""
Nov 01 03:19:23.876812 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:23.876701    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-c22dd8\"/\"default-dockercfg-khjq4\""
Nov 01 03:19:23.876812 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:23.876721    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-c22dd8\"/\"kube-root-ca.crt\""
Nov 01 03:19:23.876923 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:23.876818    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-c22dd8\"/\"simulator-w5-c22dd8-kserve-self-signed-certs\""
Nov 01 03:19:23.879813 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:23.879794    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"]
Nov 01 03:19:23.890880 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:23.890849    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:19:23.967677 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:23.967647    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6d688b8-b911-4a6a-a842-2b0fb7a75e83-tmp-dir\") pod \"simulator-w5-c22dd8-kserve-66547dfc77-8bhmw\" (UID: \"b6d688b8-b911-4a6a-a842-2b0fb7a75e83\") " pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:19:23.967864 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:23.967694    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6d688b8-b911-4a6a-a842-2b0fb7a75e83-tls-certs\") pod \"simulator-w5-c22dd8-kserve-66547dfc77-8bhmw\" (UID: \"b6d688b8-b911-4a6a-a842-2b0fb7a75e83\") " pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:19:23.967864 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:23.967796    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6d688b8-b911-4a6a-a842-2b0fb7a75e83-model-cache\") pod \"simulator-w5-c22dd8-kserve-66547dfc77-8bhmw\" (UID: \"b6d688b8-b911-4a6a-a842-2b0fb7a75e83\") " pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:19:23.967864 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:23.967815    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xltfx\" (UniqueName: \"kubernetes.io/projected/b6d688b8-b911-4a6a-a842-2b0fb7a75e83-kube-api-access-xltfx\") pod \"simulator-w5-c22dd8-kserve-66547dfc77-8bhmw\" (UID: \"b6d688b8-b911-4a6a-a842-2b0fb7a75e83\") " pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:19:23.967864 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:23.967840    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6d688b8-b911-4a6a-a842-2b0fb7a75e83-dshm\") pod \"simulator-w5-c22dd8-kserve-66547dfc77-8bhmw\" (UID: \"b6d688b8-b911-4a6a-a842-2b0fb7a75e83\") " pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:19:23.968063 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:23.967879    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6d688b8-b911-4a6a-a842-2b0fb7a75e83-home\") pod \"simulator-w5-c22dd8-kserve-66547dfc77-8bhmw\" (UID: \"b6d688b8-b911-4a6a-a842-2b0fb7a75e83\") " pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:19:23.968162 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:23.968137    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6d688b8-b911-4a6a-a842-2b0fb7a75e83-tmp-dir\") pod \"simulator-w5-c22dd8-kserve-66547dfc77-8bhmw\" (UID: \"b6d688b8-b911-4a6a-a842-2b0fb7a75e83\") " pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:19:23.968246 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:23.968226    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6d688b8-b911-4a6a-a842-2b0fb7a75e83-model-cache\") pod \"simulator-w5-c22dd8-kserve-66547dfc77-8bhmw\" (UID: \"b6d688b8-b911-4a6a-a842-2b0fb7a75e83\") " pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:19:23.968350 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:23.968322    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6d688b8-b911-4a6a-a842-2b0fb7a75e83-home\") pod \"simulator-w5-c22dd8-kserve-66547dfc77-8bhmw\" (UID: \"b6d688b8-b911-4a6a-a842-2b0fb7a75e83\") " pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:19:23.970152 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:23.970127    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6d688b8-b911-4a6a-a842-2b0fb7a75e83-dshm\") pod \"simulator-w5-c22dd8-kserve-66547dfc77-8bhmw\" (UID: \"b6d688b8-b911-4a6a-a842-2b0fb7a75e83\") " pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:19:23.970397 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:23.970377    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6d688b8-b911-4a6a-a842-2b0fb7a75e83-tls-certs\") pod \"simulator-w5-c22dd8-kserve-66547dfc77-8bhmw\" (UID: \"b6d688b8-b911-4a6a-a842-2b0fb7a75e83\") " pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:19:24.029460 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:24.029382    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xltfx\" (UniqueName: \"kubernetes.io/projected/b6d688b8-b911-4a6a-a842-2b0fb7a75e83-kube-api-access-xltfx\") pod \"simulator-w5-c22dd8-kserve-66547dfc77-8bhmw\" (UID: \"b6d688b8-b911-4a6a-a842-2b0fb7a75e83\") " pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:19:24.085977 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:24.085949    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:19:24.462586 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:24.462541    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"]
Nov 01 03:19:24.463543 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:19:24.463520    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6d688b8_b911_4a6a_a842_2b0fb7a75e83.slice/crio-6991638945d170b5ed0f0d57cfa48e5471551477e1c2c425411864f4e39acfa5 WatchSource:0}: Error finding container 6991638945d170b5ed0f0d57cfa48e5471551477e1c2c425411864f4e39acfa5: Status 404 returned error can't find the container with id 6991638945d170b5ed0f0d57cfa48e5471551477e1c2c425411864f4e39acfa5
Nov 01 03:19:24.995865 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:24.995830    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:19:25.009204 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:25.008736    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:19:25.038729 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:25.038698    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw" event={"ID":"b6d688b8-b911-4a6a-a842-2b0fb7a75e83","Type":"ContainerStarted","Data":"b4f6bd1a6e51c6e7806faf07477a12394b9750967483c5ddfae03692cfeda71e"}
Nov 01 03:19:25.038891 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:25.038741    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw" event={"ID":"b6d688b8-b911-4a6a-a842-2b0fb7a75e83","Type":"ContainerStarted","Data":"6991638945d170b5ed0f0d57cfa48e5471551477e1c2c425411864f4e39acfa5"}
Nov 01 03:19:25.041709 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:25.041681    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:19:25.050407 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:25.050387    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:19:25.097233 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:25.097190    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:19:25.363911 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:25.363852    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw" podStartSLOduration=2.363835069 podStartE2EDuration="2.363835069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:19:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:19:25.362922405 +0000 UTC m=+769.885929007" watchObservedRunningTime="2026-11-01 03:19:25.363835069 +0000 UTC m=+769.886841658"
Nov 01 03:19:29.786634 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:29.786599    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"]
Nov 01 03:19:29.789415 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:29.789394    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"
Nov 01 03:19:29.844844 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:29.844680    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2511772-66f9-40a0-9166-87c4345485c4-tls-certs\") pod \"simulator-w0-ded2d7-kserve-75bd769b95-zsr4n\" (UID: \"d2511772-66f9-40a0-9166-87c4345485c4\") " pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"
Nov 01 03:19:29.844844 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:29.844789    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2511772-66f9-40a0-9166-87c4345485c4-model-cache\") pod \"simulator-w0-ded2d7-kserve-75bd769b95-zsr4n\" (UID: \"d2511772-66f9-40a0-9166-87c4345485c4\") " pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"
Nov 01 03:19:29.844844 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:29.844818    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7prf\" (UniqueName: \"kubernetes.io/projected/d2511772-66f9-40a0-9166-87c4345485c4-kube-api-access-d7prf\") pod \"simulator-w0-ded2d7-kserve-75bd769b95-zsr4n\" (UID: \"d2511772-66f9-40a0-9166-87c4345485c4\") " pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"
Nov 01 03:19:29.845088 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:29.844960    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2511772-66f9-40a0-9166-87c4345485c4-tmp-dir\") pod \"simulator-w0-ded2d7-kserve-75bd769b95-zsr4n\" (UID: \"d2511772-66f9-40a0-9166-87c4345485c4\") " pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"
Nov 01 03:19:29.845088 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:29.845057    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2511772-66f9-40a0-9166-87c4345485c4-home\") pod \"simulator-w0-ded2d7-kserve-75bd769b95-zsr4n\" (UID: \"d2511772-66f9-40a0-9166-87c4345485c4\") " pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"
Nov 01 03:19:29.845175 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:29.845159    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2511772-66f9-40a0-9166-87c4345485c4-dshm\") pod \"simulator-w0-ded2d7-kserve-75bd769b95-zsr4n\" (UID: \"d2511772-66f9-40a0-9166-87c4345485c4\") " pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"
Nov 01 03:19:29.870886 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:29.870851    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:19:29.889372 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:29.889335    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ded2d7\"/\"openshift-service-ca.crt\""
Nov 01 03:19:29.891647 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:29.891614    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ded2d7\"/\"kube-root-ca.crt\""
Nov 01 03:19:29.891799 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:29.891694    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ded2d7\"/\"default-dockercfg-pk9sl\""
Nov 01 03:19:29.891799 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:29.891758    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ded2d7\"/\"simulator-w0-ded2d7-kserve-self-signed-certs\""
Nov 01 03:19:29.897039 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:29.897011    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"]
Nov 01 03:19:29.946758 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:29.946727    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2511772-66f9-40a0-9166-87c4345485c4-tmp-dir\") pod \"simulator-w0-ded2d7-kserve-75bd769b95-zsr4n\" (UID: \"d2511772-66f9-40a0-9166-87c4345485c4\") " pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"
Nov 01 03:19:29.946758 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:29.946765    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2511772-66f9-40a0-9166-87c4345485c4-home\") pod \"simulator-w0-ded2d7-kserve-75bd769b95-zsr4n\" (UID: \"d2511772-66f9-40a0-9166-87c4345485c4\") " pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"
Nov 01 03:19:29.947017 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:29.946825    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2511772-66f9-40a0-9166-87c4345485c4-dshm\") pod \"simulator-w0-ded2d7-kserve-75bd769b95-zsr4n\" (UID: \"d2511772-66f9-40a0-9166-87c4345485c4\") " pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"
Nov 01 03:19:29.947017 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:29.946859    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2511772-66f9-40a0-9166-87c4345485c4-tls-certs\") pod \"simulator-w0-ded2d7-kserve-75bd769b95-zsr4n\" (UID: \"d2511772-66f9-40a0-9166-87c4345485c4\") " pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"
Nov 01 03:19:29.947017 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:29.946913    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2511772-66f9-40a0-9166-87c4345485c4-model-cache\") pod \"simulator-w0-ded2d7-kserve-75bd769b95-zsr4n\" (UID: \"d2511772-66f9-40a0-9166-87c4345485c4\") " pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"
Nov 01 03:19:29.947017 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:29.946948    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7prf\" (UniqueName: \"kubernetes.io/projected/d2511772-66f9-40a0-9166-87c4345485c4-kube-api-access-d7prf\") pod \"simulator-w0-ded2d7-kserve-75bd769b95-zsr4n\" (UID: \"d2511772-66f9-40a0-9166-87c4345485c4\") " pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"
Nov 01 03:19:29.947269 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:29.947232    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2511772-66f9-40a0-9166-87c4345485c4-home\") pod \"simulator-w0-ded2d7-kserve-75bd769b95-zsr4n\" (UID: \"d2511772-66f9-40a0-9166-87c4345485c4\") " pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"
Nov 01 03:19:29.947269 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:29.947261    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2511772-66f9-40a0-9166-87c4345485c4-tmp-dir\") pod \"simulator-w0-ded2d7-kserve-75bd769b95-zsr4n\" (UID: \"d2511772-66f9-40a0-9166-87c4345485c4\") " pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"
Nov 01 03:19:29.947367 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:29.947337    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2511772-66f9-40a0-9166-87c4345485c4-model-cache\") pod \"simulator-w0-ded2d7-kserve-75bd769b95-zsr4n\" (UID: \"d2511772-66f9-40a0-9166-87c4345485c4\") " pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"
Nov 01 03:19:29.949026 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:29.948994    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2511772-66f9-40a0-9166-87c4345485c4-dshm\") pod \"simulator-w0-ded2d7-kserve-75bd769b95-zsr4n\" (UID: \"d2511772-66f9-40a0-9166-87c4345485c4\") " pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"
Nov 01 03:19:29.949301 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:29.949282    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2511772-66f9-40a0-9166-87c4345485c4-tls-certs\") pod \"simulator-w0-ded2d7-kserve-75bd769b95-zsr4n\" (UID: \"d2511772-66f9-40a0-9166-87c4345485c4\") " pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"
Nov 01 03:19:30.014874 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:30.014830    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7prf\" (UniqueName: \"kubernetes.io/projected/d2511772-66f9-40a0-9166-87c4345485c4-kube-api-access-d7prf\") pod \"simulator-w0-ded2d7-kserve-75bd769b95-zsr4n\" (UID: \"d2511772-66f9-40a0-9166-87c4345485c4\") " pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"
Nov 01 03:19:30.103844 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:30.103816    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"
Nov 01 03:19:30.123980 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:30.123421    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:19:30.490093 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:30.489927    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"]
Nov 01 03:19:30.492409 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:19:30.492383    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2511772_66f9_40a0_9166_87c4345485c4.slice/crio-f26e9cd1cce9ce8b6ef92eb6b9afa19cee0d25ab41b85732827b6b5bfe8566a8 WatchSource:0}: Error finding container f26e9cd1cce9ce8b6ef92eb6b9afa19cee0d25ab41b85732827b6b5bfe8566a8: Status 404 returned error can't find the container with id f26e9cd1cce9ce8b6ef92eb6b9afa19cee0d25ab41b85732827b6b5bfe8566a8
Nov 01 03:19:31.078821 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:31.078770    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n" event={"ID":"d2511772-66f9-40a0-9166-87c4345485c4","Type":"ContainerStarted","Data":"b972c0b7cd229c628519bb99f3aae0c1c83a5158577036c8ce6bb4401512fbff"}
Nov 01 03:19:31.079200 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:31.078839    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n" event={"ID":"d2511772-66f9-40a0-9166-87c4345485c4","Type":"ContainerStarted","Data":"f26e9cd1cce9ce8b6ef92eb6b9afa19cee0d25ab41b85732827b6b5bfe8566a8"}
Nov 01 03:19:31.104326 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:31.104298    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"
Nov 01 03:19:31.196687 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:31.196638    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n" podStartSLOduration=2.196624655 podStartE2EDuration="2.196624655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:19:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:19:31.195894985 +0000 UTC m=+775.718901572" watchObservedRunningTime="2026-11-01 03:19:31.196624655 +0000 UTC m=+775.719631243"
Nov 01 03:19:34.086787 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:34.086516    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:19:34.099588 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:34.099546    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:19:34.110670 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:34.110644    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:19:34.742060 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:34.742024    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:19:39.539013 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:39.538977    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:19:39.998656 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:39.998618    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:19:40.104964 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:40.104932    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"
Nov 01 03:19:40.117479 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:40.117456    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"
Nov 01 03:19:40.147653 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:40.147622    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"
Nov 01 03:19:42.971427 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:42.971390    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:19:44.365364 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:44.365317    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:19:49.123462 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:19:49.123426    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:19:49.138971 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:49.138935    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:19:53.950469 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:53.950435    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:19:54.823777 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:54.823723    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:19:58.851775 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:19:58.851745    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:20:03.650475 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:03.650433    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:20:03.652011 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:03.651986    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"]
Nov 01 03:20:03.654920 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:03.654902    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:20:03.709043 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:03.709010    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40881383-61bb-4fc8-9c75-f8ffce4834c9-home\") pod \"simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf\" (UID: \"40881383-61bb-4fc8-9c75-f8ffce4834c9\") " pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:20:03.709210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:03.709073    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc7gk\" (UniqueName: \"kubernetes.io/projected/40881383-61bb-4fc8-9c75-f8ffce4834c9-kube-api-access-jc7gk\") pod \"simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf\" (UID: \"40881383-61bb-4fc8-9c75-f8ffce4834c9\") " pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:20:03.709210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:03.709113    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40881383-61bb-4fc8-9c75-f8ffce4834c9-tmp-dir\") pod \"simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf\" (UID: \"40881383-61bb-4fc8-9c75-f8ffce4834c9\") " pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:20:03.709210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:03.709149    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40881383-61bb-4fc8-9c75-f8ffce4834c9-dshm\") pod \"simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf\" (UID: \"40881383-61bb-4fc8-9c75-f8ffce4834c9\") " pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:20:03.709424 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:03.709399    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40881383-61bb-4fc8-9c75-f8ffce4834c9-model-cache\") pod \"simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf\" (UID: \"40881383-61bb-4fc8-9c75-f8ffce4834c9\") " pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:20:03.709483 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:03.709439    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40881383-61bb-4fc8-9c75-f8ffce4834c9-tls-certs\") pod \"simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf\" (UID: \"40881383-61bb-4fc8-9c75-f8ffce4834c9\") " pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:20:03.756895 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:03.756862    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-c8cb9a\"/\"openshift-service-ca.crt\""
Nov 01 03:20:03.757072 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:03.756977    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-c8cb9a\"/\"default-dockercfg-dxgc9\""
Nov 01 03:20:03.757072 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:03.756992    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-c8cb9a\"/\"kube-root-ca.crt\""
Nov 01 03:20:03.757072 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:03.757035    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-c8cb9a\"/\"simulator-w2-c8cb9a-kserve-self-signed-certs\""
Nov 01 03:20:03.760750 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:03.760726    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"]
Nov 01 03:20:03.804093 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:03.804057    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:20:03.812483 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:03.812442    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40881383-61bb-4fc8-9c75-f8ffce4834c9-home\") pod \"simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf\" (UID: \"40881383-61bb-4fc8-9c75-f8ffce4834c9\") " pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:20:03.812753 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:03.812541    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc7gk\" (UniqueName: \"kubernetes.io/projected/40881383-61bb-4fc8-9c75-f8ffce4834c9-kube-api-access-jc7gk\") pod \"simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf\" (UID: \"40881383-61bb-4fc8-9c75-f8ffce4834c9\") " pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:20:03.812753 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:03.812644    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40881383-61bb-4fc8-9c75-f8ffce4834c9-tmp-dir\") pod \"simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf\" (UID: \"40881383-61bb-4fc8-9c75-f8ffce4834c9\") " pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:20:03.812753 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:03.812676    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40881383-61bb-4fc8-9c75-f8ffce4834c9-dshm\") pod \"simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf\" (UID: \"40881383-61bb-4fc8-9c75-f8ffce4834c9\") " pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:20:03.812923 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:03.812758    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40881383-61bb-4fc8-9c75-f8ffce4834c9-model-cache\") pod \"simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf\" (UID: \"40881383-61bb-4fc8-9c75-f8ffce4834c9\") " pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:20:03.812923 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:03.812783    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40881383-61bb-4fc8-9c75-f8ffce4834c9-tls-certs\") pod \"simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf\" (UID: \"40881383-61bb-4fc8-9c75-f8ffce4834c9\") " pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:20:03.813689 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:03.813663    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40881383-61bb-4fc8-9c75-f8ffce4834c9-tmp-dir\") pod \"simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf\" (UID: \"40881383-61bb-4fc8-9c75-f8ffce4834c9\") " pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:20:03.814451 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:03.814426    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40881383-61bb-4fc8-9c75-f8ffce4834c9-home\") pod \"simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf\" (UID: \"40881383-61bb-4fc8-9c75-f8ffce4834c9\") " pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:20:03.815247 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:03.815224    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40881383-61bb-4fc8-9c75-f8ffce4834c9-model-cache\") pod \"simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf\" (UID: \"40881383-61bb-4fc8-9c75-f8ffce4834c9\") " pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:20:03.820937 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:03.820903    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40881383-61bb-4fc8-9c75-f8ffce4834c9-tls-certs\") pod \"simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf\" (UID: \"40881383-61bb-4fc8-9c75-f8ffce4834c9\") " pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:20:03.821837 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:03.821817    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40881383-61bb-4fc8-9c75-f8ffce4834c9-dshm\") pod \"simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf\" (UID: \"40881383-61bb-4fc8-9c75-f8ffce4834c9\") " pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:20:03.875825 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:03.875800    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc7gk\" (UniqueName: \"kubernetes.io/projected/40881383-61bb-4fc8-9c75-f8ffce4834c9-kube-api-access-jc7gk\") pod \"simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf\" (UID: \"40881383-61bb-4fc8-9c75-f8ffce4834c9\") " pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:20:03.966251 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:03.966161    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:20:04.351240 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:04.351212    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"]
Nov 01 03:20:04.352430 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:20:04.352398    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40881383_61bb_4fc8_9c75_f8ffce4834c9.slice/crio-f6c5bb10d0c66dd0631c674a2b89f53b2a32c5c0e2e6eb5ef90a90d2189d51bc WatchSource:0}: Error finding container f6c5bb10d0c66dd0631c674a2b89f53b2a32c5c0e2e6eb5ef90a90d2189d51bc: Status 404 returned error can't find the container with id f6c5bb10d0c66dd0631c674a2b89f53b2a32c5c0e2e6eb5ef90a90d2189d51bc
Nov 01 03:20:04.808754 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:04.808682    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3efb826a-g5vr2"]
Nov 01 03:20:04.811254 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:04.811239    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efb826a-g5vr2"
Nov 01 03:20:04.851385 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:04.851355    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ffd51c9-c42a-4168-96ae-2a596c836aab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3efb826a-g5vr2\" (UID: \"7ffd51c9-c42a-4168-96ae-2a596c836aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efb826a-g5vr2"
Nov 01 03:20:04.851550 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:04.851527    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hrj8\" (UniqueName: \"kubernetes.io/projected/7ffd51c9-c42a-4168-96ae-2a596c836aab-kube-api-access-2hrj8\") pod \"maas-api-revoke-sub-3efb826a-g5vr2\" (UID: \"7ffd51c9-c42a-4168-96ae-2a596c836aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efb826a-g5vr2"
Nov 01 03:20:04.851685 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:04.851633    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ffd51c9-c42a-4168-96ae-2a596c836aab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3efb826a-g5vr2\" (UID: \"7ffd51c9-c42a-4168-96ae-2a596c836aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efb826a-g5vr2"
Nov 01 03:20:04.863785 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:04.863757    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:20:04.863891 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:04.863799    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:20:04.863954 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:04.863919    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:20:04.863954 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:04.863942    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:20:04.871233 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:04.871208    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3efb826a-g5vr2"]
Nov 01 03:20:04.952851 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:04.952822    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hrj8\" (UniqueName: \"kubernetes.io/projected/7ffd51c9-c42a-4168-96ae-2a596c836aab-kube-api-access-2hrj8\") pod \"maas-api-revoke-sub-3efb826a-g5vr2\" (UID: \"7ffd51c9-c42a-4168-96ae-2a596c836aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efb826a-g5vr2"
Nov 01 03:20:04.953020 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:04.952868    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ffd51c9-c42a-4168-96ae-2a596c836aab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3efb826a-g5vr2\" (UID: \"7ffd51c9-c42a-4168-96ae-2a596c836aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efb826a-g5vr2"
Nov 01 03:20:04.953020 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:04.952930    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ffd51c9-c42a-4168-96ae-2a596c836aab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3efb826a-g5vr2\" (UID: \"7ffd51c9-c42a-4168-96ae-2a596c836aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efb826a-g5vr2"
Nov 01 03:20:04.953622 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:04.953593    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ffd51c9-c42a-4168-96ae-2a596c836aab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3efb826a-g5vr2\" (UID: \"7ffd51c9-c42a-4168-96ae-2a596c836aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efb826a-g5vr2"
Nov 01 03:20:04.955478 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:04.955456    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ffd51c9-c42a-4168-96ae-2a596c836aab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3efb826a-g5vr2\" (UID: \"7ffd51c9-c42a-4168-96ae-2a596c836aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efb826a-g5vr2"
Nov 01 03:20:05.013614 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:05.013589    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hrj8\" (UniqueName: \"kubernetes.io/projected/7ffd51c9-c42a-4168-96ae-2a596c836aab-kube-api-access-2hrj8\") pod \"maas-api-revoke-sub-3efb826a-g5vr2\" (UID: \"7ffd51c9-c42a-4168-96ae-2a596c836aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efb826a-g5vr2"
Nov 01 03:20:05.133962 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:05.133926    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efb826a-g5vr2"
Nov 01 03:20:05.300635 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:05.300592    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf" event={"ID":"40881383-61bb-4fc8-9c75-f8ffce4834c9","Type":"ContainerStarted","Data":"c17ddece108cfda9bd3fd88e744de128173b6f42f4b663ef7ffe3d972c58daef"}
Nov 01 03:20:05.300798 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:05.300650    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf" event={"ID":"40881383-61bb-4fc8-9c75-f8ffce4834c9","Type":"ContainerStarted","Data":"f6c5bb10d0c66dd0631c674a2b89f53b2a32c5c0e2e6eb5ef90a90d2189d51bc"}
Nov 01 03:20:05.419730 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:05.419626    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf" podStartSLOduration=2.419612061 podStartE2EDuration="2.419612061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:20:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:20:05.417783851 +0000 UTC m=+809.940790439" watchObservedRunningTime="2026-11-01 03:20:05.419612061 +0000 UTC m=+809.942618688"
Nov 01 03:20:05.533533 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:05.533500    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3efb826a-g5vr2"]
Nov 01 03:20:05.966266 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:05.966241    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:20:06.307915 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:06.307878    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efb826a-g5vr2" event={"ID":"7ffd51c9-c42a-4168-96ae-2a596c836aab","Type":"ContainerStarted","Data":"d36c050b48c6294efbac6821cbff14e708a028d13888617f07533a3caa4ae7fa"}
Nov 01 03:20:06.308183 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:06.307960    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efb826a-g5vr2" event={"ID":"7ffd51c9-c42a-4168-96ae-2a596c836aab","Type":"ContainerStarted","Data":"e6a5d589b75a31a930bf2dc2b5f2f378ba80a48c913df3d0b73ee52abdfd17a8"}
Nov 01 03:20:06.426538 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:06.426489    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efb826a-g5vr2" podStartSLOduration=2.426475485 podStartE2EDuration="2.426475485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:20:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:20:06.423316534 +0000 UTC m=+810.946323139" watchObservedRunningTime="2026-11-01 03:20:06.426475485 +0000 UTC m=+810.949482073"
Nov 01 03:20:07.315543 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:07.315505    4930 generic.go:356] "Generic (PLEG): container finished" podID="7ffd51c9-c42a-4168-96ae-2a596c836aab" containerID="d36c050b48c6294efbac6821cbff14e708a028d13888617f07533a3caa4ae7fa" exitCode=0
Nov 01 03:20:07.315930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:07.315588    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efb826a-g5vr2" event={"ID":"7ffd51c9-c42a-4168-96ae-2a596c836aab","Type":"ContainerDied","Data":"d36c050b48c6294efbac6821cbff14e708a028d13888617f07533a3caa4ae7fa"}
Nov 01 03:20:08.455954 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:08.455915    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:20:08.462724 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:08.462703    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efb826a-g5vr2"
Nov 01 03:20:08.492376 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:08.492339    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ffd51c9-c42a-4168-96ae-2a596c836aab-kube-api-access-2hrj8\" (UniqueName: \"kubernetes.io/projected/7ffd51c9-c42a-4168-96ae-2a596c836aab-kube-api-access-2hrj8\") pod \"7ffd51c9-c42a-4168-96ae-2a596c836aab\" (UID: \"7ffd51c9-c42a-4168-96ae-2a596c836aab\") "
Nov 01 03:20:08.492573 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:08.492508    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ffd51c9-c42a-4168-96ae-2a596c836aab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ffd51c9-c42a-4168-96ae-2a596c836aab-maas-api-xks-ca\") pod \"7ffd51c9-c42a-4168-96ae-2a596c836aab\" (UID: \"7ffd51c9-c42a-4168-96ae-2a596c836aab\") "
Nov 01 03:20:08.492872 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:08.492656    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ffd51c9-c42a-4168-96ae-2a596c836aab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ffd51c9-c42a-4168-96ae-2a596c836aab-maas-api-openshift-ca\") pod \"7ffd51c9-c42a-4168-96ae-2a596c836aab\" (UID: \"7ffd51c9-c42a-4168-96ae-2a596c836aab\") "
Nov 01 03:20:08.493306 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:08.493280    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ffd51c9-c42a-4168-96ae-2a596c836aab-maas-api-openshift-ca" pod "7ffd51c9-c42a-4168-96ae-2a596c836aab" (UID: "7ffd51c9-c42a-4168-96ae-2a596c836aab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:20:08.497668 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:08.495934    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ffd51c9-c42a-4168-96ae-2a596c836aab-maas-api-xks-ca" pod "7ffd51c9-c42a-4168-96ae-2a596c836aab" (UID: "7ffd51c9-c42a-4168-96ae-2a596c836aab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:20:08.497668 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:08.497640    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ffd51c9-c42a-4168-96ae-2a596c836aab-kube-api-access-2hrj8" pod "7ffd51c9-c42a-4168-96ae-2a596c836aab" (UID: "7ffd51c9-c42a-4168-96ae-2a596c836aab"). InnerVolumeSpecName "kube-api-access-2hrj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:20:08.594645 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:08.594615    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ffd51c9-c42a-4168-96ae-2a596c836aab-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:20:08.594645 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:08.594644    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hrj8\" (UniqueName: \"kubernetes.io/projected/7ffd51c9-c42a-4168-96ae-2a596c836aab-kube-api-access-2hrj8\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:20:08.594858 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:08.594655    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ffd51c9-c42a-4168-96ae-2a596c836aab-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:20:09.328420 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:09.328388    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efb826a-g5vr2"
Nov 01 03:20:09.328671 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:09.328418    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3efb826a-g5vr2" event={"ID":"7ffd51c9-c42a-4168-96ae-2a596c836aab","Type":"ContainerDied","Data":"e6a5d589b75a31a930bf2dc2b5f2f378ba80a48c913df3d0b73ee52abdfd17a8"}
Nov 01 03:20:09.328671 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:09.328469    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6a5d589b75a31a930bf2dc2b5f2f378ba80a48c913df3d0b73ee52abdfd17a8"
Nov 01 03:20:13.365143 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:13.365092    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:20:13.967581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:13.966743    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:20:13.980691 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:13.980662    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:20:14.374430 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:14.374401    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:20:16.749288 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:16.749245    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"]
Nov 01 03:20:16.749810 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:16.749795    4930 cpu_manager.go:414] "containerMap: removing container" podUID="7ffd51c9-c42a-4168-96ae-2a596c836aab" containerName="revoke-keys"
Nov 01 03:20:16.749853 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:16.749811    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ffd51c9-c42a-4168-96ae-2a596c836aab" containerName="revoke-keys"
Nov 01 03:20:16.749914 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:16.749900    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ffd51c9-c42a-4168-96ae-2a596c836aab" containerName="revoke-keys"
Nov 01 03:20:16.752259 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:16.752239    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:20:16.844543 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:16.844506    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:20:16.855538 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:16.855511    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-19faa5\"/\"premium-w6-19faa5-kserve-self-signed-certs\""
Nov 01 03:20:16.858447 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:16.858422    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"]
Nov 01 03:20:16.895535 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:16.895220    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d658772c-b072-4513-a639-d19a94c78525-tls-certs\") pod \"premium-w6-19faa5-kserve-7b6997745f-8wzps\" (UID: \"d658772c-b072-4513-a639-d19a94c78525\") " pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:20:16.897017 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:16.896946    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d658772c-b072-4513-a639-d19a94c78525-home\") pod \"premium-w6-19faa5-kserve-7b6997745f-8wzps\" (UID: \"d658772c-b072-4513-a639-d19a94c78525\") " pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:20:16.897304 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:16.897258    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d658772c-b072-4513-a639-d19a94c78525-model-cache\") pod \"premium-w6-19faa5-kserve-7b6997745f-8wzps\" (UID: \"d658772c-b072-4513-a639-d19a94c78525\") " pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:20:16.901994 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:16.901968    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d658772c-b072-4513-a639-d19a94c78525-tmp-dir\") pod \"premium-w6-19faa5-kserve-7b6997745f-8wzps\" (UID: \"d658772c-b072-4513-a639-d19a94c78525\") " pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:20:16.902301 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:16.902279    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d658772c-b072-4513-a639-d19a94c78525-dshm\") pod \"premium-w6-19faa5-kserve-7b6997745f-8wzps\" (UID: \"d658772c-b072-4513-a639-d19a94c78525\") " pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:20:16.902803 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:16.902783    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g8l4\" (UniqueName: \"kubernetes.io/projected/d658772c-b072-4513-a639-d19a94c78525-kube-api-access-4g8l4\") pod \"premium-w6-19faa5-kserve-7b6997745f-8wzps\" (UID: \"d658772c-b072-4513-a639-d19a94c78525\") " pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:20:17.004632 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:17.004532    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d658772c-b072-4513-a639-d19a94c78525-model-cache\") pod \"premium-w6-19faa5-kserve-7b6997745f-8wzps\" (UID: \"d658772c-b072-4513-a639-d19a94c78525\") " pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:20:17.004821 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:17.004673    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d658772c-b072-4513-a639-d19a94c78525-tmp-dir\") pod \"premium-w6-19faa5-kserve-7b6997745f-8wzps\" (UID: \"d658772c-b072-4513-a639-d19a94c78525\") " pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:20:17.004821 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:17.004698    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d658772c-b072-4513-a639-d19a94c78525-dshm\") pod \"premium-w6-19faa5-kserve-7b6997745f-8wzps\" (UID: \"d658772c-b072-4513-a639-d19a94c78525\") " pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:20:17.004821 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:17.004733    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g8l4\" (UniqueName: \"kubernetes.io/projected/d658772c-b072-4513-a639-d19a94c78525-kube-api-access-4g8l4\") pod \"premium-w6-19faa5-kserve-7b6997745f-8wzps\" (UID: \"d658772c-b072-4513-a639-d19a94c78525\") " pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:20:17.004821 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:17.004798    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d658772c-b072-4513-a639-d19a94c78525-tls-certs\") pod \"premium-w6-19faa5-kserve-7b6997745f-8wzps\" (UID: \"d658772c-b072-4513-a639-d19a94c78525\") " pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:20:17.005023 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:17.004835    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d658772c-b072-4513-a639-d19a94c78525-home\") pod \"premium-w6-19faa5-kserve-7b6997745f-8wzps\" (UID: \"d658772c-b072-4513-a639-d19a94c78525\") " pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:20:17.005023 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:17.004955    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d658772c-b072-4513-a639-d19a94c78525-model-cache\") pod \"premium-w6-19faa5-kserve-7b6997745f-8wzps\" (UID: \"d658772c-b072-4513-a639-d19a94c78525\") " pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:20:17.005240 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:17.005201    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d658772c-b072-4513-a639-d19a94c78525-home\") pod \"premium-w6-19faa5-kserve-7b6997745f-8wzps\" (UID: \"d658772c-b072-4513-a639-d19a94c78525\") " pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:20:17.005493 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:17.005469    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d658772c-b072-4513-a639-d19a94c78525-tmp-dir\") pod \"premium-w6-19faa5-kserve-7b6997745f-8wzps\" (UID: \"d658772c-b072-4513-a639-d19a94c78525\") " pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:20:17.006974 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:17.006952    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d658772c-b072-4513-a639-d19a94c78525-dshm\") pod \"premium-w6-19faa5-kserve-7b6997745f-8wzps\" (UID: \"d658772c-b072-4513-a639-d19a94c78525\") " pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:20:17.007354 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:17.007331    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d658772c-b072-4513-a639-d19a94c78525-tls-certs\") pod \"premium-w6-19faa5-kserve-7b6997745f-8wzps\" (UID: \"d658772c-b072-4513-a639-d19a94c78525\") " pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:20:17.065459 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:17.065428    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g8l4\" (UniqueName: \"kubernetes.io/projected/d658772c-b072-4513-a639-d19a94c78525-kube-api-access-4g8l4\") pod \"premium-w6-19faa5-kserve-7b6997745f-8wzps\" (UID: \"d658772c-b072-4513-a639-d19a94c78525\") " pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:20:17.363614 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:17.363583    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:20:17.739303 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:17.739272    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"]
Nov 01 03:20:17.740737 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:20:17.740707    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd658772c_b072_4513_a639_d19a94c78525.slice/crio-d2c9a6aa6b69199b7c19744bc2ac0e2a85cc643a1947ad310ab8b99a2712182b WatchSource:0}: Error finding container d2c9a6aa6b69199b7c19744bc2ac0e2a85cc643a1947ad310ab8b99a2712182b: Status 404 returned error can't find the container with id d2c9a6aa6b69199b7c19744bc2ac0e2a85cc643a1947ad310ab8b99a2712182b
Nov 01 03:20:18.052611 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:18.052517    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:20:18.389072 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:18.389037    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps" event={"ID":"d658772c-b072-4513-a639-d19a94c78525","Type":"ContainerStarted","Data":"fbcf34847caf605aa43a2ad2d119faf20444fecc1f47218da73f3b077f32abea"}
Nov 01 03:20:18.389238 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:18.389080    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps" event={"ID":"d658772c-b072-4513-a639-d19a94c78525","Type":"ContainerStarted","Data":"d2c9a6aa6b69199b7c19744bc2ac0e2a85cc643a1947ad310ab8b99a2712182b"}
Nov 01 03:20:18.511196 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:18.511129    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps" podStartSLOduration=2.5111137279999998 podStartE2EDuration="2.511113728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:20:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:20:18.510095565 +0000 UTC m=+823.033102153" watchObservedRunningTime="2026-11-01 03:20:18.511113728 +0000 UTC m=+823.034120318"
Nov 01 03:20:19.123900 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:20:19.123863    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:20:19.363820 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:19.363790    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:20:22.884415 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:22.884374    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:20:27.363768 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:27.363735    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:20:27.376786 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:27.376760    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:20:27.459917 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:27.459892    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:20:27.633628 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:27.633252    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:20:30.212173 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:30.212130    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:20:32.379602 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:32.379550    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:20:35.383099 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:35.383063    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28a6d1f1-zsn8r"]
Nov 01 03:20:35.387140 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:35.387123    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a6d1f1-zsn8r"
Nov 01 03:20:35.440341 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:35.440317    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:20:35.489755 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:35.489722    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:20:35.489755 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:35.489751    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:20:35.489983 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:35.489799    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:20:35.493118 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:35.493089    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28a6d1f1-zsn8r"]
Nov 01 03:20:35.521645 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:35.521621    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pwdf\" (UniqueName: \"kubernetes.io/projected/fbd0fcba-9a80-42ce-abff-fee5fe08b065-kube-api-access-7pwdf\") pod \"maas-api-revoke-sub-28a6d1f1-zsn8r\" (UID: \"fbd0fcba-9a80-42ce-abff-fee5fe08b065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a6d1f1-zsn8r"
Nov 01 03:20:35.521770 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:35.521681    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbd0fcba-9a80-42ce-abff-fee5fe08b065-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28a6d1f1-zsn8r\" (UID: \"fbd0fcba-9a80-42ce-abff-fee5fe08b065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a6d1f1-zsn8r"
Nov 01 03:20:35.521833 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:35.521813    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbd0fcba-9a80-42ce-abff-fee5fe08b065-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28a6d1f1-zsn8r\" (UID: \"fbd0fcba-9a80-42ce-abff-fee5fe08b065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a6d1f1-zsn8r"
Nov 01 03:20:35.622858 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:35.622829    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbd0fcba-9a80-42ce-abff-fee5fe08b065-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28a6d1f1-zsn8r\" (UID: \"fbd0fcba-9a80-42ce-abff-fee5fe08b065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a6d1f1-zsn8r"
Nov 01 03:20:35.623016 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:35.622895    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbd0fcba-9a80-42ce-abff-fee5fe08b065-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28a6d1f1-zsn8r\" (UID: \"fbd0fcba-9a80-42ce-abff-fee5fe08b065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a6d1f1-zsn8r"
Nov 01 03:20:35.623080 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:35.623030    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pwdf\" (UniqueName: \"kubernetes.io/projected/fbd0fcba-9a80-42ce-abff-fee5fe08b065-kube-api-access-7pwdf\") pod \"maas-api-revoke-sub-28a6d1f1-zsn8r\" (UID: \"fbd0fcba-9a80-42ce-abff-fee5fe08b065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a6d1f1-zsn8r"
Nov 01 03:20:35.674414 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:35.674338    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:20:35.674414 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:35.674347    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:20:35.684363 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:35.684336    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbd0fcba-9a80-42ce-abff-fee5fe08b065-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28a6d1f1-zsn8r\" (UID: \"fbd0fcba-9a80-42ce-abff-fee5fe08b065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a6d1f1-zsn8r"
Nov 01 03:20:35.686076 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:35.686054    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbd0fcba-9a80-42ce-abff-fee5fe08b065-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28a6d1f1-zsn8r\" (UID: \"fbd0fcba-9a80-42ce-abff-fee5fe08b065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a6d1f1-zsn8r"
Nov 01 03:20:35.733138 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:35.733116    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:20:35.744930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:35.744904    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pwdf\" (UniqueName: \"kubernetes.io/projected/fbd0fcba-9a80-42ce-abff-fee5fe08b065-kube-api-access-7pwdf\") pod \"maas-api-revoke-sub-28a6d1f1-zsn8r\" (UID: \"fbd0fcba-9a80-42ce-abff-fee5fe08b065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a6d1f1-zsn8r"
Nov 01 03:20:36.052175 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:36.052097    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:20:36.061653 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:36.061633    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a6d1f1-zsn8r"
Nov 01 03:20:36.440018 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:36.439984    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28a6d1f1-zsn8r"]
Nov 01 03:20:36.440873 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:20:36.440844    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbd0fcba_9a80_42ce_abff_fee5fe08b065.slice/crio-9cdbb6031ed09d6e7ab1ff1db19b9b70e076333a93d4d3e2fb4d4f0b68674e70 WatchSource:0}: Error finding container 9cdbb6031ed09d6e7ab1ff1db19b9b70e076333a93d4d3e2fb4d4f0b68674e70: Status 404 returned error can't find the container with id 9cdbb6031ed09d6e7ab1ff1db19b9b70e076333a93d4d3e2fb4d4f0b68674e70
Nov 01 03:20:36.504408 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:36.504382    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a6d1f1-zsn8r" event={"ID":"fbd0fcba-9a80-42ce-abff-fee5fe08b065","Type":"ContainerStarted","Data":"9cdbb6031ed09d6e7ab1ff1db19b9b70e076333a93d4d3e2fb4d4f0b68674e70"}
Nov 01 03:20:37.246340 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:37.246300    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:20:37.513864 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:37.513774    4930 generic.go:356] "Generic (PLEG): container finished" podID="fbd0fcba-9a80-42ce-abff-fee5fe08b065" containerID="f5228339a7b58360f4a9cfbb872e38f56983eaf8d17cc00a2f340df45d9d4e2b" exitCode=0
Nov 01 03:20:37.514306 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:37.513855    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a6d1f1-zsn8r" event={"ID":"fbd0fcba-9a80-42ce-abff-fee5fe08b065","Type":"ContainerDied","Data":"f5228339a7b58360f4a9cfbb872e38f56983eaf8d17cc00a2f340df45d9d4e2b"}
Nov 01 03:20:38.665022 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:38.664994    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a6d1f1-zsn8r"
Nov 01 03:20:38.759441 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:38.759413    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbd0fcba-9a80-42ce-abff-fee5fe08b065-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbd0fcba-9a80-42ce-abff-fee5fe08b065-maas-api-xks-ca\") pod \"fbd0fcba-9a80-42ce-abff-fee5fe08b065\" (UID: \"fbd0fcba-9a80-42ce-abff-fee5fe08b065\") "
Nov 01 03:20:38.759617 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:38.759537    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbd0fcba-9a80-42ce-abff-fee5fe08b065-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbd0fcba-9a80-42ce-abff-fee5fe08b065-maas-api-openshift-ca\") pod \"fbd0fcba-9a80-42ce-abff-fee5fe08b065\" (UID: \"fbd0fcba-9a80-42ce-abff-fee5fe08b065\") "
Nov 01 03:20:38.759617 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:38.759608    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbd0fcba-9a80-42ce-abff-fee5fe08b065-kube-api-access-7pwdf\" (UniqueName: \"kubernetes.io/projected/fbd0fcba-9a80-42ce-abff-fee5fe08b065-kube-api-access-7pwdf\") pod \"fbd0fcba-9a80-42ce-abff-fee5fe08b065\" (UID: \"fbd0fcba-9a80-42ce-abff-fee5fe08b065\") "
Nov 01 03:20:38.759850 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:38.759824    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbd0fcba-9a80-42ce-abff-fee5fe08b065-maas-api-openshift-ca" pod "fbd0fcba-9a80-42ce-abff-fee5fe08b065" (UID: "fbd0fcba-9a80-42ce-abff-fee5fe08b065"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:20:38.760068 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:38.760049    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbd0fcba-9a80-42ce-abff-fee5fe08b065-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:20:38.761326 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:38.761306    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbd0fcba-9a80-42ce-abff-fee5fe08b065-maas-api-xks-ca" pod "fbd0fcba-9a80-42ce-abff-fee5fe08b065" (UID: "fbd0fcba-9a80-42ce-abff-fee5fe08b065"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:20:38.761518 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:38.761498    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbd0fcba-9a80-42ce-abff-fee5fe08b065-kube-api-access-7pwdf" pod "fbd0fcba-9a80-42ce-abff-fee5fe08b065" (UID: "fbd0fcba-9a80-42ce-abff-fee5fe08b065"). InnerVolumeSpecName "kube-api-access-7pwdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:20:38.860928 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:38.860897    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pwdf\" (UniqueName: \"kubernetes.io/projected/fbd0fcba-9a80-42ce-abff-fee5fe08b065-kube-api-access-7pwdf\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:20:38.860928 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:38.860924    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbd0fcba-9a80-42ce-abff-fee5fe08b065-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:20:39.527498 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:39.527465    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a6d1f1-zsn8r"
Nov 01 03:20:39.527702 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:39.527507    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a6d1f1-zsn8r" event={"ID":"fbd0fcba-9a80-42ce-abff-fee5fe08b065","Type":"ContainerDied","Data":"9cdbb6031ed09d6e7ab1ff1db19b9b70e076333a93d4d3e2fb4d4f0b68674e70"}
Nov 01 03:20:39.527702 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:39.527582    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9cdbb6031ed09d6e7ab1ff1db19b9b70e076333a93d4d3e2fb4d4f0b68674e70"
Nov 01 03:20:40.357045 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:40.357005    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:20:42.059411 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:42.059373    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:20:46.190093 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:46.190048    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:20:46.848677 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:46.848586    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:20:48.694780 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:48.694747    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:20:49.122048 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:20:49.122007    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:20:50.727647 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:50.727611    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69b8ffed-9nl8q"]
Nov 01 03:20:50.728124 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:50.728109    4930 cpu_manager.go:414] "containerMap: removing container" podUID="fbd0fcba-9a80-42ce-abff-fee5fe08b065" containerName="revoke-keys"
Nov 01 03:20:50.728163 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:50.728125    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbd0fcba-9a80-42ce-abff-fee5fe08b065" containerName="revoke-keys"
Nov 01 03:20:50.728243 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:50.728234    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbd0fcba-9a80-42ce-abff-fee5fe08b065" containerName="revoke-keys"
Nov 01 03:20:50.732710 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:50.732690    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b8ffed-9nl8q"
Nov 01 03:20:50.787098 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:50.787065    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:20:50.790730 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:50.790701    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69b8ffed-9nl8q"]
Nov 01 03:20:50.810691 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:50.810665    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6398465-9388-479e-85c9-b44b1797deff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69b8ffed-9nl8q\" (UID: \"b6398465-9388-479e-85c9-b44b1797deff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b8ffed-9nl8q"
Nov 01 03:20:50.810811 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:50.810784    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6398465-9388-479e-85c9-b44b1797deff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69b8ffed-9nl8q\" (UID: \"b6398465-9388-479e-85c9-b44b1797deff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b8ffed-9nl8q"
Nov 01 03:20:50.810941 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:50.810919    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crlvv\" (UniqueName: \"kubernetes.io/projected/b6398465-9388-479e-85c9-b44b1797deff-kube-api-access-crlvv\") pod \"maas-api-revoke-sub-69b8ffed-9nl8q\" (UID: \"b6398465-9388-479e-85c9-b44b1797deff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b8ffed-9nl8q"
Nov 01 03:20:50.836674 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:50.836645    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:20:50.836777 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:50.836671    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:20:50.836777 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:50.836719    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:20:50.912076 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:50.912050    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6398465-9388-479e-85c9-b44b1797deff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69b8ffed-9nl8q\" (UID: \"b6398465-9388-479e-85c9-b44b1797deff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b8ffed-9nl8q"
Nov 01 03:20:50.912228 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:50.912168    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crlvv\" (UniqueName: \"kubernetes.io/projected/b6398465-9388-479e-85c9-b44b1797deff-kube-api-access-crlvv\") pod \"maas-api-revoke-sub-69b8ffed-9nl8q\" (UID: \"b6398465-9388-479e-85c9-b44b1797deff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b8ffed-9nl8q"
Nov 01 03:20:50.912228 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:50.912223    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6398465-9388-479e-85c9-b44b1797deff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69b8ffed-9nl8q\" (UID: \"b6398465-9388-479e-85c9-b44b1797deff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b8ffed-9nl8q"
Nov 01 03:20:50.912992 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:50.912971    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6398465-9388-479e-85c9-b44b1797deff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69b8ffed-9nl8q\" (UID: \"b6398465-9388-479e-85c9-b44b1797deff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b8ffed-9nl8q"
Nov 01 03:20:50.914475 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:50.914453    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6398465-9388-479e-85c9-b44b1797deff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69b8ffed-9nl8q\" (UID: \"b6398465-9388-479e-85c9-b44b1797deff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b8ffed-9nl8q"
Nov 01 03:20:50.973071 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:50.973041    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crlvv\" (UniqueName: \"kubernetes.io/projected/b6398465-9388-479e-85c9-b44b1797deff-kube-api-access-crlvv\") pod \"maas-api-revoke-sub-69b8ffed-9nl8q\" (UID: \"b6398465-9388-479e-85c9-b44b1797deff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b8ffed-9nl8q"
Nov 01 03:20:51.046795 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:51.046715    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b8ffed-9nl8q"
Nov 01 03:20:51.422856 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:51.422825    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69b8ffed-9nl8q"]
Nov 01 03:20:51.425933 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:20:51.425893    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6398465_9388_479e_85c9_b44b1797deff.slice/crio-e60fec83acd0b4a3ae86a999ce774e134facad2bc2ad48c2f100f0d341e5d138 WatchSource:0}: Error finding container e60fec83acd0b4a3ae86a999ce774e134facad2bc2ad48c2f100f0d341e5d138: Status 404 returned error can't find the container with id e60fec83acd0b4a3ae86a999ce774e134facad2bc2ad48c2f100f0d341e5d138
Nov 01 03:20:51.604115 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:51.604083    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b8ffed-9nl8q" event={"ID":"b6398465-9388-479e-85c9-b44b1797deff","Type":"ContainerStarted","Data":"10560f21e02865ea1d48acaa51904b8ad35f57151d6014554bf51fc3911d6144"}
Nov 01 03:20:51.604274 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:51.604137    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b8ffed-9nl8q" event={"ID":"b6398465-9388-479e-85c9-b44b1797deff","Type":"ContainerStarted","Data":"e60fec83acd0b4a3ae86a999ce774e134facad2bc2ad48c2f100f0d341e5d138"}
Nov 01 03:20:51.742759 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:51.742680    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:20:52.613029 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:52.612995    4930 generic.go:356] "Generic (PLEG): container finished" podID="b6398465-9388-479e-85c9-b44b1797deff" containerID="10560f21e02865ea1d48acaa51904b8ad35f57151d6014554bf51fc3911d6144" exitCode=0
Nov 01 03:20:52.613219 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:52.613029    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b8ffed-9nl8q" event={"ID":"b6398465-9388-479e-85c9-b44b1797deff","Type":"ContainerDied","Data":"10560f21e02865ea1d48acaa51904b8ad35f57151d6014554bf51fc3911d6144"}
Nov 01 03:20:53.783249 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:53.783213    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b8ffed-9nl8q"
Nov 01 03:20:53.949151 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:53.949070    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6398465-9388-479e-85c9-b44b1797deff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6398465-9388-479e-85c9-b44b1797deff-maas-api-openshift-ca\") pod \"b6398465-9388-479e-85c9-b44b1797deff\" (UID: \"b6398465-9388-479e-85c9-b44b1797deff\") "
Nov 01 03:20:53.949318 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:53.949270    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6398465-9388-479e-85c9-b44b1797deff-kube-api-access-crlvv\" (UniqueName: \"kubernetes.io/projected/b6398465-9388-479e-85c9-b44b1797deff-kube-api-access-crlvv\") pod \"b6398465-9388-479e-85c9-b44b1797deff\" (UID: \"b6398465-9388-479e-85c9-b44b1797deff\") "
Nov 01 03:20:53.949318 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:53.949311    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6398465-9388-479e-85c9-b44b1797deff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6398465-9388-479e-85c9-b44b1797deff-maas-api-xks-ca\") pod \"b6398465-9388-479e-85c9-b44b1797deff\" (UID: \"b6398465-9388-479e-85c9-b44b1797deff\") "
Nov 01 03:20:53.949549 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:53.949525    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6398465-9388-479e-85c9-b44b1797deff-maas-api-openshift-ca" pod "b6398465-9388-479e-85c9-b44b1797deff" (UID: "b6398465-9388-479e-85c9-b44b1797deff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:20:53.949706 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:53.949676    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6398465-9388-479e-85c9-b44b1797deff-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:20:53.951400 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:53.951371    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6398465-9388-479e-85c9-b44b1797deff-maas-api-xks-ca" pod "b6398465-9388-479e-85c9-b44b1797deff" (UID: "b6398465-9388-479e-85c9-b44b1797deff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:20:53.951482 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:53.951431    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6398465-9388-479e-85c9-b44b1797deff-kube-api-access-crlvv" pod "b6398465-9388-479e-85c9-b44b1797deff" (UID: "b6398465-9388-479e-85c9-b44b1797deff"). InnerVolumeSpecName "kube-api-access-crlvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:20:54.051068 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:54.051035    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crlvv\" (UniqueName: \"kubernetes.io/projected/b6398465-9388-479e-85c9-b44b1797deff-kube-api-access-crlvv\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:20:54.051068 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:54.051064    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6398465-9388-479e-85c9-b44b1797deff-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:20:54.170274 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:54.170240    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"]
Nov 01 03:20:54.170734 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:54.170720    4930 cpu_manager.go:414] "containerMap: removing container" podUID="b6398465-9388-479e-85c9-b44b1797deff" containerName="revoke-keys"
Nov 01 03:20:54.170783 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:54.170736    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6398465-9388-479e-85c9-b44b1797deff" containerName="revoke-keys"
Nov 01 03:20:54.170858 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:54.170849    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6398465-9388-479e-85c9-b44b1797deff" containerName="revoke-keys"
Nov 01 03:20:54.173935 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:54.173915    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"
Nov 01 03:20:54.235821 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:54.235737    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"]
Nov 01 03:20:54.235963 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:54.235888    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ded2d7\"/\"premium-w0-ded2d7-kserve-self-signed-certs\""
Nov 01 03:20:54.252388 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:54.252359    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2637bc6-3828-4b75-9783-5415519403a0-tmp-dir\") pod \"premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm\" (UID: \"b2637bc6-3828-4b75-9783-5415519403a0\") " pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"
Nov 01 03:20:54.252528 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:54.252399    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2637bc6-3828-4b75-9783-5415519403a0-home\") pod \"premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm\" (UID: \"b2637bc6-3828-4b75-9783-5415519403a0\") " pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"
Nov 01 03:20:54.252668 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:54.252644    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2637bc6-3828-4b75-9783-5415519403a0-tls-certs\") pod \"premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm\" (UID: \"b2637bc6-3828-4b75-9783-5415519403a0\") " pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"
Nov 01 03:20:54.252716 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:54.252703    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2637bc6-3828-4b75-9783-5415519403a0-model-cache\") pod \"premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm\" (UID: \"b2637bc6-3828-4b75-9783-5415519403a0\") " pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"
Nov 01 03:20:54.252824 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:54.252800    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2637bc6-3828-4b75-9783-5415519403a0-dshm\") pod \"premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm\" (UID: \"b2637bc6-3828-4b75-9783-5415519403a0\") " pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"
Nov 01 03:20:54.252874 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:54.252840    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkpwm\" (UniqueName: \"kubernetes.io/projected/b2637bc6-3828-4b75-9783-5415519403a0-kube-api-access-hkpwm\") pod \"premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm\" (UID: \"b2637bc6-3828-4b75-9783-5415519403a0\") " pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"
Nov 01 03:20:54.272024 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:54.271998    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:20:54.354133 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:54.354099    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2637bc6-3828-4b75-9783-5415519403a0-tmp-dir\") pod \"premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm\" (UID: \"b2637bc6-3828-4b75-9783-5415519403a0\") " pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"
Nov 01 03:20:54.354315 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:54.354140    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2637bc6-3828-4b75-9783-5415519403a0-home\") pod \"premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm\" (UID: \"b2637bc6-3828-4b75-9783-5415519403a0\") " pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"
Nov 01 03:20:54.354315 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:54.354212    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2637bc6-3828-4b75-9783-5415519403a0-tls-certs\") pod \"premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm\" (UID: \"b2637bc6-3828-4b75-9783-5415519403a0\") " pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"
Nov 01 03:20:54.354315 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:54.354241    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2637bc6-3828-4b75-9783-5415519403a0-model-cache\") pod \"premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm\" (UID: \"b2637bc6-3828-4b75-9783-5415519403a0\") " pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"
Nov 01 03:20:54.354315 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:54.354271    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2637bc6-3828-4b75-9783-5415519403a0-dshm\") pod \"premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm\" (UID: \"b2637bc6-3828-4b75-9783-5415519403a0\") " pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"
Nov 01 03:20:54.354315 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:54.354294    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkpwm\" (UniqueName: \"kubernetes.io/projected/b2637bc6-3828-4b75-9783-5415519403a0-kube-api-access-hkpwm\") pod \"premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm\" (UID: \"b2637bc6-3828-4b75-9783-5415519403a0\") " pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"
Nov 01 03:20:54.354677 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:54.354652    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2637bc6-3828-4b75-9783-5415519403a0-tmp-dir\") pod \"premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm\" (UID: \"b2637bc6-3828-4b75-9783-5415519403a0\") " pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"
Nov 01 03:20:54.354731 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:54.354699    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2637bc6-3828-4b75-9783-5415519403a0-model-cache\") pod \"premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm\" (UID: \"b2637bc6-3828-4b75-9783-5415519403a0\") " pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"
Nov 01 03:20:54.354767 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:54.354705    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2637bc6-3828-4b75-9783-5415519403a0-home\") pod \"premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm\" (UID: \"b2637bc6-3828-4b75-9783-5415519403a0\") " pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"
Nov 01 03:20:54.356737 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:54.356709    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2637bc6-3828-4b75-9783-5415519403a0-dshm\") pod \"premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm\" (UID: \"b2637bc6-3828-4b75-9783-5415519403a0\") " pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"
Nov 01 03:20:54.356894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:54.356873    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2637bc6-3828-4b75-9783-5415519403a0-tls-certs\") pod \"premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm\" (UID: \"b2637bc6-3828-4b75-9783-5415519403a0\") " pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"
Nov 01 03:20:54.416202 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:54.416174    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkpwm\" (UniqueName: \"kubernetes.io/projected/b2637bc6-3828-4b75-9783-5415519403a0-kube-api-access-hkpwm\") pod \"premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm\" (UID: \"b2637bc6-3828-4b75-9783-5415519403a0\") " pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"
Nov 01 03:20:54.485719 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:54.485688    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"
Nov 01 03:20:54.632046 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:54.631332    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b8ffed-9nl8q" event={"ID":"b6398465-9388-479e-85c9-b44b1797deff","Type":"ContainerDied","Data":"e60fec83acd0b4a3ae86a999ce774e134facad2bc2ad48c2f100f0d341e5d138"}
Nov 01 03:20:54.632046 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:54.631368    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69b8ffed-9nl8q"
Nov 01 03:20:54.632046 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:54.631384    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e60fec83acd0b4a3ae86a999ce774e134facad2bc2ad48c2f100f0d341e5d138"
Nov 01 03:20:54.918444 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:20:54.918410    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2637bc6_3828_4b75_9783_5415519403a0.slice/crio-9bbd30064d5b911d6e3721e232f06f325a8c0295bc81419808a48be71c4cf616 WatchSource:0}: Error finding container 9bbd30064d5b911d6e3721e232f06f325a8c0295bc81419808a48be71c4cf616: Status 404 returned error can't find the container with id 9bbd30064d5b911d6e3721e232f06f325a8c0295bc81419808a48be71c4cf616
Nov 01 03:20:54.918867 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:54.918648    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"]
Nov 01 03:20:55.639171 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:55.639132    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm" event={"ID":"b2637bc6-3828-4b75-9783-5415519403a0","Type":"ContainerStarted","Data":"3b36a5bfcdc6efcc3187773c157d458b552ca5df74ff765a661069033055a64f"}
Nov 01 03:20:55.639338 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:55.639187    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm" event={"ID":"b2637bc6-3828-4b75-9783-5415519403a0","Type":"ContainerStarted","Data":"9bbd30064d5b911d6e3721e232f06f325a8c0295bc81419808a48be71c4cf616"}
Nov 01 03:20:55.759239 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:55.759183    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm" podStartSLOduration=1.759167476 podStartE2EDuration="1.759167476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:20:55.754918753 +0000 UTC m=+860.277925341" watchObservedRunningTime="2026-11-01 03:20:55.759167476 +0000 UTC m=+860.282174064"
Nov 01 03:20:56.486495 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:56.486460    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"
Nov 01 03:20:56.548672 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:56.548636    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:20:59.272247 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:20:59.272208    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:21:01.345098 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:01.345068    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:21:04.486427 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:04.486395    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"
Nov 01 03:21:04.498816 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:04.498792    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"
Nov 01 03:21:04.700165 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:04.700136    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"
Nov 01 03:21:06.142361 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:06.142321    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:21:06.341731 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:06.341694    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2361ef33-v6j2n"]
Nov 01 03:21:06.345828 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:06.345811    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2361ef33-v6j2n"
Nov 01 03:21:06.400499 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:06.400436    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:21:06.408265 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:06.408233    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f875872-7339-495a-8d7b-9462041bcec4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2361ef33-v6j2n\" (UID: \"4f875872-7339-495a-8d7b-9462041bcec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2361ef33-v6j2n"
Nov 01 03:21:06.408461 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:06.408437    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f875872-7339-495a-8d7b-9462041bcec4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2361ef33-v6j2n\" (UID: \"4f875872-7339-495a-8d7b-9462041bcec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2361ef33-v6j2n"
Nov 01 03:21:06.408589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:06.408567    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5znbh\" (UniqueName: \"kubernetes.io/projected/4f875872-7339-495a-8d7b-9462041bcec4-kube-api-access-5znbh\") pod \"maas-api-revoke-sub-2361ef33-v6j2n\" (UID: \"4f875872-7339-495a-8d7b-9462041bcec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2361ef33-v6j2n"
Nov 01 03:21:06.449973 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:06.449948    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:21:06.450112 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:06.450006    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:21:06.450112 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:06.450018    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:21:06.454661 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:06.454638    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2361ef33-v6j2n"]
Nov 01 03:21:06.509526 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:06.509497    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f875872-7339-495a-8d7b-9462041bcec4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2361ef33-v6j2n\" (UID: \"4f875872-7339-495a-8d7b-9462041bcec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2361ef33-v6j2n"
Nov 01 03:21:06.509526 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:06.509528    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f875872-7339-495a-8d7b-9462041bcec4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2361ef33-v6j2n\" (UID: \"4f875872-7339-495a-8d7b-9462041bcec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2361ef33-v6j2n"
Nov 01 03:21:06.509736 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:06.509624    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5znbh\" (UniqueName: \"kubernetes.io/projected/4f875872-7339-495a-8d7b-9462041bcec4-kube-api-access-5znbh\") pod \"maas-api-revoke-sub-2361ef33-v6j2n\" (UID: \"4f875872-7339-495a-8d7b-9462041bcec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2361ef33-v6j2n"
Nov 01 03:21:06.510162 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:06.510143    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f875872-7339-495a-8d7b-9462041bcec4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2361ef33-v6j2n\" (UID: \"4f875872-7339-495a-8d7b-9462041bcec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2361ef33-v6j2n"
Nov 01 03:21:06.512084 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:06.512058    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f875872-7339-495a-8d7b-9462041bcec4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2361ef33-v6j2n\" (UID: \"4f875872-7339-495a-8d7b-9462041bcec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2361ef33-v6j2n"
Nov 01 03:21:06.571015 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:06.570983    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5znbh\" (UniqueName: \"kubernetes.io/projected/4f875872-7339-495a-8d7b-9462041bcec4-kube-api-access-5znbh\") pod \"maas-api-revoke-sub-2361ef33-v6j2n\" (UID: \"4f875872-7339-495a-8d7b-9462041bcec4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2361ef33-v6j2n"
Nov 01 03:21:06.659295 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:06.659229    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2361ef33-v6j2n"
Nov 01 03:21:07.040316 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:07.040285    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2361ef33-v6j2n"]
Nov 01 03:21:07.041758 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:21:07.041717    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f875872_7339_495a_8d7b_9462041bcec4.slice/crio-ae9c487c0411b847c8c03d7974cef6f8aeeec97729064345dd6d3a6b90c69b9f WatchSource:0}: Error finding container ae9c487c0411b847c8c03d7974cef6f8aeeec97729064345dd6d3a6b90c69b9f: Status 404 returned error can't find the container with id ae9c487c0411b847c8c03d7974cef6f8aeeec97729064345dd6d3a6b90c69b9f
Nov 01 03:21:07.712495 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:07.712411    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2361ef33-v6j2n" event={"ID":"4f875872-7339-495a-8d7b-9462041bcec4","Type":"ContainerStarted","Data":"c3c91b4814cbf9a3a9c51b4d93c694ba0f59679e109119cbbcda8cb9ead4de0e"}
Nov 01 03:21:07.712495 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:07.712467    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2361ef33-v6j2n" event={"ID":"4f875872-7339-495a-8d7b-9462041bcec4","Type":"ContainerStarted","Data":"ae9c487c0411b847c8c03d7974cef6f8aeeec97729064345dd6d3a6b90c69b9f"}
Nov 01 03:21:07.830371 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:07.830319    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2361ef33-v6j2n" podStartSLOduration=1.8303062460000001 podStartE2EDuration="1.830306246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:21:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:21:07.827697462 +0000 UTC m=+872.350704052" watchObservedRunningTime="2026-11-01 03:21:07.830306246 +0000 UTC m=+872.353312833"
Nov 01 03:21:08.721137 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:08.721104    4930 generic.go:356] "Generic (PLEG): container finished" podID="4f875872-7339-495a-8d7b-9462041bcec4" containerID="c3c91b4814cbf9a3a9c51b4d93c694ba0f59679e109119cbbcda8cb9ead4de0e" exitCode=0
Nov 01 03:21:08.721664 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:08.721185    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2361ef33-v6j2n" event={"ID":"4f875872-7339-495a-8d7b-9462041bcec4","Type":"ContainerDied","Data":"c3c91b4814cbf9a3a9c51b4d93c694ba0f59679e109119cbbcda8cb9ead4de0e"}
Nov 01 03:21:09.476481 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:09.476449    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:21:09.933688 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:09.933325    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2361ef33-v6j2n"
Nov 01 03:21:09.963813 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:09.963774    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f875872-7339-495a-8d7b-9462041bcec4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f875872-7339-495a-8d7b-9462041bcec4-maas-api-xks-ca\") pod \"4f875872-7339-495a-8d7b-9462041bcec4\" (UID: \"4f875872-7339-495a-8d7b-9462041bcec4\") "
Nov 01 03:21:09.964003 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:09.963850    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f875872-7339-495a-8d7b-9462041bcec4-kube-api-access-5znbh\" (UniqueName: \"kubernetes.io/projected/4f875872-7339-495a-8d7b-9462041bcec4-kube-api-access-5znbh\") pod \"4f875872-7339-495a-8d7b-9462041bcec4\" (UID: \"4f875872-7339-495a-8d7b-9462041bcec4\") "
Nov 01 03:21:09.964003 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:09.963894    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f875872-7339-495a-8d7b-9462041bcec4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f875872-7339-495a-8d7b-9462041bcec4-maas-api-openshift-ca\") pod \"4f875872-7339-495a-8d7b-9462041bcec4\" (UID: \"4f875872-7339-495a-8d7b-9462041bcec4\") "
Nov 01 03:21:09.964671 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:09.964644    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f875872-7339-495a-8d7b-9462041bcec4-maas-api-openshift-ca" pod "4f875872-7339-495a-8d7b-9462041bcec4" (UID: "4f875872-7339-495a-8d7b-9462041bcec4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:21:09.967666 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:09.967624    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f875872-7339-495a-8d7b-9462041bcec4-maas-api-xks-ca" pod "4f875872-7339-495a-8d7b-9462041bcec4" (UID: "4f875872-7339-495a-8d7b-9462041bcec4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:21:09.969379 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:09.969356    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f875872-7339-495a-8d7b-9462041bcec4-kube-api-access-5znbh" pod "4f875872-7339-495a-8d7b-9462041bcec4" (UID: "4f875872-7339-495a-8d7b-9462041bcec4"). InnerVolumeSpecName "kube-api-access-5znbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:21:10.065385 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:10.065291    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f875872-7339-495a-8d7b-9462041bcec4-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:21:10.065385 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:10.065322    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5znbh\" (UniqueName: \"kubernetes.io/projected/4f875872-7339-495a-8d7b-9462041bcec4-kube-api-access-5znbh\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:21:10.065385 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:10.065332    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f875872-7339-495a-8d7b-9462041bcec4-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:21:10.737852 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:10.737815    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2361ef33-v6j2n"
Nov 01 03:21:10.738015 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:10.737856    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2361ef33-v6j2n" event={"ID":"4f875872-7339-495a-8d7b-9462041bcec4","Type":"ContainerDied","Data":"ae9c487c0411b847c8c03d7974cef6f8aeeec97729064345dd6d3a6b90c69b9f"}
Nov 01 03:21:10.738015 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:10.737904    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae9c487c0411b847c8c03d7974cef6f8aeeec97729064345dd6d3a6b90c69b9f"
Nov 01 03:21:10.950508 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:10.950461    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:21:15.761637 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:15.761597    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:21:19.122196 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:21:19.122157    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:21:20.657301 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:20.657264    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:21:23.211513 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:23.211479    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:21:25.537031 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:25.536992    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:21:26.580977 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:26.580941    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b908441b-vzd9t"]
Nov 01 03:21:26.581445 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:26.581435    4930 cpu_manager.go:414] "containerMap: removing container" podUID="4f875872-7339-495a-8d7b-9462041bcec4" containerName="revoke-keys"
Nov 01 03:21:26.581545 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:26.581446    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f875872-7339-495a-8d7b-9462041bcec4" containerName="revoke-keys"
Nov 01 03:21:26.581545 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:26.581521    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f875872-7339-495a-8d7b-9462041bcec4" containerName="revoke-keys"
Nov 01 03:21:26.584968 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:26.584948    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b908441b-vzd9t"
Nov 01 03:21:26.643917 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:26.643889    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b908441b-vzd9t"]
Nov 01 03:21:26.662919 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:26.662893    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21c3e8ae-f194-47f8-a35a-25e8382a41db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b908441b-vzd9t\" (UID: \"21c3e8ae-f194-47f8-a35a-25e8382a41db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b908441b-vzd9t"
Nov 01 03:21:26.663081 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:26.663055    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21c3e8ae-f194-47f8-a35a-25e8382a41db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b908441b-vzd9t\" (UID: \"21c3e8ae-f194-47f8-a35a-25e8382a41db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b908441b-vzd9t"
Nov 01 03:21:26.663161 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:26.663144    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b74dn\" (UniqueName: \"kubernetes.io/projected/21c3e8ae-f194-47f8-a35a-25e8382a41db-kube-api-access-b74dn\") pod \"maas-api-revoke-sub-b908441b-vzd9t\" (UID: \"21c3e8ae-f194-47f8-a35a-25e8382a41db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b908441b-vzd9t"
Nov 01 03:21:26.689647 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:26.689618    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:21:26.689647 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:26.689637    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:21:26.689844 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:26.689667    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:21:26.689844 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:26.689717    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:21:26.764022 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:26.763996    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b74dn\" (UniqueName: \"kubernetes.io/projected/21c3e8ae-f194-47f8-a35a-25e8382a41db-kube-api-access-b74dn\") pod \"maas-api-revoke-sub-b908441b-vzd9t\" (UID: \"21c3e8ae-f194-47f8-a35a-25e8382a41db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b908441b-vzd9t"
Nov 01 03:21:26.764163 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:26.764066    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21c3e8ae-f194-47f8-a35a-25e8382a41db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b908441b-vzd9t\" (UID: \"21c3e8ae-f194-47f8-a35a-25e8382a41db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b908441b-vzd9t"
Nov 01 03:21:26.764163 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:26.764146    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21c3e8ae-f194-47f8-a35a-25e8382a41db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b908441b-vzd9t\" (UID: \"21c3e8ae-f194-47f8-a35a-25e8382a41db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b908441b-vzd9t"
Nov 01 03:21:26.764790 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:26.764770    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21c3e8ae-f194-47f8-a35a-25e8382a41db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b908441b-vzd9t\" (UID: \"21c3e8ae-f194-47f8-a35a-25e8382a41db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b908441b-vzd9t"
Nov 01 03:21:26.766468 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:26.766448    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21c3e8ae-f194-47f8-a35a-25e8382a41db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b908441b-vzd9t\" (UID: \"21c3e8ae-f194-47f8-a35a-25e8382a41db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b908441b-vzd9t"
Nov 01 03:21:26.824770 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:26.824743    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b74dn\" (UniqueName: \"kubernetes.io/projected/21c3e8ae-f194-47f8-a35a-25e8382a41db-kube-api-access-b74dn\") pod \"maas-api-revoke-sub-b908441b-vzd9t\" (UID: \"21c3e8ae-f194-47f8-a35a-25e8382a41db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b908441b-vzd9t"
Nov 01 03:21:26.898719 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:26.898693    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b908441b-vzd9t"
Nov 01 03:21:27.277322 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:21:27.277292    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21c3e8ae_f194_47f8_a35a_25e8382a41db.slice/crio-4987ce63ecbc8fcba313cff2ca662c9830377635417bd8aa210847d84d054e81 WatchSource:0}: Error finding container 4987ce63ecbc8fcba313cff2ca662c9830377635417bd8aa210847d84d054e81: Status 404 returned error can't find the container with id 4987ce63ecbc8fcba313cff2ca662c9830377635417bd8aa210847d84d054e81
Nov 01 03:21:27.290489 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:27.290461    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b908441b-vzd9t"]
Nov 01 03:21:27.891997 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:27.891961    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b908441b-vzd9t" event={"ID":"21c3e8ae-f194-47f8-a35a-25e8382a41db","Type":"ContainerStarted","Data":"4e771283049020b6524d0bbd77c6ea1b3a3e2ebf352c375dee17b055f325727a"}
Nov 01 03:21:27.892474 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:27.892022    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b908441b-vzd9t" event={"ID":"21c3e8ae-f194-47f8-a35a-25e8382a41db","Type":"ContainerStarted","Data":"4987ce63ecbc8fcba313cff2ca662c9830377635417bd8aa210847d84d054e81"}
Nov 01 03:21:28.017718 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:28.017652    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b908441b-vzd9t" podStartSLOduration=2.017635462 podStartE2EDuration="2.017635462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:21:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:21:28.0133197 +0000 UTC m=+892.536326288" watchObservedRunningTime="2026-11-01 03:21:28.017635462 +0000 UTC m=+892.540642053"
Nov 01 03:21:28.900818 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:28.900782    4930 generic.go:356] "Generic (PLEG): container finished" podID="21c3e8ae-f194-47f8-a35a-25e8382a41db" containerID="4e771283049020b6524d0bbd77c6ea1b3a3e2ebf352c375dee17b055f325727a" exitCode=0
Nov 01 03:21:28.901207 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:28.900865    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b908441b-vzd9t" event={"ID":"21c3e8ae-f194-47f8-a35a-25e8382a41db","Type":"ContainerDied","Data":"4e771283049020b6524d0bbd77c6ea1b3a3e2ebf352c375dee17b055f325727a"}
Nov 01 03:21:30.037029 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:30.036987    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:21:30.058570 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:30.058537    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b908441b-vzd9t"
Nov 01 03:21:30.203144 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:30.203058    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21c3e8ae-f194-47f8-a35a-25e8382a41db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21c3e8ae-f194-47f8-a35a-25e8382a41db-maas-api-xks-ca\") pod \"21c3e8ae-f194-47f8-a35a-25e8382a41db\" (UID: \"21c3e8ae-f194-47f8-a35a-25e8382a41db\") "
Nov 01 03:21:30.203309 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:30.203174    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21c3e8ae-f194-47f8-a35a-25e8382a41db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21c3e8ae-f194-47f8-a35a-25e8382a41db-maas-api-openshift-ca\") pod \"21c3e8ae-f194-47f8-a35a-25e8382a41db\" (UID: \"21c3e8ae-f194-47f8-a35a-25e8382a41db\") "
Nov 01 03:21:30.203309 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:30.203250    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21c3e8ae-f194-47f8-a35a-25e8382a41db-kube-api-access-b74dn\" (UniqueName: \"kubernetes.io/projected/21c3e8ae-f194-47f8-a35a-25e8382a41db-kube-api-access-b74dn\") pod \"21c3e8ae-f194-47f8-a35a-25e8382a41db\" (UID: \"21c3e8ae-f194-47f8-a35a-25e8382a41db\") "
Nov 01 03:21:30.203641 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:30.203612    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21c3e8ae-f194-47f8-a35a-25e8382a41db-maas-api-openshift-ca" pod "21c3e8ae-f194-47f8-a35a-25e8382a41db" (UID: "21c3e8ae-f194-47f8-a35a-25e8382a41db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:21:30.203870 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:30.203832    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21c3e8ae-f194-47f8-a35a-25e8382a41db-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:21:30.205828 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:30.205802    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21c3e8ae-f194-47f8-a35a-25e8382a41db-maas-api-xks-ca" pod "21c3e8ae-f194-47f8-a35a-25e8382a41db" (UID: "21c3e8ae-f194-47f8-a35a-25e8382a41db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:21:30.205902 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:30.205818    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21c3e8ae-f194-47f8-a35a-25e8382a41db-kube-api-access-b74dn" pod "21c3e8ae-f194-47f8-a35a-25e8382a41db" (UID: "21c3e8ae-f194-47f8-a35a-25e8382a41db"). InnerVolumeSpecName "kube-api-access-b74dn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:21:30.305231 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:30.305201    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21c3e8ae-f194-47f8-a35a-25e8382a41db-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:21:30.305231 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:30.305232    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b74dn\" (UniqueName: \"kubernetes.io/projected/21c3e8ae-f194-47f8-a35a-25e8382a41db-kube-api-access-b74dn\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:21:30.346339 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:30.346304    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:21:30.554138 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:30.554047    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:21:30.915415 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:30.915377    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b908441b-vzd9t" event={"ID":"21c3e8ae-f194-47f8-a35a-25e8382a41db","Type":"ContainerDied","Data":"4987ce63ecbc8fcba313cff2ca662c9830377635417bd8aa210847d84d054e81"}
Nov 01 03:21:30.915612 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:30.915433    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4987ce63ecbc8fcba313cff2ca662c9830377635417bd8aa210847d84d054e81"
Nov 01 03:21:30.915612 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:30.915389    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b908441b-vzd9t"
Nov 01 03:21:35.260843 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:35.260810    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:21:35.759444 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:35.759413    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jh955_f29ea61e-2a78-4c11-9566-f52497e135ff/console-operator/2.log"
Nov 01 03:21:35.759899 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:35.759872    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jh955_f29ea61e-2a78-4c11-9566-f52497e135ff/console-operator/2.log"
Nov 01 03:21:37.141672 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:37.141611    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:21:37.299439 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:37.299401    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:21:37.376528 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:37.376500    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0189dc42-rk8p8"]
Nov 01 03:21:37.377005 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:37.376991    4930 cpu_manager.go:414] "containerMap: removing container" podUID="21c3e8ae-f194-47f8-a35a-25e8382a41db" containerName="revoke-keys"
Nov 01 03:21:37.377052 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:37.377007    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21c3e8ae-f194-47f8-a35a-25e8382a41db" containerName="revoke-keys"
Nov 01 03:21:37.377103 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:37.377094    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="21c3e8ae-f194-47f8-a35a-25e8382a41db" containerName="revoke-keys"
Nov 01 03:21:37.380490 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:37.380469    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0189dc42-rk8p8"
Nov 01 03:21:37.434811 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:37.434753    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:21:37.434811 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:37.434773    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:21:37.435124 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:37.435108    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:21:37.435167 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:37.435143    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:21:37.489269 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:37.489239    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0189dc42-rk8p8"]
Nov 01 03:21:37.491981 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:37.491953    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vvj9\" (UniqueName: \"kubernetes.io/projected/c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d-kube-api-access-2vvj9\") pod \"maas-api-revoke-sub-0189dc42-rk8p8\" (UID: \"c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0189dc42-rk8p8"
Nov 01 03:21:37.492058 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:37.492004    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0189dc42-rk8p8\" (UID: \"c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0189dc42-rk8p8"
Nov 01 03:21:37.492118 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:37.492099    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0189dc42-rk8p8\" (UID: \"c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0189dc42-rk8p8"
Nov 01 03:21:37.593250 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:37.593220    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0189dc42-rk8p8\" (UID: \"c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0189dc42-rk8p8"
Nov 01 03:21:37.593456 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:37.593329    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vvj9\" (UniqueName: \"kubernetes.io/projected/c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d-kube-api-access-2vvj9\") pod \"maas-api-revoke-sub-0189dc42-rk8p8\" (UID: \"c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0189dc42-rk8p8"
Nov 01 03:21:37.593456 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:37.593365    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0189dc42-rk8p8\" (UID: \"c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0189dc42-rk8p8"
Nov 01 03:21:37.594164 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:37.594133    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0189dc42-rk8p8\" (UID: \"c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0189dc42-rk8p8"
Nov 01 03:21:37.596043 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:37.596013    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0189dc42-rk8p8\" (UID: \"c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0189dc42-rk8p8"
Nov 01 03:21:37.653834 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:37.653807    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vvj9\" (UniqueName: \"kubernetes.io/projected/c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d-kube-api-access-2vvj9\") pod \"maas-api-revoke-sub-0189dc42-rk8p8\" (UID: \"c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0189dc42-rk8p8"
Nov 01 03:21:37.693864 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:37.693799    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0189dc42-rk8p8"
Nov 01 03:21:38.074068 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:38.074038    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0189dc42-rk8p8"]
Nov 01 03:21:38.074677 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:21:38.074652    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2dc4dbf_ea70_4bf5_8693_9de7ea1ba33d.slice/crio-31d2da1692e62edd32a39739a67afc514f7b7f059fdcf990ce828ec788077b9d WatchSource:0}: Error finding container 31d2da1692e62edd32a39739a67afc514f7b7f059fdcf990ce828ec788077b9d: Status 404 returned error can't find the container with id 31d2da1692e62edd32a39739a67afc514f7b7f059fdcf990ce828ec788077b9d
Nov 01 03:21:38.982681 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:38.982649    4930 generic.go:356] "Generic (PLEG): container finished" podID="c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d" containerID="59f1c118b65f5ae4087b1408fe234e5034851a3a2d8129b434b5c92148697f2c" exitCode=0
Nov 01 03:21:38.983022 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:38.982728    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0189dc42-rk8p8" event={"ID":"c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d","Type":"ContainerDied","Data":"59f1c118b65f5ae4087b1408fe234e5034851a3a2d8129b434b5c92148697f2c"}
Nov 01 03:21:38.983022 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:38.982776    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0189dc42-rk8p8" event={"ID":"c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d","Type":"ContainerStarted","Data":"31d2da1692e62edd32a39739a67afc514f7b7f059fdcf990ce828ec788077b9d"}
Nov 01 03:21:40.100420 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:40.100375    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:21:40.169069 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:40.169042    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0189dc42-rk8p8"
Nov 01 03:21:40.322671 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:40.322546    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d-maas-api-xks-ca\") pod \"c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d\" (UID: \"c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d\") "
Nov 01 03:21:40.322820 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:40.322793    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d-maas-api-openshift-ca\") pod \"c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d\" (UID: \"c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d\") "
Nov 01 03:21:40.322933 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:40.322911    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d-kube-api-access-2vvj9\" (UniqueName: \"kubernetes.io/projected/c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d-kube-api-access-2vvj9\") pod \"c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d\" (UID: \"c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d\") "
Nov 01 03:21:40.323204 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:40.323174    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d-maas-api-openshift-ca" pod "c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d" (UID: "c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:21:40.323628 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:40.323600    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:21:40.324968 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:40.324941    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d-kube-api-access-2vvj9" pod "c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d" (UID: "c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d"). InnerVolumeSpecName "kube-api-access-2vvj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:21:40.325190 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:40.325170    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d-maas-api-xks-ca" pod "c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d" (UID: "c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:21:40.425451 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:40.425415    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vvj9\" (UniqueName: \"kubernetes.io/projected/c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d-kube-api-access-2vvj9\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:21:40.425451 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:40.425450    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:21:40.701929 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:40.701899    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:21:40.996874 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:40.996801    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0189dc42-rk8p8"
Nov 01 03:21:40.996874 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:40.996846    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0189dc42-rk8p8" event={"ID":"c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d","Type":"ContainerDied","Data":"31d2da1692e62edd32a39739a67afc514f7b7f059fdcf990ce828ec788077b9d"}
Nov 01 03:21:40.997039 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:40.996897    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31d2da1692e62edd32a39739a67afc514f7b7f059fdcf990ce828ec788077b9d"
Nov 01 03:21:44.846600 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:44.845486    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:21:49.120467 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:21:49.120430    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:21:49.644068 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:49.644032    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:21:54.441653 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:54.441612    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:21:59.340948 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:21:59.340907    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:22:01.510384 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:01.510347    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce6047d8-wnpkp"]
Nov 01 03:22:01.511154 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:01.511128    4930 cpu_manager.go:414] "containerMap: removing container" podUID="c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d" containerName="revoke-keys"
Nov 01 03:22:01.511279 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:01.511157    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d" containerName="revoke-keys"
Nov 01 03:22:01.511367 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:01.511324    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d" containerName="revoke-keys"
Nov 01 03:22:01.520088 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:01.520064    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6047d8-wnpkp"
Nov 01 03:22:01.565235 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:01.565200    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce6047d8-wnpkp"]
Nov 01 03:22:01.609450 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:01.609411    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:22:01.609791 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:01.609754    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:22:01.609791 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:01.609755    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:22:01.615141 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:01.615111    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d58c6470-aafb-44c0-ae6a-6b61ddacfd2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce6047d8-wnpkp\" (UID: \"d58c6470-aafb-44c0-ae6a-6b61ddacfd2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6047d8-wnpkp"
Nov 01 03:22:01.615328 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:01.615310    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d58c6470-aafb-44c0-ae6a-6b61ddacfd2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce6047d8-wnpkp\" (UID: \"d58c6470-aafb-44c0-ae6a-6b61ddacfd2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6047d8-wnpkp"
Nov 01 03:22:01.615649 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:01.615603    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnfr7\" (UniqueName: \"kubernetes.io/projected/d58c6470-aafb-44c0-ae6a-6b61ddacfd2d-kube-api-access-nnfr7\") pod \"maas-api-revoke-sub-ce6047d8-wnpkp\" (UID: \"d58c6470-aafb-44c0-ae6a-6b61ddacfd2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6047d8-wnpkp"
Nov 01 03:22:01.621915 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:01.621881    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:22:01.716652 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:01.716606    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d58c6470-aafb-44c0-ae6a-6b61ddacfd2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce6047d8-wnpkp\" (UID: \"d58c6470-aafb-44c0-ae6a-6b61ddacfd2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6047d8-wnpkp"
Nov 01 03:22:01.716854 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:01.716666    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d58c6470-aafb-44c0-ae6a-6b61ddacfd2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce6047d8-wnpkp\" (UID: \"d58c6470-aafb-44c0-ae6a-6b61ddacfd2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6047d8-wnpkp"
Nov 01 03:22:01.716854 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:01.716766    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnfr7\" (UniqueName: \"kubernetes.io/projected/d58c6470-aafb-44c0-ae6a-6b61ddacfd2d-kube-api-access-nnfr7\") pod \"maas-api-revoke-sub-ce6047d8-wnpkp\" (UID: \"d58c6470-aafb-44c0-ae6a-6b61ddacfd2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6047d8-wnpkp"
Nov 01 03:22:01.718116 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:01.718076    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d58c6470-aafb-44c0-ae6a-6b61ddacfd2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce6047d8-wnpkp\" (UID: \"d58c6470-aafb-44c0-ae6a-6b61ddacfd2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6047d8-wnpkp"
Nov 01 03:22:01.720297 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:01.720263    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d58c6470-aafb-44c0-ae6a-6b61ddacfd2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce6047d8-wnpkp\" (UID: \"d58c6470-aafb-44c0-ae6a-6b61ddacfd2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6047d8-wnpkp"
Nov 01 03:22:01.779774 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:01.779696    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnfr7\" (UniqueName: \"kubernetes.io/projected/d58c6470-aafb-44c0-ae6a-6b61ddacfd2d-kube-api-access-nnfr7\") pod \"maas-api-revoke-sub-ce6047d8-wnpkp\" (UID: \"d58c6470-aafb-44c0-ae6a-6b61ddacfd2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6047d8-wnpkp"
Nov 01 03:22:01.851919 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:01.851880    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6047d8-wnpkp"
Nov 01 03:22:02.246382 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:02.246353    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce6047d8-wnpkp"]
Nov 01 03:22:02.247643 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:22:02.247613    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd58c6470_aafb_44c0_ae6a_6b61ddacfd2d.slice/crio-5b438adab7cfe5a669cf40a9792501ed5857c5b4b6204e3ca206efe9479dd5d4 WatchSource:0}: Error finding container 5b438adab7cfe5a669cf40a9792501ed5857c5b4b6204e3ca206efe9479dd5d4: Status 404 returned error can't find the container with id 5b438adab7cfe5a669cf40a9792501ed5857c5b4b6204e3ca206efe9479dd5d4
Nov 01 03:22:03.178908 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:03.178880    4930 generic.go:356] "Generic (PLEG): container finished" podID="d58c6470-aafb-44c0-ae6a-6b61ddacfd2d" containerID="40e51fcaf088ad0baf41f77b1aaf25966e82157368190aaa052cfb29c5fadc56" exitCode=0
Nov 01 03:22:03.179222 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:03.178957    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6047d8-wnpkp" event={"ID":"d58c6470-aafb-44c0-ae6a-6b61ddacfd2d","Type":"ContainerDied","Data":"40e51fcaf088ad0baf41f77b1aaf25966e82157368190aaa052cfb29c5fadc56"}
Nov 01 03:22:03.179222 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:03.179009    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6047d8-wnpkp" event={"ID":"d58c6470-aafb-44c0-ae6a-6b61ddacfd2d","Type":"ContainerStarted","Data":"5b438adab7cfe5a669cf40a9792501ed5857c5b4b6204e3ca206efe9479dd5d4"}
Nov 01 03:22:04.338078 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:04.338053    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6047d8-wnpkp"
Nov 01 03:22:04.344272 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:04.344248    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d58c6470-aafb-44c0-ae6a-6b61ddacfd2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d58c6470-aafb-44c0-ae6a-6b61ddacfd2d-maas-api-xks-ca\") pod \"d58c6470-aafb-44c0-ae6a-6b61ddacfd2d\" (UID: \"d58c6470-aafb-44c0-ae6a-6b61ddacfd2d\") "
Nov 01 03:22:04.344418 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:04.344400    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d58c6470-aafb-44c0-ae6a-6b61ddacfd2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d58c6470-aafb-44c0-ae6a-6b61ddacfd2d-maas-api-openshift-ca\") pod \"d58c6470-aafb-44c0-ae6a-6b61ddacfd2d\" (UID: \"d58c6470-aafb-44c0-ae6a-6b61ddacfd2d\") "
Nov 01 03:22:04.344733 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:04.344711    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d58c6470-aafb-44c0-ae6a-6b61ddacfd2d-maas-api-openshift-ca" pod "d58c6470-aafb-44c0-ae6a-6b61ddacfd2d" (UID: "d58c6470-aafb-44c0-ae6a-6b61ddacfd2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:22:04.344824 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:04.344787    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d58c6470-aafb-44c0-ae6a-6b61ddacfd2d-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:04.346194 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:04.346174    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d58c6470-aafb-44c0-ae6a-6b61ddacfd2d-maas-api-xks-ca" pod "d58c6470-aafb-44c0-ae6a-6b61ddacfd2d" (UID: "d58c6470-aafb-44c0-ae6a-6b61ddacfd2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:22:04.401944 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:04.401910    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:22:04.446086 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:04.445995    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d58c6470-aafb-44c0-ae6a-6b61ddacfd2d-kube-api-access-nnfr7\" (UniqueName: \"kubernetes.io/projected/d58c6470-aafb-44c0-ae6a-6b61ddacfd2d-kube-api-access-nnfr7\") pod \"d58c6470-aafb-44c0-ae6a-6b61ddacfd2d\" (UID: \"d58c6470-aafb-44c0-ae6a-6b61ddacfd2d\") "
Nov 01 03:22:04.446502 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:04.446484    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d58c6470-aafb-44c0-ae6a-6b61ddacfd2d-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:04.448149 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:04.448123    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d58c6470-aafb-44c0-ae6a-6b61ddacfd2d-kube-api-access-nnfr7" pod "d58c6470-aafb-44c0-ae6a-6b61ddacfd2d" (UID: "d58c6470-aafb-44c0-ae6a-6b61ddacfd2d"). InnerVolumeSpecName "kube-api-access-nnfr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:22:04.547772 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:04.547742    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnfr7\" (UniqueName: \"kubernetes.io/projected/d58c6470-aafb-44c0-ae6a-6b61ddacfd2d-kube-api-access-nnfr7\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:05.195760 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:05.195728    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6047d8-wnpkp"
Nov 01 03:22:05.195760 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:05.195756    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce6047d8-wnpkp" event={"ID":"d58c6470-aafb-44c0-ae6a-6b61ddacfd2d","Type":"ContainerDied","Data":"5b438adab7cfe5a669cf40a9792501ed5857c5b4b6204e3ca206efe9479dd5d4"}
Nov 01 03:22:05.196014 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:05.195809    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b438adab7cfe5a669cf40a9792501ed5857c5b4b6204e3ca206efe9479dd5d4"
Nov 01 03:22:09.234698 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:09.234645    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:22:09.652401 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:09.652365    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73cabe60-pd29j"]
Nov 01 03:22:09.652907 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:09.652891    4930 cpu_manager.go:414] "containerMap: removing container" podUID="d58c6470-aafb-44c0-ae6a-6b61ddacfd2d" containerName="revoke-keys"
Nov 01 03:22:09.652967 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:09.652909    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d58c6470-aafb-44c0-ae6a-6b61ddacfd2d" containerName="revoke-keys"
Nov 01 03:22:09.653013 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:09.653003    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="d58c6470-aafb-44c0-ae6a-6b61ddacfd2d" containerName="revoke-keys"
Nov 01 03:22:09.656418 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:09.656401    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73cabe60-pd29j"
Nov 01 03:22:09.707143 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:09.707117    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73cabe60-pd29j\" (UID: \"2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73cabe60-pd29j"
Nov 01 03:22:09.707259 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:09.707206    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73cabe60-pd29j\" (UID: \"2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73cabe60-pd29j"
Nov 01 03:22:09.707355 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:09.707332    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwjvg\" (UniqueName: \"kubernetes.io/projected/2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1-kube-api-access-rwjvg\") pod \"maas-api-revoke-sub-73cabe60-pd29j\" (UID: \"2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73cabe60-pd29j"
Nov 01 03:22:09.715168 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:09.715143    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73cabe60-pd29j"]
Nov 01 03:22:09.761003 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:09.760933    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:22:09.761003 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:09.760936    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:22:09.761194 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:09.761029    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:22:09.761194 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:09.761079    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:22:09.808826 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:09.808794    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73cabe60-pd29j\" (UID: \"2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73cabe60-pd29j"
Nov 01 03:22:09.808987 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:09.808845    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73cabe60-pd29j\" (UID: \"2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73cabe60-pd29j"
Nov 01 03:22:09.808987 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:09.808968    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwjvg\" (UniqueName: \"kubernetes.io/projected/2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1-kube-api-access-rwjvg\") pod \"maas-api-revoke-sub-73cabe60-pd29j\" (UID: \"2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73cabe60-pd29j"
Nov 01 03:22:09.809700 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:09.809675    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73cabe60-pd29j\" (UID: \"2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73cabe60-pd29j"
Nov 01 03:22:09.811424 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:09.811402    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73cabe60-pd29j\" (UID: \"2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73cabe60-pd29j"
Nov 01 03:22:09.869999 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:09.869967    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwjvg\" (UniqueName: \"kubernetes.io/projected/2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1-kube-api-access-rwjvg\") pod \"maas-api-revoke-sub-73cabe60-pd29j\" (UID: \"2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73cabe60-pd29j"
Nov 01 03:22:09.969536 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:09.969449    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73cabe60-pd29j"
Nov 01 03:22:10.346465 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:10.346436    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73cabe60-pd29j"]
Nov 01 03:22:10.347433 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:22:10.347404    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2158baa8_ec88_4f7c_bb5b_42d7eba3e9f1.slice/crio-f123d296c30ddcf1e52ae5732c3be9695672f1bc5edcfa9c97effec98389f75b WatchSource:0}: Error finding container f123d296c30ddcf1e52ae5732c3be9695672f1bc5edcfa9c97effec98389f75b: Status 404 returned error can't find the container with id f123d296c30ddcf1e52ae5732c3be9695672f1bc5edcfa9c97effec98389f75b
Nov 01 03:22:11.238787 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:11.238757    4930 generic.go:356] "Generic (PLEG): container finished" podID="2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1" containerID="a34cc1bfa7a268f120bcbb3db0038e16d8d2064c5eb930c60cd72019601db001" exitCode=0
Nov 01 03:22:11.238899 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:11.238822    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73cabe60-pd29j" event={"ID":"2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1","Type":"ContainerDied","Data":"a34cc1bfa7a268f120bcbb3db0038e16d8d2064c5eb930c60cd72019601db001"}
Nov 01 03:22:11.238899 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:11.238859    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73cabe60-pd29j" event={"ID":"2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1","Type":"ContainerStarted","Data":"f123d296c30ddcf1e52ae5732c3be9695672f1bc5edcfa9c97effec98389f75b"}
Nov 01 03:22:12.397664 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:12.397628    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73cabe60-pd29j"
Nov 01 03:22:12.438168 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:12.438143    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1-kube-api-access-rwjvg\" (UniqueName: \"kubernetes.io/projected/2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1-kube-api-access-rwjvg\") pod \"2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1\" (UID: \"2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1\") "
Nov 01 03:22:12.438293 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:12.438215    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1-maas-api-openshift-ca\") pod \"2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1\" (UID: \"2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1\") "
Nov 01 03:22:12.438293 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:12.438241    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1-maas-api-xks-ca\") pod \"2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1\" (UID: \"2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1\") "
Nov 01 03:22:12.438605 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:12.438575    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1-maas-api-openshift-ca" pod "2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1" (UID: "2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:22:12.440215 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:12.440193    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1-maas-api-xks-ca" pod "2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1" (UID: "2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:22:12.440310 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:12.440290    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1-kube-api-access-rwjvg" pod "2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1" (UID: "2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1"). InnerVolumeSpecName "kube-api-access-rwjvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:22:12.539644 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:12.539574    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwjvg\" (UniqueName: \"kubernetes.io/projected/2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1-kube-api-access-rwjvg\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:12.539644 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:12.539604    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:12.539644 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:12.539615    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:13.252362 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:13.252333    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73cabe60-pd29j"
Nov 01 03:22:13.252631 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:13.252333    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73cabe60-pd29j" event={"ID":"2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1","Type":"ContainerDied","Data":"f123d296c30ddcf1e52ae5732c3be9695672f1bc5edcfa9c97effec98389f75b"}
Nov 01 03:22:13.252631 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:13.252467    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f123d296c30ddcf1e52ae5732c3be9695672f1bc5edcfa9c97effec98389f75b"
Nov 01 03:22:14.040491 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:14.040454    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:22:15.197872 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:15.197836    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a673175-gpzwp"]
Nov 01 03:22:15.198536 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:15.198515    4930 cpu_manager.go:414] "containerMap: removing container" podUID="2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1" containerName="revoke-keys"
Nov 01 03:22:15.198647 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:15.198537    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1" containerName="revoke-keys"
Nov 01 03:22:15.198760 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:15.198746    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1" containerName="revoke-keys"
Nov 01 03:22:15.202167 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:15.202148    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a673175-gpzwp"
Nov 01 03:22:15.256894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:15.256871    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:22:15.256996 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:15.256901    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:22:15.257108 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:15.257096    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:22:15.268624 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:15.268600    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j562h\" (UniqueName: \"kubernetes.io/projected/aa78a11f-6052-4a54-aabf-dc93c44310d5-kube-api-access-j562h\") pod \"maas-api-revoke-sub-6a673175-gpzwp\" (UID: \"aa78a11f-6052-4a54-aabf-dc93c44310d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a673175-gpzwp"
Nov 01 03:22:15.268744 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:15.268633    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa78a11f-6052-4a54-aabf-dc93c44310d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a673175-gpzwp\" (UID: \"aa78a11f-6052-4a54-aabf-dc93c44310d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a673175-gpzwp"
Nov 01 03:22:15.268808 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:15.268752    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa78a11f-6052-4a54-aabf-dc93c44310d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a673175-gpzwp\" (UID: \"aa78a11f-6052-4a54-aabf-dc93c44310d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a673175-gpzwp"
Nov 01 03:22:15.306496 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:15.306465    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:22:15.309922 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:15.309900    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a673175-gpzwp"]
Nov 01 03:22:15.370214 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:15.370186    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j562h\" (UniqueName: \"kubernetes.io/projected/aa78a11f-6052-4a54-aabf-dc93c44310d5-kube-api-access-j562h\") pod \"maas-api-revoke-sub-6a673175-gpzwp\" (UID: \"aa78a11f-6052-4a54-aabf-dc93c44310d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a673175-gpzwp"
Nov 01 03:22:15.370319 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:15.370221    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa78a11f-6052-4a54-aabf-dc93c44310d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a673175-gpzwp\" (UID: \"aa78a11f-6052-4a54-aabf-dc93c44310d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a673175-gpzwp"
Nov 01 03:22:15.370319 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:15.370289    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa78a11f-6052-4a54-aabf-dc93c44310d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a673175-gpzwp\" (UID: \"aa78a11f-6052-4a54-aabf-dc93c44310d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a673175-gpzwp"
Nov 01 03:22:15.371077 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:15.371050    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa78a11f-6052-4a54-aabf-dc93c44310d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a673175-gpzwp\" (UID: \"aa78a11f-6052-4a54-aabf-dc93c44310d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a673175-gpzwp"
Nov 01 03:22:15.372738 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:15.372719    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa78a11f-6052-4a54-aabf-dc93c44310d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a673175-gpzwp\" (UID: \"aa78a11f-6052-4a54-aabf-dc93c44310d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a673175-gpzwp"
Nov 01 03:22:15.437791 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:15.437763    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j562h\" (UniqueName: \"kubernetes.io/projected/aa78a11f-6052-4a54-aabf-dc93c44310d5-kube-api-access-j562h\") pod \"maas-api-revoke-sub-6a673175-gpzwp\" (UID: \"aa78a11f-6052-4a54-aabf-dc93c44310d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a673175-gpzwp"
Nov 01 03:22:15.515203 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:15.515122    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a673175-gpzwp"
Nov 01 03:22:15.897673 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:15.897643    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a673175-gpzwp"]
Nov 01 03:22:15.898874 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:22:15.898842    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa78a11f_6052_4a54_aabf_dc93c44310d5.slice/crio-2886aa354c89bc902b419643a9b4a262739ebb36b7a067c2e21a15694f0313a6 WatchSource:0}: Error finding container 2886aa354c89bc902b419643a9b4a262739ebb36b7a067c2e21a15694f0313a6: Status 404 returned error can't find the container with id 2886aa354c89bc902b419643a9b4a262739ebb36b7a067c2e21a15694f0313a6
Nov 01 03:22:16.275376 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:16.275341    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a673175-gpzwp" event={"ID":"aa78a11f-6052-4a54-aabf-dc93c44310d5","Type":"ContainerStarted","Data":"02ac00a89967f15549a1a180f55aeea5c3aea43b3eac0a260fada7bb3daa836f"}
Nov 01 03:22:16.275794 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:16.275397    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a673175-gpzwp" event={"ID":"aa78a11f-6052-4a54-aabf-dc93c44310d5","Type":"ContainerStarted","Data":"2886aa354c89bc902b419643a9b4a262739ebb36b7a067c2e21a15694f0313a6"}
Nov 01 03:22:16.391811 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:16.391763    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a673175-gpzwp" podStartSLOduration=1.391749756 podStartE2EDuration="1.391749756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:22:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:22:16.39023368 +0000 UTC m=+940.913240269" watchObservedRunningTime="2026-11-01 03:22:16.391749756 +0000 UTC m=+940.914756344"
Nov 01 03:22:17.283269 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:17.283234    4930 generic.go:356] "Generic (PLEG): container finished" podID="aa78a11f-6052-4a54-aabf-dc93c44310d5" containerID="02ac00a89967f15549a1a180f55aeea5c3aea43b3eac0a260fada7bb3daa836f" exitCode=0
Nov 01 03:22:17.283736 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:17.283321    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a673175-gpzwp" event={"ID":"aa78a11f-6052-4a54-aabf-dc93c44310d5","Type":"ContainerDied","Data":"02ac00a89967f15549a1a180f55aeea5c3aea43b3eac0a260fada7bb3daa836f"}
Nov 01 03:22:18.439730 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:18.434355    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69e6dcb6-x7gdw"]
Nov 01 03:22:18.439730 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:18.439292    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e6dcb6-x7gdw"
Nov 01 03:22:18.480367 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:18.480329    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a673175-gpzwp"
Nov 01 03:22:18.481795 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:18.481770    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69e6dcb6-x7gdw"]
Nov 01 03:22:18.504792 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:18.504761    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k87k6\" (UniqueName: \"kubernetes.io/projected/94819519-6491-4744-9a04-c323de973006-kube-api-access-k87k6\") pod \"maas-api-revoke-sub-69e6dcb6-x7gdw\" (UID: \"94819519-6491-4744-9a04-c323de973006\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e6dcb6-x7gdw"
Nov 01 03:22:18.504907 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:18.504843    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94819519-6491-4744-9a04-c323de973006-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69e6dcb6-x7gdw\" (UID: \"94819519-6491-4744-9a04-c323de973006\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e6dcb6-x7gdw"
Nov 01 03:22:18.504984 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:18.504957    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94819519-6491-4744-9a04-c323de973006-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69e6dcb6-x7gdw\" (UID: \"94819519-6491-4744-9a04-c323de973006\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e6dcb6-x7gdw"
Nov 01 03:22:18.535001 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:18.534972    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh"]
Nov 01 03:22:18.535239 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:18.535216    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh" podUID="97f3268e-176f-46ef-8431-0b3e985caa0f" containerName="istio-proxy" containerID="cri-o://71075b263b0356ade2cb8a16ed245ad3bf547c27d53d4691287cb9f8ab038f30" gracePeriod=30
Nov 01 03:22:18.538049 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:18.538028    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n"]
Nov 01 03:22:18.538396 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:18.538368    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n" podUID="35b89d8d-ab16-41ea-8a99-bcc80e212838" containerName="istio-proxy" containerID="cri-o://1f5d54f9f7fbe2e03833c6dddedd330b6aba8f0cee43ba61d529a0683894b586" gracePeriod=30
Nov 01 03:22:18.540803 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:18.540741    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"]
Nov 01 03:22:18.540949 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:18.540928    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5" podUID="24e9b838-6a66-49b0-9aec-c43bfd9221ae" containerName="istio-proxy" containerID="cri-o://7da7748e972490cf4d1f1b2791d55b107b7bcf9eb09d19ece15ddc709ae82f77" gracePeriod=30
Nov 01 03:22:18.610351 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:18.605315    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa78a11f-6052-4a54-aabf-dc93c44310d5-kube-api-access-j562h\" (UniqueName: \"kubernetes.io/projected/aa78a11f-6052-4a54-aabf-dc93c44310d5-kube-api-access-j562h\") pod \"aa78a11f-6052-4a54-aabf-dc93c44310d5\" (UID: \"aa78a11f-6052-4a54-aabf-dc93c44310d5\") "
Nov 01 03:22:18.610351 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:18.605497    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa78a11f-6052-4a54-aabf-dc93c44310d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa78a11f-6052-4a54-aabf-dc93c44310d5-maas-api-openshift-ca\") pod \"aa78a11f-6052-4a54-aabf-dc93c44310d5\" (UID: \"aa78a11f-6052-4a54-aabf-dc93c44310d5\") "
Nov 01 03:22:18.610351 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:18.605593    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa78a11f-6052-4a54-aabf-dc93c44310d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa78a11f-6052-4a54-aabf-dc93c44310d5-maas-api-xks-ca\") pod \"aa78a11f-6052-4a54-aabf-dc93c44310d5\" (UID: \"aa78a11f-6052-4a54-aabf-dc93c44310d5\") "
Nov 01 03:22:18.610351 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:18.605926    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k87k6\" (UniqueName: \"kubernetes.io/projected/94819519-6491-4744-9a04-c323de973006-kube-api-access-k87k6\") pod \"maas-api-revoke-sub-69e6dcb6-x7gdw\" (UID: \"94819519-6491-4744-9a04-c323de973006\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e6dcb6-x7gdw"
Nov 01 03:22:18.610351 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:18.606011    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94819519-6491-4744-9a04-c323de973006-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69e6dcb6-x7gdw\" (UID: \"94819519-6491-4744-9a04-c323de973006\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e6dcb6-x7gdw"
Nov 01 03:22:18.610351 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:18.606097    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94819519-6491-4744-9a04-c323de973006-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69e6dcb6-x7gdw\" (UID: \"94819519-6491-4744-9a04-c323de973006\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e6dcb6-x7gdw"
Nov 01 03:22:18.610351 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:18.606400    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa78a11f-6052-4a54-aabf-dc93c44310d5-maas-api-openshift-ca" pod "aa78a11f-6052-4a54-aabf-dc93c44310d5" (UID: "aa78a11f-6052-4a54-aabf-dc93c44310d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:22:18.610351 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:18.606957    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94819519-6491-4744-9a04-c323de973006-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69e6dcb6-x7gdw\" (UID: \"94819519-6491-4744-9a04-c323de973006\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e6dcb6-x7gdw"
Nov 01 03:22:18.610351 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:18.609784    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa78a11f-6052-4a54-aabf-dc93c44310d5-kube-api-access-j562h" pod "aa78a11f-6052-4a54-aabf-dc93c44310d5" (UID: "aa78a11f-6052-4a54-aabf-dc93c44310d5"). InnerVolumeSpecName "kube-api-access-j562h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:22:18.610351 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:18.609984    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94819519-6491-4744-9a04-c323de973006-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69e6dcb6-x7gdw\" (UID: \"94819519-6491-4744-9a04-c323de973006\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e6dcb6-x7gdw"
Nov 01 03:22:18.610351 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:18.610206    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa78a11f-6052-4a54-aabf-dc93c44310d5-maas-api-xks-ca" pod "aa78a11f-6052-4a54-aabf-dc93c44310d5" (UID: "aa78a11f-6052-4a54-aabf-dc93c44310d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:22:18.667811 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:18.667765    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k87k6\" (UniqueName: \"kubernetes.io/projected/94819519-6491-4744-9a04-c323de973006-kube-api-access-k87k6\") pod \"maas-api-revoke-sub-69e6dcb6-x7gdw\" (UID: \"94819519-6491-4744-9a04-c323de973006\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e6dcb6-x7gdw"
Nov 01 03:22:18.707882 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:18.707732    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j562h\" (UniqueName: \"kubernetes.io/projected/aa78a11f-6052-4a54-aabf-dc93c44310d5-kube-api-access-j562h\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:18.707882 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:18.707767    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa78a11f-6052-4a54-aabf-dc93c44310d5-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:18.707882 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:18.707783    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa78a11f-6052-4a54-aabf-dc93c44310d5-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:18.778391 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:18.778363    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e6dcb6-x7gdw"
Nov 01 03:22:18.935190 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:18.935151    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:22:19.124081 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:22:19.124047    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:22:19.164838 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:19.164795    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69e6dcb6-x7gdw"]
Nov 01 03:22:19.166341 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:22:19.166310    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94819519_6491_4744_9a04_c323de973006.slice/crio-9b9873d1c1fdc8035fa2a6e8ac88f37e18cacb530d6d385908083c0baa6922f7 WatchSource:0}: Error finding container 9b9873d1c1fdc8035fa2a6e8ac88f37e18cacb530d6d385908083c0baa6922f7: Status 404 returned error can't find the container with id 9b9873d1c1fdc8035fa2a6e8ac88f37e18cacb530d6d385908083c0baa6922f7
Nov 01 03:22:19.296810 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:19.296770    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e6dcb6-x7gdw" event={"ID":"94819519-6491-4744-9a04-c323de973006","Type":"ContainerStarted","Data":"9b9873d1c1fdc8035fa2a6e8ac88f37e18cacb530d6d385908083c0baa6922f7"}
Nov 01 03:22:19.298130 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:19.298099    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a673175-gpzwp" event={"ID":"aa78a11f-6052-4a54-aabf-dc93c44310d5","Type":"ContainerDied","Data":"2886aa354c89bc902b419643a9b4a262739ebb36b7a067c2e21a15694f0313a6"}
Nov 01 03:22:19.298130 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:19.298128    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a673175-gpzwp"
Nov 01 03:22:19.298297 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:19.298149    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2886aa354c89bc902b419643a9b4a262739ebb36b7a067c2e21a15694f0313a6"
Nov 01 03:22:20.307363 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:20.307268    4930 generic.go:356] "Generic (PLEG): container finished" podID="94819519-6491-4744-9a04-c323de973006" containerID="694d2edc6a51ea2a838cfd9ddc26ff654b87e680bbcd8145dc2ead2a4d2fff30" exitCode=0
Nov 01 03:22:20.307812 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:20.307369    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e6dcb6-x7gdw" event={"ID":"94819519-6491-4744-9a04-c323de973006","Type":"ContainerDied","Data":"694d2edc6a51ea2a838cfd9ddc26ff654b87e680bbcd8145dc2ead2a4d2fff30"}
Nov 01 03:22:21.455688 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:21.455660    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e6dcb6-x7gdw"
Nov 01 03:22:21.538845 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:21.538814    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94819519-6491-4744-9a04-c323de973006-kube-api-access-k87k6\" (UniqueName: \"kubernetes.io/projected/94819519-6491-4744-9a04-c323de973006-kube-api-access-k87k6\") pod \"94819519-6491-4744-9a04-c323de973006\" (UID: \"94819519-6491-4744-9a04-c323de973006\") "
Nov 01 03:22:21.539025 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:21.538867    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94819519-6491-4744-9a04-c323de973006-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94819519-6491-4744-9a04-c323de973006-maas-api-xks-ca\") pod \"94819519-6491-4744-9a04-c323de973006\" (UID: \"94819519-6491-4744-9a04-c323de973006\") "
Nov 01 03:22:21.539025 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:21.538933    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94819519-6491-4744-9a04-c323de973006-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94819519-6491-4744-9a04-c323de973006-maas-api-openshift-ca\") pod \"94819519-6491-4744-9a04-c323de973006\" (UID: \"94819519-6491-4744-9a04-c323de973006\") "
Nov 01 03:22:21.539350 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:21.539325    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94819519-6491-4744-9a04-c323de973006-maas-api-openshift-ca" pod "94819519-6491-4744-9a04-c323de973006" (UID: "94819519-6491-4744-9a04-c323de973006"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:22:21.541026 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:21.540999    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94819519-6491-4744-9a04-c323de973006-maas-api-xks-ca" pod "94819519-6491-4744-9a04-c323de973006" (UID: "94819519-6491-4744-9a04-c323de973006"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:22:21.541128 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:21.541101    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94819519-6491-4744-9a04-c323de973006-kube-api-access-k87k6" pod "94819519-6491-4744-9a04-c323de973006" (UID: "94819519-6491-4744-9a04-c323de973006"). InnerVolumeSpecName "kube-api-access-k87k6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:22:21.640491 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:21.640466    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94819519-6491-4744-9a04-c323de973006-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:21.640491 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:21.640491    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k87k6\" (UniqueName: \"kubernetes.io/projected/94819519-6491-4744-9a04-c323de973006-kube-api-access-k87k6\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:21.640670 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:21.640502    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94819519-6491-4744-9a04-c323de973006-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:22.321249 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:22.321215    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e6dcb6-x7gdw"
Nov 01 03:22:22.321249 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:22.321238    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69e6dcb6-x7gdw" event={"ID":"94819519-6491-4744-9a04-c323de973006","Type":"ContainerDied","Data":"9b9873d1c1fdc8035fa2a6e8ac88f37e18cacb530d6d385908083c0baa6922f7"}
Nov 01 03:22:22.321461 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:22.321290    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b9873d1c1fdc8035fa2a6e8ac88f37e18cacb530d6d385908083c0baa6922f7"
Nov 01 03:22:23.788077 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:23.788000    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:22:24.236655 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.236080    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh"
Nov 01 03:22:24.248624 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.248582    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n"
Nov 01 03:22:24.270514 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.270483    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"
Nov 01 03:22:24.339810 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.339780    4930 generic.go:356] "Generic (PLEG): container finished" podID="35b89d8d-ab16-41ea-8a99-bcc80e212838" containerID="1f5d54f9f7fbe2e03833c6dddedd330b6aba8f0cee43ba61d529a0683894b586" exitCode=0
Nov 01 03:22:24.340003 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.339847    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n"
Nov 01 03:22:24.340063 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.340008    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n" event={"ID":"35b89d8d-ab16-41ea-8a99-bcc80e212838","Type":"ContainerDied","Data":"1f5d54f9f7fbe2e03833c6dddedd330b6aba8f0cee43ba61d529a0683894b586"}
Nov 01 03:22:24.340063 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.340053    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n" event={"ID":"35b89d8d-ab16-41ea-8a99-bcc80e212838","Type":"ContainerDied","Data":"f18cf66c59f9a44cfbdacedd06ed893a8dbc949eb4fa7b0153377e59b936f876"}
Nov 01 03:22:24.340132 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.340083    4930 scope.go:122] "RemoveContainer" containerID="1f5d54f9f7fbe2e03833c6dddedd330b6aba8f0cee43ba61d529a0683894b586"
Nov 01 03:22:24.342464 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.342433    4930 generic.go:356] "Generic (PLEG): container finished" podID="97f3268e-176f-46ef-8431-0b3e985caa0f" containerID="71075b263b0356ade2cb8a16ed245ad3bf547c27d53d4691287cb9f8ab038f30" exitCode=0
Nov 01 03:22:24.342745 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.342727    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh"
Nov 01 03:22:24.342900 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.342878    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh" event={"ID":"97f3268e-176f-46ef-8431-0b3e985caa0f","Type":"ContainerDied","Data":"71075b263b0356ade2cb8a16ed245ad3bf547c27d53d4691287cb9f8ab038f30"}
Nov 01 03:22:24.342985 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.342925    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh" event={"ID":"97f3268e-176f-46ef-8431-0b3e985caa0f","Type":"ContainerDied","Data":"daa2bc383bb9dd49191ec3edbdae198f1f5d89b741f3827d76c4a30af6efb91e"}
Nov 01 03:22:24.345924 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.345846    4930 generic.go:356] "Generic (PLEG): container finished" podID="24e9b838-6a66-49b0-9aec-c43bfd9221ae" containerID="7da7748e972490cf4d1f1b2791d55b107b7bcf9eb09d19ece15ddc709ae82f77" exitCode=0
Nov 01 03:22:24.345924 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.345887    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5" event={"ID":"24e9b838-6a66-49b0-9aec-c43bfd9221ae","Type":"ContainerDied","Data":"7da7748e972490cf4d1f1b2791d55b107b7bcf9eb09d19ece15ddc709ae82f77"}
Nov 01 03:22:24.345924 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.345920    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5" event={"ID":"24e9b838-6a66-49b0-9aec-c43bfd9221ae","Type":"ContainerDied","Data":"a026cb953a800c68f752276f2786b3b3a714f4ee360e5d30ec995c28ccba09c7"}
Nov 01 03:22:24.346150 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.345982    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"
Nov 01 03:22:24.354193 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.354147    4930 scope.go:122] "RemoveContainer" containerID="1f5d54f9f7fbe2e03833c6dddedd330b6aba8f0cee43ba61d529a0683894b586"
Nov 01 03:22:24.354478 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:22:24.354434    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f5d54f9f7fbe2e03833c6dddedd330b6aba8f0cee43ba61d529a0683894b586\": container with ID starting with 1f5d54f9f7fbe2e03833c6dddedd330b6aba8f0cee43ba61d529a0683894b586 not found: ID does not exist" containerID="1f5d54f9f7fbe2e03833c6dddedd330b6aba8f0cee43ba61d529a0683894b586"
Nov 01 03:22:24.354584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.354473    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f5d54f9f7fbe2e03833c6dddedd330b6aba8f0cee43ba61d529a0683894b586"} err="failed to get container status \"1f5d54f9f7fbe2e03833c6dddedd330b6aba8f0cee43ba61d529a0683894b586\": rpc error: code = NotFound desc = could not find container \"1f5d54f9f7fbe2e03833c6dddedd330b6aba8f0cee43ba61d529a0683894b586\": container with ID starting with 1f5d54f9f7fbe2e03833c6dddedd330b6aba8f0cee43ba61d529a0683894b586 not found: ID does not exist"
Nov 01 03:22:24.354584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.354495    4930 scope.go:122] "RemoveContainer" containerID="71075b263b0356ade2cb8a16ed245ad3bf547c27d53d4691287cb9f8ab038f30"
Nov 01 03:22:24.372084 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.371668    4930 scope.go:122] "RemoveContainer" containerID="71075b263b0356ade2cb8a16ed245ad3bf547c27d53d4691287cb9f8ab038f30"
Nov 01 03:22:24.374902 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:22:24.374758    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71075b263b0356ade2cb8a16ed245ad3bf547c27d53d4691287cb9f8ab038f30\": container with ID starting with 71075b263b0356ade2cb8a16ed245ad3bf547c27d53d4691287cb9f8ab038f30 not found: ID does not exist" containerID="71075b263b0356ade2cb8a16ed245ad3bf547c27d53d4691287cb9f8ab038f30"
Nov 01 03:22:24.374902 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.374794    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71075b263b0356ade2cb8a16ed245ad3bf547c27d53d4691287cb9f8ab038f30"} err="failed to get container status \"71075b263b0356ade2cb8a16ed245ad3bf547c27d53d4691287cb9f8ab038f30\": rpc error: code = NotFound desc = could not find container \"71075b263b0356ade2cb8a16ed245ad3bf547c27d53d4691287cb9f8ab038f30\": container with ID starting with 71075b263b0356ade2cb8a16ed245ad3bf547c27d53d4691287cb9f8ab038f30 not found: ID does not exist"
Nov 01 03:22:24.374902 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.374816    4930 scope.go:122] "RemoveContainer" containerID="7da7748e972490cf4d1f1b2791d55b107b7bcf9eb09d19ece15ddc709ae82f77"
Nov 01 03:22:24.399970 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.399946    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97f3268e-176f-46ef-8431-0b3e985caa0f-istio-token\" (UniqueName: \"kubernetes.io/projected/97f3268e-176f-46ef-8431-0b3e985caa0f-istio-token\") pod \"97f3268e-176f-46ef-8431-0b3e985caa0f\" (UID: \"97f3268e-176f-46ef-8431-0b3e985caa0f\") "
Nov 01 03:22:24.400085 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.399991    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35b89d8d-ab16-41ea-8a99-bcc80e212838-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35b89d8d-ab16-41ea-8a99-bcc80e212838-istio-data\") pod \"35b89d8d-ab16-41ea-8a99-bcc80e212838\" (UID: \"35b89d8d-ab16-41ea-8a99-bcc80e212838\") "
Nov 01 03:22:24.400085 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.400040    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35b89d8d-ab16-41ea-8a99-bcc80e212838-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35b89d8d-ab16-41ea-8a99-bcc80e212838-istio-envoy\") pod \"35b89d8d-ab16-41ea-8a99-bcc80e212838\" (UID: \"35b89d8d-ab16-41ea-8a99-bcc80e212838\") "
Nov 01 03:22:24.400214 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.400122    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24e9b838-6a66-49b0-9aec-c43bfd9221ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24e9b838-6a66-49b0-9aec-c43bfd9221ae-istiod-ca-cert\") pod \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\" (UID: \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\") "
Nov 01 03:22:24.400214 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.400170    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35b89d8d-ab16-41ea-8a99-bcc80e212838-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35b89d8d-ab16-41ea-8a99-bcc80e212838-istiod-ca-cert\") pod \"35b89d8d-ab16-41ea-8a99-bcc80e212838\" (UID: \"35b89d8d-ab16-41ea-8a99-bcc80e212838\") "
Nov 01 03:22:24.400214 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.400198    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97f3268e-176f-46ef-8431-0b3e985caa0f-kube-api-access-zgx5g\" (UniqueName: \"kubernetes.io/projected/97f3268e-176f-46ef-8431-0b3e985caa0f-kube-api-access-zgx5g\") pod \"97f3268e-176f-46ef-8431-0b3e985caa0f\" (UID: \"97f3268e-176f-46ef-8431-0b3e985caa0f\") "
Nov 01 03:22:24.400364 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.400234    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24e9b838-6a66-49b0-9aec-c43bfd9221ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e9b838-6a66-49b0-9aec-c43bfd9221ae-credential-socket\") pod \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\" (UID: \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\") "
Nov 01 03:22:24.400364 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.400270    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24e9b838-6a66-49b0-9aec-c43bfd9221ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24e9b838-6a66-49b0-9aec-c43bfd9221ae-istio-podinfo\") pod \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\" (UID: \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\") "
Nov 01 03:22:24.400364 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.400327    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35b89d8d-ab16-41ea-8a99-bcc80e212838-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35b89d8d-ab16-41ea-8a99-bcc80e212838-istio-podinfo\") pod \"35b89d8d-ab16-41ea-8a99-bcc80e212838\" (UID: \"35b89d8d-ab16-41ea-8a99-bcc80e212838\") "
Nov 01 03:22:24.400510 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.400377    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24e9b838-6a66-49b0-9aec-c43bfd9221ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24e9b838-6a66-49b0-9aec-c43bfd9221ae-workload-certs\") pod \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\" (UID: \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\") "
Nov 01 03:22:24.400510 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.400422    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97f3268e-176f-46ef-8431-0b3e985caa0f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f3268e-176f-46ef-8431-0b3e985caa0f-workload-socket\") pod \"97f3268e-176f-46ef-8431-0b3e985caa0f\" (UID: \"97f3268e-176f-46ef-8431-0b3e985caa0f\") "
Nov 01 03:22:24.400510 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.400472    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35b89d8d-ab16-41ea-8a99-bcc80e212838-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35b89d8d-ab16-41ea-8a99-bcc80e212838-workload-socket\") pod \"35b89d8d-ab16-41ea-8a99-bcc80e212838\" (UID: \"35b89d8d-ab16-41ea-8a99-bcc80e212838\") "
Nov 01 03:22:24.401151 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.400511    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24e9b838-6a66-49b0-9aec-c43bfd9221ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24e9b838-6a66-49b0-9aec-c43bfd9221ae-istio-envoy\") pod \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\" (UID: \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\") "
Nov 01 03:22:24.401151 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.400582    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35b89d8d-ab16-41ea-8a99-bcc80e212838-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35b89d8d-ab16-41ea-8a99-bcc80e212838-credential-socket\") pod \"35b89d8d-ab16-41ea-8a99-bcc80e212838\" (UID: \"35b89d8d-ab16-41ea-8a99-bcc80e212838\") "
Nov 01 03:22:24.401151 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.400626    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97f3268e-176f-46ef-8431-0b3e985caa0f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97f3268e-176f-46ef-8431-0b3e985caa0f-istio-envoy\") pod \"97f3268e-176f-46ef-8431-0b3e985caa0f\" (UID: \"97f3268e-176f-46ef-8431-0b3e985caa0f\") "
Nov 01 03:22:24.401151 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.400659    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35b89d8d-ab16-41ea-8a99-bcc80e212838-kube-api-access-jm6sz\" (UniqueName: \"kubernetes.io/projected/35b89d8d-ab16-41ea-8a99-bcc80e212838-kube-api-access-jm6sz\") pod \"35b89d8d-ab16-41ea-8a99-bcc80e212838\" (UID: \"35b89d8d-ab16-41ea-8a99-bcc80e212838\") "
Nov 01 03:22:24.401151 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.400685    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24e9b838-6a66-49b0-9aec-c43bfd9221ae-istio-token\" (UniqueName: \"kubernetes.io/projected/24e9b838-6a66-49b0-9aec-c43bfd9221ae-istio-token\") pod \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\" (UID: \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\") "
Nov 01 03:22:24.401151 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.400722    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97f3268e-176f-46ef-8431-0b3e985caa0f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97f3268e-176f-46ef-8431-0b3e985caa0f-istio-data\") pod \"97f3268e-176f-46ef-8431-0b3e985caa0f\" (UID: \"97f3268e-176f-46ef-8431-0b3e985caa0f\") "
Nov 01 03:22:24.401151 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.400755    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97f3268e-176f-46ef-8431-0b3e985caa0f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f3268e-176f-46ef-8431-0b3e985caa0f-workload-certs\") pod \"97f3268e-176f-46ef-8431-0b3e985caa0f\" (UID: \"97f3268e-176f-46ef-8431-0b3e985caa0f\") "
Nov 01 03:22:24.401151 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.400789    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35b89d8d-ab16-41ea-8a99-bcc80e212838-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35b89d8d-ab16-41ea-8a99-bcc80e212838-workload-certs\") pod \"35b89d8d-ab16-41ea-8a99-bcc80e212838\" (UID: \"35b89d8d-ab16-41ea-8a99-bcc80e212838\") "
Nov 01 03:22:24.401151 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.400836    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24e9b838-6a66-49b0-9aec-c43bfd9221ae-kube-api-access-ts9nd\" (UniqueName: \"kubernetes.io/projected/24e9b838-6a66-49b0-9aec-c43bfd9221ae-kube-api-access-ts9nd\") pod \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\" (UID: \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\") "
Nov 01 03:22:24.401151 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.400863    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24e9b838-6a66-49b0-9aec-c43bfd9221ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e9b838-6a66-49b0-9aec-c43bfd9221ae-workload-socket\") pod \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\" (UID: \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\") "
Nov 01 03:22:24.401151 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.400903    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97f3268e-176f-46ef-8431-0b3e985caa0f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97f3268e-176f-46ef-8431-0b3e985caa0f-istio-podinfo\") pod \"97f3268e-176f-46ef-8431-0b3e985caa0f\" (UID: \"97f3268e-176f-46ef-8431-0b3e985caa0f\") "
Nov 01 03:22:24.401151 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.401114    4930 scope.go:122] "RemoveContainer" containerID="bf4395ebac7bfe9834d0aa8d5038db9353479279937c9e6d106f6e515b87399f"
Nov 01 03:22:24.402043 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.401983    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35b89d8d-ab16-41ea-8a99-bcc80e212838-istiod-ca-cert" pod "35b89d8d-ab16-41ea-8a99-bcc80e212838" (UID: "35b89d8d-ab16-41ea-8a99-bcc80e212838"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:22:24.403261 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.402880    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35b89d8d-ab16-41ea-8a99-bcc80e212838-workload-certs" pod "35b89d8d-ab16-41ea-8a99-bcc80e212838" (UID: "35b89d8d-ab16-41ea-8a99-bcc80e212838"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:22:24.403261 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.403060    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f3268e-176f-46ef-8431-0b3e985caa0f-istio-data" pod "97f3268e-176f-46ef-8431-0b3e985caa0f" (UID: "97f3268e-176f-46ef-8431-0b3e985caa0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:22:24.403261 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.403233    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f3268e-176f-46ef-8431-0b3e985caa0f-workload-certs" pod "97f3268e-176f-46ef-8431-0b3e985caa0f" (UID: "97f3268e-176f-46ef-8431-0b3e985caa0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:22:24.403932 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.403649    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35b89d8d-ab16-41ea-8a99-bcc80e212838-istio-token\" (UniqueName: \"kubernetes.io/projected/35b89d8d-ab16-41ea-8a99-bcc80e212838-istio-token\") pod \"35b89d8d-ab16-41ea-8a99-bcc80e212838\" (UID: \"35b89d8d-ab16-41ea-8a99-bcc80e212838\") "
Nov 01 03:22:24.403932 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.403700    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24e9b838-6a66-49b0-9aec-c43bfd9221ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24e9b838-6a66-49b0-9aec-c43bfd9221ae-istio-data\") pod \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\" (UID: \"24e9b838-6a66-49b0-9aec-c43bfd9221ae\") "
Nov 01 03:22:24.403932 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.403737    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97f3268e-176f-46ef-8431-0b3e985caa0f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97f3268e-176f-46ef-8431-0b3e985caa0f-istiod-ca-cert\") pod \"97f3268e-176f-46ef-8431-0b3e985caa0f\" (UID: \"97f3268e-176f-46ef-8431-0b3e985caa0f\") "
Nov 01 03:22:24.403932 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.403762    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97f3268e-176f-46ef-8431-0b3e985caa0f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f3268e-176f-46ef-8431-0b3e985caa0f-credential-socket\") pod \"97f3268e-176f-46ef-8431-0b3e985caa0f\" (UID: \"97f3268e-176f-46ef-8431-0b3e985caa0f\") "
Nov 01 03:22:24.404501 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.404321    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97f3268e-176f-46ef-8431-0b3e985caa0f-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:24.404501 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.404341    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f3268e-176f-46ef-8431-0b3e985caa0f-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:24.404501 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.404357    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35b89d8d-ab16-41ea-8a99-bcc80e212838-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:24.404501 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.404373    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35b89d8d-ab16-41ea-8a99-bcc80e212838-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:24.407772 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.407747    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f3268e-176f-46ef-8431-0b3e985caa0f-workload-socket" pod "97f3268e-176f-46ef-8431-0b3e985caa0f" (UID: "97f3268e-176f-46ef-8431-0b3e985caa0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:22:24.408065 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.407984    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24e9b838-6a66-49b0-9aec-c43bfd9221ae-workload-certs" pod "24e9b838-6a66-49b0-9aec-c43bfd9221ae" (UID: "24e9b838-6a66-49b0-9aec-c43bfd9221ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:22:24.408196 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.408174    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24e9b838-6a66-49b0-9aec-c43bfd9221ae-workload-socket" pod "24e9b838-6a66-49b0-9aec-c43bfd9221ae" (UID: "24e9b838-6a66-49b0-9aec-c43bfd9221ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:22:24.408380 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.408356    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35b89d8d-ab16-41ea-8a99-bcc80e212838-credential-socket" pod "35b89d8d-ab16-41ea-8a99-bcc80e212838" (UID: "35b89d8d-ab16-41ea-8a99-bcc80e212838"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:22:24.408834 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.408811    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35b89d8d-ab16-41ea-8a99-bcc80e212838-workload-socket" pod "35b89d8d-ab16-41ea-8a99-bcc80e212838" (UID: "35b89d8d-ab16-41ea-8a99-bcc80e212838"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:22:24.408940 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.408868    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f3268e-176f-46ef-8431-0b3e985caa0f-credential-socket" pod "97f3268e-176f-46ef-8431-0b3e985caa0f" (UID: "97f3268e-176f-46ef-8431-0b3e985caa0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:22:24.411294 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.411270    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24e9b838-6a66-49b0-9aec-c43bfd9221ae-credential-socket" pod "24e9b838-6a66-49b0-9aec-c43bfd9221ae" (UID: "24e9b838-6a66-49b0-9aec-c43bfd9221ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:22:24.416384 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.413466    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35b89d8d-ab16-41ea-8a99-bcc80e212838-istio-podinfo" pod "35b89d8d-ab16-41ea-8a99-bcc80e212838" (UID: "35b89d8d-ab16-41ea-8a99-bcc80e212838"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:22:24.416384 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.413778    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97f3268e-176f-46ef-8431-0b3e985caa0f-istiod-ca-cert" pod "97f3268e-176f-46ef-8431-0b3e985caa0f" (UID: "97f3268e-176f-46ef-8431-0b3e985caa0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:22:24.416384 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.413844    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97f3268e-176f-46ef-8431-0b3e985caa0f-istio-podinfo" pod "97f3268e-176f-46ef-8431-0b3e985caa0f" (UID: "97f3268e-176f-46ef-8431-0b3e985caa0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:22:24.416384 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.414878    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24e9b838-6a66-49b0-9aec-c43bfd9221ae-istiod-ca-cert" pod "24e9b838-6a66-49b0-9aec-c43bfd9221ae" (UID: "24e9b838-6a66-49b0-9aec-c43bfd9221ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:22:24.417060 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.417036    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35b89d8d-ab16-41ea-8a99-bcc80e212838-istio-data" pod "35b89d8d-ab16-41ea-8a99-bcc80e212838" (UID: "35b89d8d-ab16-41ea-8a99-bcc80e212838"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:22:24.417208 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.417190    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24e9b838-6a66-49b0-9aec-c43bfd9221ae-istio-data" pod "24e9b838-6a66-49b0-9aec-c43bfd9221ae" (UID: "24e9b838-6a66-49b0-9aec-c43bfd9221ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:22:24.417363 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.417349    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35b89d8d-ab16-41ea-8a99-bcc80e212838-istio-token" pod "35b89d8d-ab16-41ea-8a99-bcc80e212838" (UID: "35b89d8d-ab16-41ea-8a99-bcc80e212838"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:22:24.435081 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.434684    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f3268e-176f-46ef-8431-0b3e985caa0f-istio-token" pod "97f3268e-176f-46ef-8431-0b3e985caa0f" (UID: "97f3268e-176f-46ef-8431-0b3e985caa0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:22:24.435081 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.434751    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f3268e-176f-46ef-8431-0b3e985caa0f-kube-api-access-zgx5g" pod "97f3268e-176f-46ef-8431-0b3e985caa0f" (UID: "97f3268e-176f-46ef-8431-0b3e985caa0f"). InnerVolumeSpecName "kube-api-access-zgx5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:22:24.435081 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.434800    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24e9b838-6a66-49b0-9aec-c43bfd9221ae-istio-podinfo" pod "24e9b838-6a66-49b0-9aec-c43bfd9221ae" (UID: "24e9b838-6a66-49b0-9aec-c43bfd9221ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:22:24.435373 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.435356    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24e9b838-6a66-49b0-9aec-c43bfd9221ae-istio-token" pod "24e9b838-6a66-49b0-9aec-c43bfd9221ae" (UID: "24e9b838-6a66-49b0-9aec-c43bfd9221ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:22:24.443657 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.443613    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24e9b838-6a66-49b0-9aec-c43bfd9221ae-istio-envoy" pod "24e9b838-6a66-49b0-9aec-c43bfd9221ae" (UID: "24e9b838-6a66-49b0-9aec-c43bfd9221ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:22:24.446853 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.446733    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35b89d8d-ab16-41ea-8a99-bcc80e212838-kube-api-access-jm6sz" pod "35b89d8d-ab16-41ea-8a99-bcc80e212838" (UID: "35b89d8d-ab16-41ea-8a99-bcc80e212838"). InnerVolumeSpecName "kube-api-access-jm6sz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:22:24.446853 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.446745    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f3268e-176f-46ef-8431-0b3e985caa0f-istio-envoy" pod "97f3268e-176f-46ef-8431-0b3e985caa0f" (UID: "97f3268e-176f-46ef-8431-0b3e985caa0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:22:24.446853 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.446814    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24e9b838-6a66-49b0-9aec-c43bfd9221ae-kube-api-access-ts9nd" pod "24e9b838-6a66-49b0-9aec-c43bfd9221ae" (UID: "24e9b838-6a66-49b0-9aec-c43bfd9221ae"). InnerVolumeSpecName "kube-api-access-ts9nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:22:24.447099 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.447085    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35b89d8d-ab16-41ea-8a99-bcc80e212838-istio-envoy" pod "35b89d8d-ab16-41ea-8a99-bcc80e212838" (UID: "35b89d8d-ab16-41ea-8a99-bcc80e212838"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:22:24.463031 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.462058    4930 scope.go:122] "RemoveContainer" containerID="7da7748e972490cf4d1f1b2791d55b107b7bcf9eb09d19ece15ddc709ae82f77"
Nov 01 03:22:24.463031 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:22:24.462416    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7da7748e972490cf4d1f1b2791d55b107b7bcf9eb09d19ece15ddc709ae82f77\": container with ID starting with 7da7748e972490cf4d1f1b2791d55b107b7bcf9eb09d19ece15ddc709ae82f77 not found: ID does not exist" containerID="7da7748e972490cf4d1f1b2791d55b107b7bcf9eb09d19ece15ddc709ae82f77"
Nov 01 03:22:24.463031 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.462449    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7da7748e972490cf4d1f1b2791d55b107b7bcf9eb09d19ece15ddc709ae82f77"} err="failed to get container status \"7da7748e972490cf4d1f1b2791d55b107b7bcf9eb09d19ece15ddc709ae82f77\": rpc error: code = NotFound desc = could not find container \"7da7748e972490cf4d1f1b2791d55b107b7bcf9eb09d19ece15ddc709ae82f77\": container with ID starting with 7da7748e972490cf4d1f1b2791d55b107b7bcf9eb09d19ece15ddc709ae82f77 not found: ID does not exist"
Nov 01 03:22:24.463031 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.462471    4930 scope.go:122] "RemoveContainer" containerID="bf4395ebac7bfe9834d0aa8d5038db9353479279937c9e6d106f6e515b87399f"
Nov 01 03:22:24.463031 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:22:24.462756    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf4395ebac7bfe9834d0aa8d5038db9353479279937c9e6d106f6e515b87399f\": container with ID starting with bf4395ebac7bfe9834d0aa8d5038db9353479279937c9e6d106f6e515b87399f not found: ID does not exist" containerID="bf4395ebac7bfe9834d0aa8d5038db9353479279937c9e6d106f6e515b87399f"
Nov 01 03:22:24.463031 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.462783    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf4395ebac7bfe9834d0aa8d5038db9353479279937c9e6d106f6e515b87399f"} err="failed to get container status \"bf4395ebac7bfe9834d0aa8d5038db9353479279937c9e6d106f6e515b87399f\": rpc error: code = NotFound desc = could not find container \"bf4395ebac7bfe9834d0aa8d5038db9353479279937c9e6d106f6e515b87399f\": container with ID starting with bf4395ebac7bfe9834d0aa8d5038db9353479279937c9e6d106f6e515b87399f not found: ID does not exist"
Nov 01 03:22:24.516346 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.513348    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24e9b838-6a66-49b0-9aec-c43bfd9221ae-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:24.516346 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.513382    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgx5g\" (UniqueName: \"kubernetes.io/projected/97f3268e-176f-46ef-8431-0b3e985caa0f-kube-api-access-zgx5g\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:24.516346 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.513397    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e9b838-6a66-49b0-9aec-c43bfd9221ae-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:24.516346 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.513410    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24e9b838-6a66-49b0-9aec-c43bfd9221ae-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:24.516346 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.513424    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35b89d8d-ab16-41ea-8a99-bcc80e212838-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:24.516346 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.513439    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24e9b838-6a66-49b0-9aec-c43bfd9221ae-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:24.516346 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.513451    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f3268e-176f-46ef-8431-0b3e985caa0f-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:24.516346 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.513466    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35b89d8d-ab16-41ea-8a99-bcc80e212838-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:24.516346 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.513479    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24e9b838-6a66-49b0-9aec-c43bfd9221ae-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:24.516346 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.513491    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35b89d8d-ab16-41ea-8a99-bcc80e212838-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:24.516346 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.513504    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97f3268e-176f-46ef-8431-0b3e985caa0f-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:24.516346 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.513518    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm6sz\" (UniqueName: \"kubernetes.io/projected/35b89d8d-ab16-41ea-8a99-bcc80e212838-kube-api-access-jm6sz\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:24.516346 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.513530    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24e9b838-6a66-49b0-9aec-c43bfd9221ae-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:24.516346 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.513545    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts9nd\" (UniqueName: \"kubernetes.io/projected/24e9b838-6a66-49b0-9aec-c43bfd9221ae-kube-api-access-ts9nd\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:24.516346 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.513576    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e9b838-6a66-49b0-9aec-c43bfd9221ae-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:24.516346 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.513589    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97f3268e-176f-46ef-8431-0b3e985caa0f-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:24.516346 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.513600    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35b89d8d-ab16-41ea-8a99-bcc80e212838-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:24.516346 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.513612    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24e9b838-6a66-49b0-9aec-c43bfd9221ae-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:24.516346 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.513625    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97f3268e-176f-46ef-8431-0b3e985caa0f-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:24.516346 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.513639    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f3268e-176f-46ef-8431-0b3e985caa0f-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:24.516346 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.513651    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97f3268e-176f-46ef-8431-0b3e985caa0f-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:24.516346 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.513664    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35b89d8d-ab16-41ea-8a99-bcc80e212838-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:24.516346 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.513676    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35b89d8d-ab16-41ea-8a99-bcc80e212838-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:24.923544 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.923473    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh"]
Nov 01 03:22:24.926811 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:24.926786    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xm5sh"]
Nov 01 03:22:25.034517 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:25.034423    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"]
Nov 01 03:22:25.047955 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:25.047922    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zscr5"]
Nov 01 03:22:25.153092 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:25.153052    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n"]
Nov 01 03:22:25.157306 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:25.157278    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-4sm7n"]
Nov 01 03:22:25.623400 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:25.623364    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24e9b838-6a66-49b0-9aec-c43bfd9221ae" path="/var/lib/kubelet/pods/24e9b838-6a66-49b0-9aec-c43bfd9221ae/volumes"
Nov 01 03:22:25.623952 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:25.623930    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35b89d8d-ab16-41ea-8a99-bcc80e212838" path="/var/lib/kubelet/pods/35b89d8d-ab16-41ea-8a99-bcc80e212838/volumes"
Nov 01 03:22:25.624809 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:25.624787    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97f3268e-176f-46ef-8431-0b3e985caa0f" path="/var/lib/kubelet/pods/97f3268e-176f-46ef-8431-0b3e985caa0f/volumes"
Nov 01 03:22:25.952408 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:25.952318    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fba185c6-kdrjn"]
Nov 01 03:22:25.953005 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:25.952984    4930 cpu_manager.go:414] "containerMap: removing container" podUID="94819519-6491-4744-9a04-c323de973006" containerName="revoke-keys"
Nov 01 03:22:25.953096 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:25.953008    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94819519-6491-4744-9a04-c323de973006" containerName="revoke-keys"
Nov 01 03:22:25.953096 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:25.953027    4930 cpu_manager.go:414] "containerMap: removing container" podUID="24e9b838-6a66-49b0-9aec-c43bfd9221ae" containerName="istio-proxy"
Nov 01 03:22:25.953096 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:25.953036    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24e9b838-6a66-49b0-9aec-c43bfd9221ae" containerName="istio-proxy"
Nov 01 03:22:25.953096 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:25.953061    4930 cpu_manager.go:414] "containerMap: removing container" podUID="35b89d8d-ab16-41ea-8a99-bcc80e212838" containerName="istio-proxy"
Nov 01 03:22:25.953096 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:25.953069    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35b89d8d-ab16-41ea-8a99-bcc80e212838" containerName="istio-proxy"
Nov 01 03:22:25.953096 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:25.953082    4930 cpu_manager.go:414] "containerMap: removing container" podUID="24e9b838-6a66-49b0-9aec-c43bfd9221ae" containerName="istio-proxy"
Nov 01 03:22:25.953096 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:25.953090    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24e9b838-6a66-49b0-9aec-c43bfd9221ae" containerName="istio-proxy"
Nov 01 03:22:25.953425 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:25.953106    4930 cpu_manager.go:414] "containerMap: removing container" podUID="aa78a11f-6052-4a54-aabf-dc93c44310d5" containerName="revoke-keys"
Nov 01 03:22:25.953425 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:25.953113    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa78a11f-6052-4a54-aabf-dc93c44310d5" containerName="revoke-keys"
Nov 01 03:22:25.953425 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:25.953135    4930 cpu_manager.go:414] "containerMap: removing container" podUID="97f3268e-176f-46ef-8431-0b3e985caa0f" containerName="istio-proxy"
Nov 01 03:22:25.953425 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:25.953142    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97f3268e-176f-46ef-8431-0b3e985caa0f" containerName="istio-proxy"
Nov 01 03:22:25.953425 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:25.953257    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="24e9b838-6a66-49b0-9aec-c43bfd9221ae" containerName="istio-proxy"
Nov 01 03:22:25.953425 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:25.953274    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="35b89d8d-ab16-41ea-8a99-bcc80e212838" containerName="istio-proxy"
Nov 01 03:22:25.953425 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:25.953284    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="24e9b838-6a66-49b0-9aec-c43bfd9221ae" containerName="istio-proxy"
Nov 01 03:22:25.953425 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:25.953294    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa78a11f-6052-4a54-aabf-dc93c44310d5" containerName="revoke-keys"
Nov 01 03:22:25.953425 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:25.953305    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="97f3268e-176f-46ef-8431-0b3e985caa0f" containerName="istio-proxy"
Nov 01 03:22:25.953425 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:25.953317    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="94819519-6491-4744-9a04-c323de973006" containerName="revoke-keys"
Nov 01 03:22:25.958348 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:25.958326    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba185c6-kdrjn"
Nov 01 03:22:26.019497 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:26.018943    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fba185c6-kdrjn"]
Nov 01 03:22:26.052183 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:26.052154    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkkv6\" (UniqueName: \"kubernetes.io/projected/8ea59bb2-5f2c-439a-8407-55c99376a22b-kube-api-access-nkkv6\") pod \"maas-api-revoke-sub-fba185c6-kdrjn\" (UID: \"8ea59bb2-5f2c-439a-8407-55c99376a22b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba185c6-kdrjn"
Nov 01 03:22:26.052339 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:26.052299    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ea59bb2-5f2c-439a-8407-55c99376a22b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fba185c6-kdrjn\" (UID: \"8ea59bb2-5f2c-439a-8407-55c99376a22b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba185c6-kdrjn"
Nov 01 03:22:26.052339 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:26.052325    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ea59bb2-5f2c-439a-8407-55c99376a22b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fba185c6-kdrjn\" (UID: \"8ea59bb2-5f2c-439a-8407-55c99376a22b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba185c6-kdrjn"
Nov 01 03:22:26.060392 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:26.060340    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:22:26.060392 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:26.060343    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:22:26.060615 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:26.060459    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:22:26.060749 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:26.060731    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:22:26.153690 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:26.153649    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ea59bb2-5f2c-439a-8407-55c99376a22b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fba185c6-kdrjn\" (UID: \"8ea59bb2-5f2c-439a-8407-55c99376a22b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba185c6-kdrjn"
Nov 01 03:22:26.154319 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:26.154294    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ea59bb2-5f2c-439a-8407-55c99376a22b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fba185c6-kdrjn\" (UID: \"8ea59bb2-5f2c-439a-8407-55c99376a22b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba185c6-kdrjn"
Nov 01 03:22:26.154434 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:26.154373    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ea59bb2-5f2c-439a-8407-55c99376a22b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fba185c6-kdrjn\" (UID: \"8ea59bb2-5f2c-439a-8407-55c99376a22b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba185c6-kdrjn"
Nov 01 03:22:26.154499 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:26.154484    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkkv6\" (UniqueName: \"kubernetes.io/projected/8ea59bb2-5f2c-439a-8407-55c99376a22b-kube-api-access-nkkv6\") pod \"maas-api-revoke-sub-fba185c6-kdrjn\" (UID: \"8ea59bb2-5f2c-439a-8407-55c99376a22b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba185c6-kdrjn"
Nov 01 03:22:26.157122 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:26.157091    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ea59bb2-5f2c-439a-8407-55c99376a22b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fba185c6-kdrjn\" (UID: \"8ea59bb2-5f2c-439a-8407-55c99376a22b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba185c6-kdrjn"
Nov 01 03:22:26.220483 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:26.220407    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkkv6\" (UniqueName: \"kubernetes.io/projected/8ea59bb2-5f2c-439a-8407-55c99376a22b-kube-api-access-nkkv6\") pod \"maas-api-revoke-sub-fba185c6-kdrjn\" (UID: \"8ea59bb2-5f2c-439a-8407-55c99376a22b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba185c6-kdrjn"
Nov 01 03:22:26.275139 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:26.275098    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba185c6-kdrjn"
Nov 01 03:22:26.667415 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:26.667386    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fba185c6-kdrjn"]
Nov 01 03:22:27.390652 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:27.390615    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba185c6-kdrjn" event={"ID":"8ea59bb2-5f2c-439a-8407-55c99376a22b","Type":"ContainerStarted","Data":"2a6dfd371c11f0e968ad6c03efe78333b31774ac158ce3e54933f448c2bfc64d"}
Nov 01 03:22:27.391223 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:27.390693    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba185c6-kdrjn" event={"ID":"8ea59bb2-5f2c-439a-8407-55c99376a22b","Type":"ContainerStarted","Data":"f28d83cefc8749a6d827f5f42a0d5d951001559abe4c289c03f9fcaac55dd5c9"}
Nov 01 03:22:27.508593 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:27.508510    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba185c6-kdrjn" podStartSLOduration=2.5084844840000002 podStartE2EDuration="2.508484484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:22:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:22:27.508254251 +0000 UTC m=+952.031260844" watchObservedRunningTime="2026-11-01 03:22:27.508484484 +0000 UTC m=+952.031491072"
Nov 01 03:22:28.400042 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:28.400004    4930 generic.go:356] "Generic (PLEG): container finished" podID="8ea59bb2-5f2c-439a-8407-55c99376a22b" containerID="2a6dfd371c11f0e968ad6c03efe78333b31774ac158ce3e54933f448c2bfc64d" exitCode=0
Nov 01 03:22:28.400410 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:28.400091    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba185c6-kdrjn" event={"ID":"8ea59bb2-5f2c-439a-8407-55c99376a22b","Type":"ContainerDied","Data":"2a6dfd371c11f0e968ad6c03efe78333b31774ac158ce3e54933f448c2bfc64d"}
Nov 01 03:22:28.783370 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:28.783291    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:22:29.558821 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:29.558794    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba185c6-kdrjn"
Nov 01 03:22:29.593037 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:29.593005    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ea59bb2-5f2c-439a-8407-55c99376a22b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ea59bb2-5f2c-439a-8407-55c99376a22b-maas-api-openshift-ca\") pod \"8ea59bb2-5f2c-439a-8407-55c99376a22b\" (UID: \"8ea59bb2-5f2c-439a-8407-55c99376a22b\") "
Nov 01 03:22:29.593186 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:29.593085    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ea59bb2-5f2c-439a-8407-55c99376a22b-kube-api-access-nkkv6\" (UniqueName: \"kubernetes.io/projected/8ea59bb2-5f2c-439a-8407-55c99376a22b-kube-api-access-nkkv6\") pod \"8ea59bb2-5f2c-439a-8407-55c99376a22b\" (UID: \"8ea59bb2-5f2c-439a-8407-55c99376a22b\") "
Nov 01 03:22:29.593186 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:29.593155    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ea59bb2-5f2c-439a-8407-55c99376a22b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ea59bb2-5f2c-439a-8407-55c99376a22b-maas-api-xks-ca\") pod \"8ea59bb2-5f2c-439a-8407-55c99376a22b\" (UID: \"8ea59bb2-5f2c-439a-8407-55c99376a22b\") "
Nov 01 03:22:29.593363 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:29.593338    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ea59bb2-5f2c-439a-8407-55c99376a22b-maas-api-openshift-ca" pod "8ea59bb2-5f2c-439a-8407-55c99376a22b" (UID: "8ea59bb2-5f2c-439a-8407-55c99376a22b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:22:29.593660 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:29.593638    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ea59bb2-5f2c-439a-8407-55c99376a22b-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:29.595148 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:29.595123    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ea59bb2-5f2c-439a-8407-55c99376a22b-maas-api-xks-ca" pod "8ea59bb2-5f2c-439a-8407-55c99376a22b" (UID: "8ea59bb2-5f2c-439a-8407-55c99376a22b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:22:29.595241 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:29.595204    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea59bb2-5f2c-439a-8407-55c99376a22b-kube-api-access-nkkv6" pod "8ea59bb2-5f2c-439a-8407-55c99376a22b" (UID: "8ea59bb2-5f2c-439a-8407-55c99376a22b"). InnerVolumeSpecName "kube-api-access-nkkv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:22:29.694826 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:29.694740    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ea59bb2-5f2c-439a-8407-55c99376a22b-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:29.694826 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:29.694767    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkkv6\" (UniqueName: \"kubernetes.io/projected/8ea59bb2-5f2c-439a-8407-55c99376a22b-kube-api-access-nkkv6\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:30.414749 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:30.414719    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba185c6-kdrjn"
Nov 01 03:22:30.415058 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:30.414762    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fba185c6-kdrjn" event={"ID":"8ea59bb2-5f2c-439a-8407-55c99376a22b","Type":"ContainerDied","Data":"f28d83cefc8749a6d827f5f42a0d5d951001559abe4c289c03f9fcaac55dd5c9"}
Nov 01 03:22:30.415058 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:30.414809    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f28d83cefc8749a6d827f5f42a0d5d951001559abe4c289c03f9fcaac55dd5c9"
Nov 01 03:22:30.909905 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:30.909862    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:22:31.901294 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:31.901262    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c3bbbf0-flzbv"]
Nov 01 03:22:31.901739 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:31.901725    4930 cpu_manager.go:414] "containerMap: removing container" podUID="8ea59bb2-5f2c-439a-8407-55c99376a22b" containerName="revoke-keys"
Nov 01 03:22:31.901739 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:31.901740    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ea59bb2-5f2c-439a-8407-55c99376a22b" containerName="revoke-keys"
Nov 01 03:22:31.901849 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:31.901839    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ea59bb2-5f2c-439a-8407-55c99376a22b" containerName="revoke-keys"
Nov 01 03:22:31.905429 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:31.905413    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c3bbbf0-flzbv"
Nov 01 03:22:31.919594 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:31.919568    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da0448da-122c-47c5-aa81-b3c5cc5193c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c3bbbf0-flzbv\" (UID: \"da0448da-122c-47c5-aa81-b3c5cc5193c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c3bbbf0-flzbv"
Nov 01 03:22:31.919888 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:31.919622    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da0448da-122c-47c5-aa81-b3c5cc5193c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c3bbbf0-flzbv\" (UID: \"da0448da-122c-47c5-aa81-b3c5cc5193c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c3bbbf0-flzbv"
Nov 01 03:22:31.919888 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:31.919827    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qdm6\" (UniqueName: \"kubernetes.io/projected/da0448da-122c-47c5-aa81-b3c5cc5193c3-kube-api-access-8qdm6\") pod \"maas-api-revoke-sub-6c3bbbf0-flzbv\" (UID: \"da0448da-122c-47c5-aa81-b3c5cc5193c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c3bbbf0-flzbv"
Nov 01 03:22:31.960078 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:31.960049    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:22:31.960078 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:31.960070    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:22:32.009778 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:32.009754    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:22:32.009778 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:32.009757    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:22:32.012945 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:32.012922    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c3bbbf0-flzbv"]
Nov 01 03:22:32.020615 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:32.020599    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da0448da-122c-47c5-aa81-b3c5cc5193c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c3bbbf0-flzbv\" (UID: \"da0448da-122c-47c5-aa81-b3c5cc5193c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c3bbbf0-flzbv"
Nov 01 03:22:32.020695 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:32.020645    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da0448da-122c-47c5-aa81-b3c5cc5193c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c3bbbf0-flzbv\" (UID: \"da0448da-122c-47c5-aa81-b3c5cc5193c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c3bbbf0-flzbv"
Nov 01 03:22:32.020752 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:32.020726    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qdm6\" (UniqueName: \"kubernetes.io/projected/da0448da-122c-47c5-aa81-b3c5cc5193c3-kube-api-access-8qdm6\") pod \"maas-api-revoke-sub-6c3bbbf0-flzbv\" (UID: \"da0448da-122c-47c5-aa81-b3c5cc5193c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c3bbbf0-flzbv"
Nov 01 03:22:32.021362 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:32.021340    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da0448da-122c-47c5-aa81-b3c5cc5193c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c3bbbf0-flzbv\" (UID: \"da0448da-122c-47c5-aa81-b3c5cc5193c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c3bbbf0-flzbv"
Nov 01 03:22:32.023237 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:32.023210    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da0448da-122c-47c5-aa81-b3c5cc5193c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c3bbbf0-flzbv\" (UID: \"da0448da-122c-47c5-aa81-b3c5cc5193c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c3bbbf0-flzbv"
Nov 01 03:22:32.082158 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:32.082117    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qdm6\" (UniqueName: \"kubernetes.io/projected/da0448da-122c-47c5-aa81-b3c5cc5193c3-kube-api-access-8qdm6\") pod \"maas-api-revoke-sub-6c3bbbf0-flzbv\" (UID: \"da0448da-122c-47c5-aa81-b3c5cc5193c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c3bbbf0-flzbv"
Nov 01 03:22:32.218471 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:32.218379    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c3bbbf0-flzbv"
Nov 01 03:22:32.597738 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:32.597710    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c3bbbf0-flzbv"]
Nov 01 03:22:32.598131 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:22:32.598098    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda0448da_122c_47c5_aa81_b3c5cc5193c3.slice/crio-d0833feff68a227027448426dce3da794ac712a1cab3fcdbf90f21874544a56f WatchSource:0}: Error finding container d0833feff68a227027448426dce3da794ac712a1cab3fcdbf90f21874544a56f: Status 404 returned error can't find the container with id d0833feff68a227027448426dce3da794ac712a1cab3fcdbf90f21874544a56f
Nov 01 03:22:33.437259 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:33.437220    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c3bbbf0-flzbv" event={"ID":"da0448da-122c-47c5-aa81-b3c5cc5193c3","Type":"ContainerStarted","Data":"5e9bd79aad194779114270da6c2341c46d670157ceeb244bfa6ecd7282ca5546"}
Nov 01 03:22:33.437801 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:33.437298    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c3bbbf0-flzbv" event={"ID":"da0448da-122c-47c5-aa81-b3c5cc5193c3","Type":"ContainerStarted","Data":"d0833feff68a227027448426dce3da794ac712a1cab3fcdbf90f21874544a56f"}
Nov 01 03:22:33.650920 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:33.650890    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:22:34.444805 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:34.444771    4930 generic.go:356] "Generic (PLEG): container finished" podID="da0448da-122c-47c5-aa81-b3c5cc5193c3" containerID="5e9bd79aad194779114270da6c2341c46d670157ceeb244bfa6ecd7282ca5546" exitCode=0
Nov 01 03:22:34.445204 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:34.444854    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c3bbbf0-flzbv" event={"ID":"da0448da-122c-47c5-aa81-b3c5cc5193c3","Type":"ContainerDied","Data":"5e9bd79aad194779114270da6c2341c46d670157ceeb244bfa6ecd7282ca5546"}
Nov 01 03:22:35.605988 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:35.605962    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c3bbbf0-flzbv"
Nov 01 03:22:35.661168 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:35.661146    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da0448da-122c-47c5-aa81-b3c5cc5193c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da0448da-122c-47c5-aa81-b3c5cc5193c3-maas-api-openshift-ca\") pod \"da0448da-122c-47c5-aa81-b3c5cc5193c3\" (UID: \"da0448da-122c-47c5-aa81-b3c5cc5193c3\") "
Nov 01 03:22:35.661302 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:35.661215    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da0448da-122c-47c5-aa81-b3c5cc5193c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da0448da-122c-47c5-aa81-b3c5cc5193c3-maas-api-xks-ca\") pod \"da0448da-122c-47c5-aa81-b3c5cc5193c3\" (UID: \"da0448da-122c-47c5-aa81-b3c5cc5193c3\") "
Nov 01 03:22:35.661302 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:35.661255    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da0448da-122c-47c5-aa81-b3c5cc5193c3-kube-api-access-8qdm6\" (UniqueName: \"kubernetes.io/projected/da0448da-122c-47c5-aa81-b3c5cc5193c3-kube-api-access-8qdm6\") pod \"da0448da-122c-47c5-aa81-b3c5cc5193c3\" (UID: \"da0448da-122c-47c5-aa81-b3c5cc5193c3\") "
Nov 01 03:22:35.661454 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:35.661427    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da0448da-122c-47c5-aa81-b3c5cc5193c3-maas-api-openshift-ca" pod "da0448da-122c-47c5-aa81-b3c5cc5193c3" (UID: "da0448da-122c-47c5-aa81-b3c5cc5193c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:22:35.661751 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:35.661730    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da0448da-122c-47c5-aa81-b3c5cc5193c3-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:35.663150 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:35.663131    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da0448da-122c-47c5-aa81-b3c5cc5193c3-maas-api-xks-ca" pod "da0448da-122c-47c5-aa81-b3c5cc5193c3" (UID: "da0448da-122c-47c5-aa81-b3c5cc5193c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:22:35.663351 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:35.663332    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da0448da-122c-47c5-aa81-b3c5cc5193c3-kube-api-access-8qdm6" pod "da0448da-122c-47c5-aa81-b3c5cc5193c3" (UID: "da0448da-122c-47c5-aa81-b3c5cc5193c3"). InnerVolumeSpecName "kube-api-access-8qdm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:22:35.762893 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:35.762803    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qdm6\" (UniqueName: \"kubernetes.io/projected/da0448da-122c-47c5-aa81-b3c5cc5193c3-kube-api-access-8qdm6\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:35.762893 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:35.762831    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da0448da-122c-47c5-aa81-b3c5cc5193c3-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:36.114724 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:36.114691    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9305848-8fqmz"]
Nov 01 03:22:36.115167 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:36.115154    4930 cpu_manager.go:414] "containerMap: removing container" podUID="da0448da-122c-47c5-aa81-b3c5cc5193c3" containerName="revoke-keys"
Nov 01 03:22:36.115214 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:36.115169    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da0448da-122c-47c5-aa81-b3c5cc5193c3" containerName="revoke-keys"
Nov 01 03:22:36.115277 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:36.115267    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="da0448da-122c-47c5-aa81-b3c5cc5193c3" containerName="revoke-keys"
Nov 01 03:22:36.119994 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:36.119973    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9305848-8fqmz"
Nov 01 03:22:36.167303 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:36.167271    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvqk7\" (UniqueName: \"kubernetes.io/projected/b9008621-ca43-429a-9d79-7bcf46ebbd55-kube-api-access-dvqk7\") pod \"maas-api-revoke-sub-e9305848-8fqmz\" (UID: \"b9008621-ca43-429a-9d79-7bcf46ebbd55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9305848-8fqmz"
Nov 01 03:22:36.167452 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:36.167388    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9008621-ca43-429a-9d79-7bcf46ebbd55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9305848-8fqmz\" (UID: \"b9008621-ca43-429a-9d79-7bcf46ebbd55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9305848-8fqmz"
Nov 01 03:22:36.167526 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:36.167469    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9008621-ca43-429a-9d79-7bcf46ebbd55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9305848-8fqmz\" (UID: \"b9008621-ca43-429a-9d79-7bcf46ebbd55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9305848-8fqmz"
Nov 01 03:22:36.178381 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:36.178357    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9305848-8fqmz"]
Nov 01 03:22:36.268455 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:36.268423    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvqk7\" (UniqueName: \"kubernetes.io/projected/b9008621-ca43-429a-9d79-7bcf46ebbd55-kube-api-access-dvqk7\") pod \"maas-api-revoke-sub-e9305848-8fqmz\" (UID: \"b9008621-ca43-429a-9d79-7bcf46ebbd55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9305848-8fqmz"
Nov 01 03:22:36.268663 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:36.268483    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9008621-ca43-429a-9d79-7bcf46ebbd55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9305848-8fqmz\" (UID: \"b9008621-ca43-429a-9d79-7bcf46ebbd55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9305848-8fqmz"
Nov 01 03:22:36.268663 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:36.268525    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9008621-ca43-429a-9d79-7bcf46ebbd55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9305848-8fqmz\" (UID: \"b9008621-ca43-429a-9d79-7bcf46ebbd55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9305848-8fqmz"
Nov 01 03:22:36.269144 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:36.269122    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9008621-ca43-429a-9d79-7bcf46ebbd55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9305848-8fqmz\" (UID: \"b9008621-ca43-429a-9d79-7bcf46ebbd55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9305848-8fqmz"
Nov 01 03:22:36.270836 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:36.270815    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9008621-ca43-429a-9d79-7bcf46ebbd55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9305848-8fqmz\" (UID: \"b9008621-ca43-429a-9d79-7bcf46ebbd55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9305848-8fqmz"
Nov 01 03:22:36.329127 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:36.329098    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvqk7\" (UniqueName: \"kubernetes.io/projected/b9008621-ca43-429a-9d79-7bcf46ebbd55-kube-api-access-dvqk7\") pod \"maas-api-revoke-sub-e9305848-8fqmz\" (UID: \"b9008621-ca43-429a-9d79-7bcf46ebbd55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9305848-8fqmz"
Nov 01 03:22:36.434751 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:36.434686    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9305848-8fqmz"
Nov 01 03:22:36.459179 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:36.459156    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c3bbbf0-flzbv"
Nov 01 03:22:36.459318 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:36.459175    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c3bbbf0-flzbv" event={"ID":"da0448da-122c-47c5-aa81-b3c5cc5193c3","Type":"ContainerDied","Data":"d0833feff68a227027448426dce3da794ac712a1cab3fcdbf90f21874544a56f"}
Nov 01 03:22:36.459318 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:36.459256    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0833feff68a227027448426dce3da794ac712a1cab3fcdbf90f21874544a56f"
Nov 01 03:22:36.808761 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:36.808731    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9305848-8fqmz"]
Nov 01 03:22:36.809592 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:22:36.809549    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9008621_ca43_429a_9d79_7bcf46ebbd55.slice/crio-a24074c83a2ddcb370e623ea00eb311cbd70ae8fc1a17f875e0fc1bd45c46d9a WatchSource:0}: Error finding container a24074c83a2ddcb370e623ea00eb311cbd70ae8fc1a17f875e0fc1bd45c46d9a: Status 404 returned error can't find the container with id a24074c83a2ddcb370e623ea00eb311cbd70ae8fc1a17f875e0fc1bd45c46d9a
Nov 01 03:22:37.467700 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:37.467664    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9305848-8fqmz" event={"ID":"b9008621-ca43-429a-9d79-7bcf46ebbd55","Type":"ContainerStarted","Data":"0f32a582e190b21a77a35a407d25bf0d86c221776ee182fc8808c3423a13ed9b"}
Nov 01 03:22:37.467879 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:37.467719    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9305848-8fqmz" event={"ID":"b9008621-ca43-429a-9d79-7bcf46ebbd55","Type":"ContainerStarted","Data":"a24074c83a2ddcb370e623ea00eb311cbd70ae8fc1a17f875e0fc1bd45c46d9a"}
Nov 01 03:22:37.585844 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:37.585792    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9305848-8fqmz" podStartSLOduration=1.585777411 podStartE2EDuration="1.585777411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:22:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:22:37.584354445 +0000 UTC m=+962.107361034" watchObservedRunningTime="2026-11-01 03:22:37.585777411 +0000 UTC m=+962.108784019"
Nov 01 03:22:38.426324 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:38.425607    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:22:38.484021 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:38.483990    4930 generic.go:356] "Generic (PLEG): container finished" podID="b9008621-ca43-429a-9d79-7bcf46ebbd55" containerID="0f32a582e190b21a77a35a407d25bf0d86c221776ee182fc8808c3423a13ed9b" exitCode=0
Nov 01 03:22:38.484181 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:38.484073    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9305848-8fqmz" event={"ID":"b9008621-ca43-429a-9d79-7bcf46ebbd55","Type":"ContainerDied","Data":"0f32a582e190b21a77a35a407d25bf0d86c221776ee182fc8808c3423a13ed9b"}
Nov 01 03:22:39.521014 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:39.520947    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq"]
Nov 01 03:22:39.524983 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:39.524963    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq"
Nov 01 03:22:39.577791 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:39.577761    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-d7f535-openshift-default-dockercfg-7hqkk\""
Nov 01 03:22:39.644128 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:39.644089    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq"]
Nov 01 03:22:39.649133 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:39.649096    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d42cb7e-b420-4bcb-a086-8ce40cc24849-istio-envoy\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq\" (UID: \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq"
Nov 01 03:22:39.649300 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:39.649240    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d42cb7e-b420-4bcb-a086-8ce40cc24849-istio-token\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq\" (UID: \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq"
Nov 01 03:22:39.649300 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:39.649278    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d42cb7e-b420-4bcb-a086-8ce40cc24849-istio-podinfo\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq\" (UID: \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq"
Nov 01 03:22:39.649418 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:39.649356    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d42cb7e-b420-4bcb-a086-8ce40cc24849-istiod-ca-cert\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq\" (UID: \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq"
Nov 01 03:22:39.649462 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:39.649439    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmgg6\" (UniqueName: \"kubernetes.io/projected/4d42cb7e-b420-4bcb-a086-8ce40cc24849-kube-api-access-hmgg6\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq\" (UID: \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq"
Nov 01 03:22:39.649505 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:39.649478    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d42cb7e-b420-4bcb-a086-8ce40cc24849-workload-certs\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq\" (UID: \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq"
Nov 01 03:22:39.649543 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:39.649504    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d42cb7e-b420-4bcb-a086-8ce40cc24849-workload-socket\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq\" (UID: \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq"
Nov 01 03:22:39.649543 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:39.649525    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d42cb7e-b420-4bcb-a086-8ce40cc24849-credential-socket\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq\" (UID: \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq"
Nov 01 03:22:39.649695 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:39.649674    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d42cb7e-b420-4bcb-a086-8ce40cc24849-istio-data\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq\" (UID: \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq"
Nov 01 03:22:39.653497 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:39.653475    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9305848-8fqmz"
Nov 01 03:22:39.750633 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:39.750510    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9008621-ca43-429a-9d79-7bcf46ebbd55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9008621-ca43-429a-9d79-7bcf46ebbd55-maas-api-openshift-ca\") pod \"b9008621-ca43-429a-9d79-7bcf46ebbd55\" (UID: \"b9008621-ca43-429a-9d79-7bcf46ebbd55\") "
Nov 01 03:22:39.750785 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:39.750669    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9008621-ca43-429a-9d79-7bcf46ebbd55-kube-api-access-dvqk7\" (UniqueName: \"kubernetes.io/projected/b9008621-ca43-429a-9d79-7bcf46ebbd55-kube-api-access-dvqk7\") pod \"b9008621-ca43-429a-9d79-7bcf46ebbd55\" (UID: \"b9008621-ca43-429a-9d79-7bcf46ebbd55\") "
Nov 01 03:22:39.750785 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:39.750745    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9008621-ca43-429a-9d79-7bcf46ebbd55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9008621-ca43-429a-9d79-7bcf46ebbd55-maas-api-xks-ca\") pod \"b9008621-ca43-429a-9d79-7bcf46ebbd55\" (UID: \"b9008621-ca43-429a-9d79-7bcf46ebbd55\") "
Nov 01 03:22:39.750904 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:39.750873    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9008621-ca43-429a-9d79-7bcf46ebbd55-maas-api-openshift-ca" pod "b9008621-ca43-429a-9d79-7bcf46ebbd55" (UID: "b9008621-ca43-429a-9d79-7bcf46ebbd55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:22:39.751010 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:39.750992    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d42cb7e-b420-4bcb-a086-8ce40cc24849-istio-token\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq\" (UID: \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq"
Nov 01 03:22:39.751083 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:39.751027    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d42cb7e-b420-4bcb-a086-8ce40cc24849-istio-podinfo\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq\" (UID: \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq"
Nov 01 03:22:39.751083 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:39.751067    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d42cb7e-b420-4bcb-a086-8ce40cc24849-istiod-ca-cert\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq\" (UID: \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq"
Nov 01 03:22:39.751164 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:39.751130    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmgg6\" (UniqueName: \"kubernetes.io/projected/4d42cb7e-b420-4bcb-a086-8ce40cc24849-kube-api-access-hmgg6\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq\" (UID: \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq"
Nov 01 03:22:39.751197 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:39.751181    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d42cb7e-b420-4bcb-a086-8ce40cc24849-workload-certs\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq\" (UID: \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq"
Nov 01 03:22:39.751252 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:39.751222    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d42cb7e-b420-4bcb-a086-8ce40cc24849-workload-socket\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq\" (UID: \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq"
Nov 01 03:22:39.751305 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:39.751247    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d42cb7e-b420-4bcb-a086-8ce40cc24849-credential-socket\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq\" (UID: \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq"
Nov 01 03:22:39.751425 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:39.751345    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d42cb7e-b420-4bcb-a086-8ce40cc24849-istio-data\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq\" (UID: \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq"
Nov 01 03:22:39.751492 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:39.751445    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d42cb7e-b420-4bcb-a086-8ce40cc24849-istio-envoy\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq\" (UID: \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq"
Nov 01 03:22:39.751587 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:39.751516    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9008621-ca43-429a-9d79-7bcf46ebbd55-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:39.751780 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:39.751712    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d42cb7e-b420-4bcb-a086-8ce40cc24849-workload-certs\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq\" (UID: \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq"
Nov 01 03:22:39.751976 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:39.751950    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d42cb7e-b420-4bcb-a086-8ce40cc24849-istiod-ca-cert\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq\" (UID: \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq"
Nov 01 03:22:39.752129 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:39.751968    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d42cb7e-b420-4bcb-a086-8ce40cc24849-credential-socket\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq\" (UID: \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq"
Nov 01 03:22:39.752277 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:39.752169    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d42cb7e-b420-4bcb-a086-8ce40cc24849-workload-socket\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq\" (UID: \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq"
Nov 01 03:22:39.752355 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:39.752278    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d42cb7e-b420-4bcb-a086-8ce40cc24849-istio-data\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq\" (UID: \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq"
Nov 01 03:22:39.753204 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:39.753178    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9008621-ca43-429a-9d79-7bcf46ebbd55-kube-api-access-dvqk7" pod "b9008621-ca43-429a-9d79-7bcf46ebbd55" (UID: "b9008621-ca43-429a-9d79-7bcf46ebbd55"). InnerVolumeSpecName "kube-api-access-dvqk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:22:39.753663 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:39.753644    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d42cb7e-b420-4bcb-a086-8ce40cc24849-istio-envoy\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq\" (UID: \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq"
Nov 01 03:22:39.753937 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:39.753916    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9008621-ca43-429a-9d79-7bcf46ebbd55-maas-api-xks-ca" pod "b9008621-ca43-429a-9d79-7bcf46ebbd55" (UID: "b9008621-ca43-429a-9d79-7bcf46ebbd55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:22:39.753982 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:39.753918    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d42cb7e-b420-4bcb-a086-8ce40cc24849-istio-podinfo\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq\" (UID: \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq"
Nov 01 03:22:39.815451 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:39.815421    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmgg6\" (UniqueName: \"kubernetes.io/projected/4d42cb7e-b420-4bcb-a086-8ce40cc24849-kube-api-access-hmgg6\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq\" (UID: \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq"
Nov 01 03:22:39.815818 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:39.815798    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d42cb7e-b420-4bcb-a086-8ce40cc24849-istio-token\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq\" (UID: \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq"
Nov 01 03:22:39.839979 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:39.839953    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq"
Nov 01 03:22:39.852663 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:39.852638    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvqk7\" (UniqueName: \"kubernetes.io/projected/b9008621-ca43-429a-9d79-7bcf46ebbd55-kube-api-access-dvqk7\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:39.852737 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:39.852670    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9008621-ca43-429a-9d79-7bcf46ebbd55-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:40.271213 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:22:40.271181    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d42cb7e_b420_4bcb_a086_8ce40cc24849.slice/crio-06c7cacbe156e70b8eb172ec4981346e65a5464a9592d45547cde933fe6cc897 WatchSource:0}: Error finding container 06c7cacbe156e70b8eb172ec4981346e65a5464a9592d45547cde933fe6cc897: Status 404 returned error can't find the container with id 06c7cacbe156e70b8eb172ec4981346e65a5464a9592d45547cde933fe6cc897
Nov 01 03:22:40.272040 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:40.272009    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq"]
Nov 01 03:22:40.273853 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:40.273818    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:22:40.273978 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:40.273883    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:22:40.273978 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:40.273913    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:22:40.499397 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:40.499371    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9305848-8fqmz"
Nov 01 03:22:40.499588 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:40.499408    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9305848-8fqmz" event={"ID":"b9008621-ca43-429a-9d79-7bcf46ebbd55","Type":"ContainerDied","Data":"a24074c83a2ddcb370e623ea00eb311cbd70ae8fc1a17f875e0fc1bd45c46d9a"}
Nov 01 03:22:40.499588 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:40.499464    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a24074c83a2ddcb370e623ea00eb311cbd70ae8fc1a17f875e0fc1bd45c46d9a"
Nov 01 03:22:40.501574 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:40.501519    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq" event={"ID":"4d42cb7e-b420-4bcb-a086-8ce40cc24849","Type":"ContainerStarted","Data":"f0008cb73ee130a8a17f6189321b0f5da7589a3cd2bebb0571ca6785b0f754a3"}
Nov 01 03:22:40.501717 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:40.501597    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq" event={"ID":"4d42cb7e-b420-4bcb-a086-8ce40cc24849","Type":"ContainerStarted","Data":"06c7cacbe156e70b8eb172ec4981346e65a5464a9592d45547cde933fe6cc897"}
Nov 01 03:22:40.625546 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:40.625487    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq" podStartSLOduration=1.625468497 podStartE2EDuration="1.625468497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:22:40.621125817 +0000 UTC m=+965.144132409" watchObservedRunningTime="2026-11-01 03:22:40.625468497 +0000 UTC m=+965.148475086"
Nov 01 03:22:40.840735 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:40.840681    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq"
Nov 01 03:22:41.041589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:41.041476    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:22:41.841941 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:41.841904    4930 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 01 03:22:41.842431 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:41.841956    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq" podUID="4d42cb7e-b420-4bcb-a086-8ce40cc24849" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 01 03:22:41.893646 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:41.893614    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02454908-l2l78"]
Nov 01 03:22:41.894097 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:41.894083    4930 cpu_manager.go:414] "containerMap: removing container" podUID="b9008621-ca43-429a-9d79-7bcf46ebbd55" containerName="revoke-keys"
Nov 01 03:22:41.894097 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:41.894098    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9008621-ca43-429a-9d79-7bcf46ebbd55" containerName="revoke-keys"
Nov 01 03:22:41.894208 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:41.894202    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9008621-ca43-429a-9d79-7bcf46ebbd55" containerName="revoke-keys"
Nov 01 03:22:41.897602 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:41.897579    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02454908-l2l78"
Nov 01 03:22:41.952528 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:41.952487    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:22:41.952943 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:41.952920    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:22:41.956943 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:41.956919    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02454908-l2l78"]
Nov 01 03:22:42.002342 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:42.002310    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:22:42.002342 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:42.002329    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:22:42.094302 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:42.094205    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa58e66b-8dbc-430c-a8c0-479fbd7999c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02454908-l2l78\" (UID: \"fa58e66b-8dbc-430c-a8c0-479fbd7999c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02454908-l2l78"
Nov 01 03:22:42.094302 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:42.094277    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nww7s\" (UniqueName: \"kubernetes.io/projected/fa58e66b-8dbc-430c-a8c0-479fbd7999c1-kube-api-access-nww7s\") pod \"maas-api-revoke-sub-02454908-l2l78\" (UID: \"fa58e66b-8dbc-430c-a8c0-479fbd7999c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02454908-l2l78"
Nov 01 03:22:42.094530 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:42.094429    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa58e66b-8dbc-430c-a8c0-479fbd7999c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02454908-l2l78\" (UID: \"fa58e66b-8dbc-430c-a8c0-479fbd7999c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02454908-l2l78"
Nov 01 03:22:42.196084 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:42.196051    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa58e66b-8dbc-430c-a8c0-479fbd7999c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02454908-l2l78\" (UID: \"fa58e66b-8dbc-430c-a8c0-479fbd7999c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02454908-l2l78"
Nov 01 03:22:42.196228 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:42.196115    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nww7s\" (UniqueName: \"kubernetes.io/projected/fa58e66b-8dbc-430c-a8c0-479fbd7999c1-kube-api-access-nww7s\") pod \"maas-api-revoke-sub-02454908-l2l78\" (UID: \"fa58e66b-8dbc-430c-a8c0-479fbd7999c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02454908-l2l78"
Nov 01 03:22:42.196228 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:42.196204    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa58e66b-8dbc-430c-a8c0-479fbd7999c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02454908-l2l78\" (UID: \"fa58e66b-8dbc-430c-a8c0-479fbd7999c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02454908-l2l78"
Nov 01 03:22:42.196881 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:42.196861    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa58e66b-8dbc-430c-a8c0-479fbd7999c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02454908-l2l78\" (UID: \"fa58e66b-8dbc-430c-a8c0-479fbd7999c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02454908-l2l78"
Nov 01 03:22:42.198309 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:42.198288    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa58e66b-8dbc-430c-a8c0-479fbd7999c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02454908-l2l78\" (UID: \"fa58e66b-8dbc-430c-a8c0-479fbd7999c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02454908-l2l78"
Nov 01 03:22:42.257481 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:42.257452    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nww7s\" (UniqueName: \"kubernetes.io/projected/fa58e66b-8dbc-430c-a8c0-479fbd7999c1-kube-api-access-nww7s\") pod \"maas-api-revoke-sub-02454908-l2l78\" (UID: \"fa58e66b-8dbc-430c-a8c0-479fbd7999c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02454908-l2l78"
Nov 01 03:22:42.513154 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:42.513064    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02454908-l2l78"
Nov 01 03:22:42.691697 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:42.691661    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46ed40ca-qhc7q"]
Nov 01 03:22:42.696702 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:42.696678    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46ed40ca-qhc7q"
Nov 01 03:22:42.701662 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:42.701509    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cbb22dc-a592-4452-b87f-fb1eaba7fd47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46ed40ca-qhc7q\" (UID: \"6cbb22dc-a592-4452-b87f-fb1eaba7fd47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46ed40ca-qhc7q"
Nov 01 03:22:42.701662 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:42.701603    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt44m\" (UniqueName: \"kubernetes.io/projected/6cbb22dc-a592-4452-b87f-fb1eaba7fd47-kube-api-access-rt44m\") pod \"maas-api-revoke-sub-46ed40ca-qhc7q\" (UID: \"6cbb22dc-a592-4452-b87f-fb1eaba7fd47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46ed40ca-qhc7q"
Nov 01 03:22:42.701802 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:42.701671    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cbb22dc-a592-4452-b87f-fb1eaba7fd47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46ed40ca-qhc7q\" (UID: \"6cbb22dc-a592-4452-b87f-fb1eaba7fd47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46ed40ca-qhc7q"
Nov 01 03:22:42.741888 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:42.741859    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46ed40ca-qhc7q"]
Nov 01 03:22:42.803964 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:42.803061    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cbb22dc-a592-4452-b87f-fb1eaba7fd47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46ed40ca-qhc7q\" (UID: \"6cbb22dc-a592-4452-b87f-fb1eaba7fd47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46ed40ca-qhc7q"
Nov 01 03:22:42.803964 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:42.803113    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt44m\" (UniqueName: \"kubernetes.io/projected/6cbb22dc-a592-4452-b87f-fb1eaba7fd47-kube-api-access-rt44m\") pod \"maas-api-revoke-sub-46ed40ca-qhc7q\" (UID: \"6cbb22dc-a592-4452-b87f-fb1eaba7fd47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46ed40ca-qhc7q"
Nov 01 03:22:42.803964 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:42.803150    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cbb22dc-a592-4452-b87f-fb1eaba7fd47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46ed40ca-qhc7q\" (UID: \"6cbb22dc-a592-4452-b87f-fb1eaba7fd47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46ed40ca-qhc7q"
Nov 01 03:22:42.803964 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:42.803869    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cbb22dc-a592-4452-b87f-fb1eaba7fd47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46ed40ca-qhc7q\" (UID: \"6cbb22dc-a592-4452-b87f-fb1eaba7fd47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46ed40ca-qhc7q"
Nov 01 03:22:42.806612 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:42.806544    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cbb22dc-a592-4452-b87f-fb1eaba7fd47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46ed40ca-qhc7q\" (UID: \"6cbb22dc-a592-4452-b87f-fb1eaba7fd47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46ed40ca-qhc7q"
Nov 01 03:22:42.841369 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:42.841340    4930 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 01 03:22:42.841461 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:42.841394    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq" podUID="4d42cb7e-b420-4bcb-a086-8ce40cc24849" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 01 03:22:42.887992 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:42.887952    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt44m\" (UniqueName: \"kubernetes.io/projected/6cbb22dc-a592-4452-b87f-fb1eaba7fd47-kube-api-access-rt44m\") pod \"maas-api-revoke-sub-46ed40ca-qhc7q\" (UID: \"6cbb22dc-a592-4452-b87f-fb1eaba7fd47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46ed40ca-qhc7q"
Nov 01 03:22:42.928628 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:42.928603    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02454908-l2l78"]
Nov 01 03:22:42.929061 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:22:42.929036    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa58e66b_8dbc_430c_a8c0_479fbd7999c1.slice/crio-cfb02cdedffc2d6449b39e68087d2e89844555a434b014ea6d0f46b49c7adc91 WatchSource:0}: Error finding container cfb02cdedffc2d6449b39e68087d2e89844555a434b014ea6d0f46b49c7adc91: Status 404 returned error can't find the container with id cfb02cdedffc2d6449b39e68087d2e89844555a434b014ea6d0f46b49c7adc91
Nov 01 03:22:43.016222 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:43.016184    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46ed40ca-qhc7q"
Nov 01 03:22:43.214664 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:43.214626    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:22:43.429105 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:43.429077    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46ed40ca-qhc7q"]
Nov 01 03:22:43.430093 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:22:43.430062    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cbb22dc_a592_4452_b87f_fb1eaba7fd47.slice/crio-0388f4f83fb10ec08a3359d104ab86d9f9db6808e60c10f2723b7e54aa70b3ab WatchSource:0}: Error finding container 0388f4f83fb10ec08a3359d104ab86d9f9db6808e60c10f2723b7e54aa70b3ab: Status 404 returned error can't find the container with id 0388f4f83fb10ec08a3359d104ab86d9f9db6808e60c10f2723b7e54aa70b3ab
Nov 01 03:22:43.524651 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:43.524595    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46ed40ca-qhc7q" event={"ID":"6cbb22dc-a592-4452-b87f-fb1eaba7fd47","Type":"ContainerStarted","Data":"0388f4f83fb10ec08a3359d104ab86d9f9db6808e60c10f2723b7e54aa70b3ab"}
Nov 01 03:22:43.526178 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:43.526151    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02454908-l2l78" event={"ID":"fa58e66b-8dbc-430c-a8c0-479fbd7999c1","Type":"ContainerStarted","Data":"b0d1dae7166b333d8de5b9b2f1deebcb0e0daa167476945bb5bab796297ffe17"}
Nov 01 03:22:43.526289 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:43.526196    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02454908-l2l78" event={"ID":"fa58e66b-8dbc-430c-a8c0-479fbd7999c1","Type":"ContainerStarted","Data":"cfb02cdedffc2d6449b39e68087d2e89844555a434b014ea6d0f46b49c7adc91"}
Nov 01 03:22:43.647100 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:43.647005    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02454908-l2l78" podStartSLOduration=2.646988629 podStartE2EDuration="2.646988629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:22:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:22:43.64661132 +0000 UTC m=+968.169617908" watchObservedRunningTime="2026-11-01 03:22:43.646988629 +0000 UTC m=+968.169995271"
Nov 01 03:22:43.841364 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:43.841336    4930 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 01 03:22:43.841481 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:43.841379    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq" podUID="4d42cb7e-b420-4bcb-a086-8ce40cc24849" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 01 03:22:44.535538 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:44.535507    4930 generic.go:356] "Generic (PLEG): container finished" podID="6cbb22dc-a592-4452-b87f-fb1eaba7fd47" containerID="15cbfc6500a39e08475ae301f35233e47d7be9739a633417bfc9940e32be6d1a" exitCode=0
Nov 01 03:22:44.535950 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:44.535587    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46ed40ca-qhc7q" event={"ID":"6cbb22dc-a592-4452-b87f-fb1eaba7fd47","Type":"ContainerDied","Data":"15cbfc6500a39e08475ae301f35233e47d7be9739a633417bfc9940e32be6d1a"}
Nov 01 03:22:44.537332 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:44.537303    4930 generic.go:356] "Generic (PLEG): container finished" podID="fa58e66b-8dbc-430c-a8c0-479fbd7999c1" containerID="b0d1dae7166b333d8de5b9b2f1deebcb0e0daa167476945bb5bab796297ffe17" exitCode=0
Nov 01 03:22:44.537449 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:44.537365    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02454908-l2l78" event={"ID":"fa58e66b-8dbc-430c-a8c0-479fbd7999c1","Type":"ContainerDied","Data":"b0d1dae7166b333d8de5b9b2f1deebcb0e0daa167476945bb5bab796297ffe17"}
Nov 01 03:22:44.840521 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:44.840491    4930 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 01 03:22:44.840700 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:44.840546    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq" podUID="4d42cb7e-b420-4bcb-a086-8ce40cc24849" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 01 03:22:45.800498 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:45.800312    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02454908-l2l78"
Nov 01 03:22:45.803739 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:45.803720    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46ed40ca-qhc7q"
Nov 01 03:22:45.838980 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:45.838952    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa58e66b-8dbc-430c-a8c0-479fbd7999c1-kube-api-access-nww7s\" (UniqueName: \"kubernetes.io/projected/fa58e66b-8dbc-430c-a8c0-479fbd7999c1-kube-api-access-nww7s\") pod \"fa58e66b-8dbc-430c-a8c0-479fbd7999c1\" (UID: \"fa58e66b-8dbc-430c-a8c0-479fbd7999c1\") "
Nov 01 03:22:45.839078 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:45.839039    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa58e66b-8dbc-430c-a8c0-479fbd7999c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa58e66b-8dbc-430c-a8c0-479fbd7999c1-maas-api-xks-ca\") pod \"fa58e66b-8dbc-430c-a8c0-479fbd7999c1\" (UID: \"fa58e66b-8dbc-430c-a8c0-479fbd7999c1\") "
Nov 01 03:22:45.839133 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:45.839088    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cbb22dc-a592-4452-b87f-fb1eaba7fd47-kube-api-access-rt44m\" (UniqueName: \"kubernetes.io/projected/6cbb22dc-a592-4452-b87f-fb1eaba7fd47-kube-api-access-rt44m\") pod \"6cbb22dc-a592-4452-b87f-fb1eaba7fd47\" (UID: \"6cbb22dc-a592-4452-b87f-fb1eaba7fd47\") "
Nov 01 03:22:45.839198 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:45.839153    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cbb22dc-a592-4452-b87f-fb1eaba7fd47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cbb22dc-a592-4452-b87f-fb1eaba7fd47-maas-api-openshift-ca\") pod \"6cbb22dc-a592-4452-b87f-fb1eaba7fd47\" (UID: \"6cbb22dc-a592-4452-b87f-fb1eaba7fd47\") "
Nov 01 03:22:45.839254 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:45.839233    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cbb22dc-a592-4452-b87f-fb1eaba7fd47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cbb22dc-a592-4452-b87f-fb1eaba7fd47-maas-api-xks-ca\") pod \"6cbb22dc-a592-4452-b87f-fb1eaba7fd47\" (UID: \"6cbb22dc-a592-4452-b87f-fb1eaba7fd47\") "
Nov 01 03:22:45.839308 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:45.839285    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa58e66b-8dbc-430c-a8c0-479fbd7999c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa58e66b-8dbc-430c-a8c0-479fbd7999c1-maas-api-openshift-ca\") pod \"fa58e66b-8dbc-430c-a8c0-479fbd7999c1\" (UID: \"fa58e66b-8dbc-430c-a8c0-479fbd7999c1\") "
Nov 01 03:22:45.839760 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:45.839527    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cbb22dc-a592-4452-b87f-fb1eaba7fd47-maas-api-openshift-ca" pod "6cbb22dc-a592-4452-b87f-fb1eaba7fd47" (UID: "6cbb22dc-a592-4452-b87f-fb1eaba7fd47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:22:45.839760 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:45.839722    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cbb22dc-a592-4452-b87f-fb1eaba7fd47-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:45.839932 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:45.839906    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa58e66b-8dbc-430c-a8c0-479fbd7999c1-maas-api-openshift-ca" pod "fa58e66b-8dbc-430c-a8c0-479fbd7999c1" (UID: "fa58e66b-8dbc-430c-a8c0-479fbd7999c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:22:45.840596 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:45.840514    4930 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 01 03:22:45.840703 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:45.840609    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq" podUID="4d42cb7e-b420-4bcb-a086-8ce40cc24849" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 01 03:22:45.841487 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:45.841460    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cbb22dc-a592-4452-b87f-fb1eaba7fd47-maas-api-xks-ca" pod "6cbb22dc-a592-4452-b87f-fb1eaba7fd47" (UID: "6cbb22dc-a592-4452-b87f-fb1eaba7fd47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:22:45.841678 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:45.841651    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa58e66b-8dbc-430c-a8c0-479fbd7999c1-maas-api-xks-ca" pod "fa58e66b-8dbc-430c-a8c0-479fbd7999c1" (UID: "fa58e66b-8dbc-430c-a8c0-479fbd7999c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:22:45.841828 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:45.841799    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cbb22dc-a592-4452-b87f-fb1eaba7fd47-kube-api-access-rt44m" pod "6cbb22dc-a592-4452-b87f-fb1eaba7fd47" (UID: "6cbb22dc-a592-4452-b87f-fb1eaba7fd47"). InnerVolumeSpecName "kube-api-access-rt44m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:22:45.841898 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:45.841829    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa58e66b-8dbc-430c-a8c0-479fbd7999c1-kube-api-access-nww7s" pod "fa58e66b-8dbc-430c-a8c0-479fbd7999c1" (UID: "fa58e66b-8dbc-430c-a8c0-479fbd7999c1"). InnerVolumeSpecName "kube-api-access-nww7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:22:45.940797 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:45.940761    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cbb22dc-a592-4452-b87f-fb1eaba7fd47-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:45.940797 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:45.940788    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa58e66b-8dbc-430c-a8c0-479fbd7999c1-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:45.940797 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:45.940798    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nww7s\" (UniqueName: \"kubernetes.io/projected/fa58e66b-8dbc-430c-a8c0-479fbd7999c1-kube-api-access-nww7s\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:45.940797 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:45.940807    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa58e66b-8dbc-430c-a8c0-479fbd7999c1-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:45.941153 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:45.940817    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt44m\" (UniqueName: \"kubernetes.io/projected/6cbb22dc-a592-4452-b87f-fb1eaba7fd47-kube-api-access-rt44m\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:22:46.552212 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:46.552181    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46ed40ca-qhc7q"
Nov 01 03:22:46.552212 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:46.552199    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46ed40ca-qhc7q" event={"ID":"6cbb22dc-a592-4452-b87f-fb1eaba7fd47","Type":"ContainerDied","Data":"0388f4f83fb10ec08a3359d104ab86d9f9db6808e60c10f2723b7e54aa70b3ab"}
Nov 01 03:22:46.552458 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:46.552251    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0388f4f83fb10ec08a3359d104ab86d9f9db6808e60c10f2723b7e54aa70b3ab"
Nov 01 03:22:46.553821 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:46.553790    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02454908-l2l78" event={"ID":"fa58e66b-8dbc-430c-a8c0-479fbd7999c1","Type":"ContainerDied","Data":"cfb02cdedffc2d6449b39e68087d2e89844555a434b014ea6d0f46b49c7adc91"}
Nov 01 03:22:46.553941 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:46.553830    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02454908-l2l78"
Nov 01 03:22:46.553941 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:46.553837    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfb02cdedffc2d6449b39e68087d2e89844555a434b014ea6d0f46b49c7adc91"
Nov 01 03:22:46.704540 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:46.704500    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"]
Nov 01 03:22:46.705194 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:46.705169    4930 cpu_manager.go:414] "containerMap: removing container" podUID="fa58e66b-8dbc-430c-a8c0-479fbd7999c1" containerName="revoke-keys"
Nov 01 03:22:46.705301 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:46.705195    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa58e66b-8dbc-430c-a8c0-479fbd7999c1" containerName="revoke-keys"
Nov 01 03:22:46.705301 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:46.705210    4930 cpu_manager.go:414] "containerMap: removing container" podUID="6cbb22dc-a592-4452-b87f-fb1eaba7fd47" containerName="revoke-keys"
Nov 01 03:22:46.705301 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:46.705217    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cbb22dc-a592-4452-b87f-fb1eaba7fd47" containerName="revoke-keys"
Nov 01 03:22:46.705468 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:46.705373    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa58e66b-8dbc-430c-a8c0-479fbd7999c1" containerName="revoke-keys"
Nov 01 03:22:46.705468 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:46.705384    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cbb22dc-a592-4452-b87f-fb1eaba7fd47" containerName="revoke-keys"
Nov 01 03:22:46.709478 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:46.709454    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:22:46.749168 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:46.749134    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prt4c\" (UniqueName: \"kubernetes.io/projected/88a5c7e9-2bd5-4a18-87ad-44feac2eb94e-kube-api-access-prt4c\") pod \"distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8\" (UID: \"88a5c7e9-2bd5-4a18-87ad-44feac2eb94e\") " pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:22:46.749327 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:46.749252    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88a5c7e9-2bd5-4a18-87ad-44feac2eb94e-home\") pod \"distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8\" (UID: \"88a5c7e9-2bd5-4a18-87ad-44feac2eb94e\") " pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:22:46.749327 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:46.749274    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88a5c7e9-2bd5-4a18-87ad-44feac2eb94e-tls-certs\") pod \"distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8\" (UID: \"88a5c7e9-2bd5-4a18-87ad-44feac2eb94e\") " pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:22:46.749444 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:46.749403    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88a5c7e9-2bd5-4a18-87ad-44feac2eb94e-dshm\") pod \"distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8\" (UID: \"88a5c7e9-2bd5-4a18-87ad-44feac2eb94e\") " pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:22:46.749506 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:46.749452    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88a5c7e9-2bd5-4a18-87ad-44feac2eb94e-model-cache\") pod \"distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8\" (UID: \"88a5c7e9-2bd5-4a18-87ad-44feac2eb94e\") " pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:22:46.749595 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:46.749570    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88a5c7e9-2bd5-4a18-87ad-44feac2eb94e-tmp-dir\") pod \"distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8\" (UID: \"88a5c7e9-2bd5-4a18-87ad-44feac2eb94e\") " pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:22:46.762018 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:46.761994    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ded2d7\"/\"distinct-w0-ded2d7-kserve-self-signed-certs\""
Nov 01 03:22:46.767364 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:46.767335    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"]
Nov 01 03:22:46.785710 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:46.785679    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:22:46.841082 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:46.841056    4930 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 01 03:22:46.841458 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:46.841114    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq" podUID="4d42cb7e-b420-4bcb-a086-8ce40cc24849" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 01 03:22:46.850861 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:46.850825    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88a5c7e9-2bd5-4a18-87ad-44feac2eb94e-dshm\") pod \"distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8\" (UID: \"88a5c7e9-2bd5-4a18-87ad-44feac2eb94e\") " pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:22:46.851028 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:46.850989    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88a5c7e9-2bd5-4a18-87ad-44feac2eb94e-model-cache\") pod \"distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8\" (UID: \"88a5c7e9-2bd5-4a18-87ad-44feac2eb94e\") " pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:22:46.851095 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:46.851038    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88a5c7e9-2bd5-4a18-87ad-44feac2eb94e-tmp-dir\") pod \"distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8\" (UID: \"88a5c7e9-2bd5-4a18-87ad-44feac2eb94e\") " pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:22:46.851095 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:46.851065    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prt4c\" (UniqueName: \"kubernetes.io/projected/88a5c7e9-2bd5-4a18-87ad-44feac2eb94e-kube-api-access-prt4c\") pod \"distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8\" (UID: \"88a5c7e9-2bd5-4a18-87ad-44feac2eb94e\") " pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:22:46.851214 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:46.851149    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88a5c7e9-2bd5-4a18-87ad-44feac2eb94e-home\") pod \"distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8\" (UID: \"88a5c7e9-2bd5-4a18-87ad-44feac2eb94e\") " pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:22:46.851214 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:46.851174    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88a5c7e9-2bd5-4a18-87ad-44feac2eb94e-tls-certs\") pod \"distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8\" (UID: \"88a5c7e9-2bd5-4a18-87ad-44feac2eb94e\") " pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:22:46.852090 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:46.852038    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88a5c7e9-2bd5-4a18-87ad-44feac2eb94e-home\") pod \"distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8\" (UID: \"88a5c7e9-2bd5-4a18-87ad-44feac2eb94e\") " pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:22:46.852231 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:46.852207    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88a5c7e9-2bd5-4a18-87ad-44feac2eb94e-tmp-dir\") pod \"distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8\" (UID: \"88a5c7e9-2bd5-4a18-87ad-44feac2eb94e\") " pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:22:46.852380 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:46.852358    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88a5c7e9-2bd5-4a18-87ad-44feac2eb94e-model-cache\") pod \"distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8\" (UID: \"88a5c7e9-2bd5-4a18-87ad-44feac2eb94e\") " pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:22:46.854198 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:46.854178    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88a5c7e9-2bd5-4a18-87ad-44feac2eb94e-dshm\") pod \"distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8\" (UID: \"88a5c7e9-2bd5-4a18-87ad-44feac2eb94e\") " pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:22:46.856065 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:46.856016    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88a5c7e9-2bd5-4a18-87ad-44feac2eb94e-tls-certs\") pod \"distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8\" (UID: \"88a5c7e9-2bd5-4a18-87ad-44feac2eb94e\") " pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:22:46.912981 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:46.912944    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prt4c\" (UniqueName: \"kubernetes.io/projected/88a5c7e9-2bd5-4a18-87ad-44feac2eb94e-kube-api-access-prt4c\") pod \"distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8\" (UID: \"88a5c7e9-2bd5-4a18-87ad-44feac2eb94e\") " pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:22:47.023963 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:47.023930    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:22:47.434644 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:47.433675    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"]
Nov 01 03:22:47.564188 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:47.564147    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8" event={"ID":"88a5c7e9-2bd5-4a18-87ad-44feac2eb94e","Type":"ContainerStarted","Data":"a07531139b08b831be771d6d57d9790db21302d081233cf78f49e3d56c68a0b7"}
Nov 01 03:22:47.564334 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:47.564201    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8" event={"ID":"88a5c7e9-2bd5-4a18-87ad-44feac2eb94e","Type":"ContainerStarted","Data":"08e1c3e654e3a874326829863a525666d1f95d01b88f6c785f74cb282112f63f"}
Nov 01 03:22:47.683944 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:47.683849    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8" podStartSLOduration=1.683833846 podStartE2EDuration="1.683833846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:22:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:22:47.679979821 +0000 UTC m=+972.202986408" watchObservedRunningTime="2026-11-01 03:22:47.683833846 +0000 UTC m=+972.206840435"
Nov 01 03:22:47.845192 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:47.845161    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq"
Nov 01 03:22:47.845667 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:47.845585    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq"
Nov 01 03:22:47.846208 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:47.846192    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq"
Nov 01 03:22:48.024082 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.023992    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:22:48.060651 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.060607    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:22:48.756181 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.756151    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx"]
Nov 01 03:22:48.760115 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.760090    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx"
Nov 01 03:22:48.807627 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.807599    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k"]
Nov 01 03:22:48.811341 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.811325    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k"
Nov 01 03:22:48.812019 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.812001    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts"]
Nov 01 03:22:48.815500 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.815484    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts"
Nov 01 03:22:48.865515 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.865492    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx"]
Nov 01 03:22:48.874900 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.874870    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltsb7\" (UniqueName: \"kubernetes.io/projected/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-kube-api-access-ltsb7\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-c29ts\" (UID: \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts"
Nov 01 03:22:48.875009 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.874918    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc508d65-2f4b-4cf1-aec1-8d229baef624-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-f4djx\" (UID: \"fc508d65-2f4b-4cf1-aec1-8d229baef624\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx"
Nov 01 03:22:48.875009 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.874950    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-c29ts\" (UID: \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts"
Nov 01 03:22:48.875009 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.874988    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc508d65-2f4b-4cf1-aec1-8d229baef624-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-f4djx\" (UID: \"fc508d65-2f4b-4cf1-aec1-8d229baef624\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx"
Nov 01 03:22:48.875389 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.875032    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-c29ts\" (UID: \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts"
Nov 01 03:22:48.875389 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.875077    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc508d65-2f4b-4cf1-aec1-8d229baef624-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-f4djx\" (UID: \"fc508d65-2f4b-4cf1-aec1-8d229baef624\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx"
Nov 01 03:22:48.875389 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.875105    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-c29ts\" (UID: \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts"
Nov 01 03:22:48.875389 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.875134    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85529ed7-e2d9-470f-a897-1711fd4abfa0-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xpl7k\" (UID: \"85529ed7-e2d9-470f-a897-1711fd4abfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k"
Nov 01 03:22:48.875389 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.875174    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-c29ts\" (UID: \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts"
Nov 01 03:22:48.875389 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.875203    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzkdx\" (UniqueName: \"kubernetes.io/projected/fc508d65-2f4b-4cf1-aec1-8d229baef624-kube-api-access-nzkdx\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-f4djx\" (UID: \"fc508d65-2f4b-4cf1-aec1-8d229baef624\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx"
Nov 01 03:22:48.875389 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.875269    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-c29ts\" (UID: \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts"
Nov 01 03:22:48.875389 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.875297    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-c29ts\" (UID: \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts"
Nov 01 03:22:48.875389 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.875323    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-c29ts\" (UID: \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts"
Nov 01 03:22:48.875389 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.875362    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc508d65-2f4b-4cf1-aec1-8d229baef624-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-f4djx\" (UID: \"fc508d65-2f4b-4cf1-aec1-8d229baef624\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx"
Nov 01 03:22:48.875389 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.875388    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhnzk\" (UniqueName: \"kubernetes.io/projected/85529ed7-e2d9-470f-a897-1711fd4abfa0-kube-api-access-zhnzk\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xpl7k\" (UID: \"85529ed7-e2d9-470f-a897-1711fd4abfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k"
Nov 01 03:22:48.875941 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.875420    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85529ed7-e2d9-470f-a897-1711fd4abfa0-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xpl7k\" (UID: \"85529ed7-e2d9-470f-a897-1711fd4abfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k"
Nov 01 03:22:48.875941 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.875457    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc508d65-2f4b-4cf1-aec1-8d229baef624-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-f4djx\" (UID: \"fc508d65-2f4b-4cf1-aec1-8d229baef624\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx"
Nov 01 03:22:48.875941 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.875491    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc508d65-2f4b-4cf1-aec1-8d229baef624-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-f4djx\" (UID: \"fc508d65-2f4b-4cf1-aec1-8d229baef624\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx"
Nov 01 03:22:48.875941 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.875523    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc508d65-2f4b-4cf1-aec1-8d229baef624-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-f4djx\" (UID: \"fc508d65-2f4b-4cf1-aec1-8d229baef624\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx"
Nov 01 03:22:48.875941 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.875584    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85529ed7-e2d9-470f-a897-1711fd4abfa0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xpl7k\" (UID: \"85529ed7-e2d9-470f-a897-1711fd4abfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k"
Nov 01 03:22:48.875941 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.875630    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc508d65-2f4b-4cf1-aec1-8d229baef624-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-f4djx\" (UID: \"fc508d65-2f4b-4cf1-aec1-8d229baef624\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx"
Nov 01 03:22:48.875941 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.875655    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-c29ts\" (UID: \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts"
Nov 01 03:22:48.875941 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.875682    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85529ed7-e2d9-470f-a897-1711fd4abfa0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xpl7k\" (UID: \"85529ed7-e2d9-470f-a897-1711fd4abfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k"
Nov 01 03:22:48.875941 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.875716    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85529ed7-e2d9-470f-a897-1711fd4abfa0-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xpl7k\" (UID: \"85529ed7-e2d9-470f-a897-1711fd4abfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k"
Nov 01 03:22:48.875941 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.875741    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85529ed7-e2d9-470f-a897-1711fd4abfa0-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xpl7k\" (UID: \"85529ed7-e2d9-470f-a897-1711fd4abfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k"
Nov 01 03:22:48.875941 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.875772    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85529ed7-e2d9-470f-a897-1711fd4abfa0-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xpl7k\" (UID: \"85529ed7-e2d9-470f-a897-1711fd4abfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k"
Nov 01 03:22:48.875941 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.875810    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85529ed7-e2d9-470f-a897-1711fd4abfa0-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xpl7k\" (UID: \"85529ed7-e2d9-470f-a897-1711fd4abfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k"
Nov 01 03:22:48.936097 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.936069    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts"]
Nov 01 03:22:48.941244 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.941215    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k"]
Nov 01 03:22:48.976785 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.976756    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc508d65-2f4b-4cf1-aec1-8d229baef624-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-f4djx\" (UID: \"fc508d65-2f4b-4cf1-aec1-8d229baef624\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx"
Nov 01 03:22:48.976897 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.976792    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc508d65-2f4b-4cf1-aec1-8d229baef624-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-f4djx\" (UID: \"fc508d65-2f4b-4cf1-aec1-8d229baef624\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx"
Nov 01 03:22:48.976897 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.976810    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85529ed7-e2d9-470f-a897-1711fd4abfa0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xpl7k\" (UID: \"85529ed7-e2d9-470f-a897-1711fd4abfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k"
Nov 01 03:22:48.976897 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.976864    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc508d65-2f4b-4cf1-aec1-8d229baef624-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-f4djx\" (UID: \"fc508d65-2f4b-4cf1-aec1-8d229baef624\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx"
Nov 01 03:22:48.977017 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.976898    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-c29ts\" (UID: \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts"
Nov 01 03:22:48.977017 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.976928    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85529ed7-e2d9-470f-a897-1711fd4abfa0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xpl7k\" (UID: \"85529ed7-e2d9-470f-a897-1711fd4abfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k"
Nov 01 03:22:48.977017 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.976958    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85529ed7-e2d9-470f-a897-1711fd4abfa0-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xpl7k\" (UID: \"85529ed7-e2d9-470f-a897-1711fd4abfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k"
Nov 01 03:22:48.977017 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.976984    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85529ed7-e2d9-470f-a897-1711fd4abfa0-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xpl7k\" (UID: \"85529ed7-e2d9-470f-a897-1711fd4abfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k"
Nov 01 03:22:48.977647 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.977312    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc508d65-2f4b-4cf1-aec1-8d229baef624-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-f4djx\" (UID: \"fc508d65-2f4b-4cf1-aec1-8d229baef624\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx"
Nov 01 03:22:48.977647 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.977404    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85529ed7-e2d9-470f-a897-1711fd4abfa0-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xpl7k\" (UID: \"85529ed7-e2d9-470f-a897-1711fd4abfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k"
Nov 01 03:22:48.977647 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.977055    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85529ed7-e2d9-470f-a897-1711fd4abfa0-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xpl7k\" (UID: \"85529ed7-e2d9-470f-a897-1711fd4abfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k"
Nov 01 03:22:48.978997 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.977736    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85529ed7-e2d9-470f-a897-1711fd4abfa0-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xpl7k\" (UID: \"85529ed7-e2d9-470f-a897-1711fd4abfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k"
Nov 01 03:22:48.978997 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.977741    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85529ed7-e2d9-470f-a897-1711fd4abfa0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xpl7k\" (UID: \"85529ed7-e2d9-470f-a897-1711fd4abfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k"
Nov 01 03:22:48.978997 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.977794    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85529ed7-e2d9-470f-a897-1711fd4abfa0-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xpl7k\" (UID: \"85529ed7-e2d9-470f-a897-1711fd4abfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k"
Nov 01 03:22:48.978997 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.977877    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltsb7\" (UniqueName: \"kubernetes.io/projected/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-kube-api-access-ltsb7\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-c29ts\" (UID: \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts"
Nov 01 03:22:48.978997 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.977963    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85529ed7-e2d9-470f-a897-1711fd4abfa0-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xpl7k\" (UID: \"85529ed7-e2d9-470f-a897-1711fd4abfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k"
Nov 01 03:22:48.978997 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.977913    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc508d65-2f4b-4cf1-aec1-8d229baef624-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-f4djx\" (UID: \"fc508d65-2f4b-4cf1-aec1-8d229baef624\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx"
Nov 01 03:22:48.978997 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.978234    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-c29ts\" (UID: \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts"
Nov 01 03:22:48.978997 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.978761    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc508d65-2f4b-4cf1-aec1-8d229baef624-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-f4djx\" (UID: \"fc508d65-2f4b-4cf1-aec1-8d229baef624\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx"
Nov 01 03:22:48.979412 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.979012    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-c29ts\" (UID: \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts"
Nov 01 03:22:48.979412 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.979085    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc508d65-2f4b-4cf1-aec1-8d229baef624-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-f4djx\" (UID: \"fc508d65-2f4b-4cf1-aec1-8d229baef624\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx"
Nov 01 03:22:48.979412 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.979384    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc508d65-2f4b-4cf1-aec1-8d229baef624-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-f4djx\" (UID: \"fc508d65-2f4b-4cf1-aec1-8d229baef624\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx"
Nov 01 03:22:48.979835 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.979811    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-c29ts\" (UID: \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts"
Nov 01 03:22:48.979923 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.979466    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-c29ts\" (UID: \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts"
Nov 01 03:22:48.979923 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.979875    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85529ed7-e2d9-470f-a897-1711fd4abfa0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xpl7k\" (UID: \"85529ed7-e2d9-470f-a897-1711fd4abfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k"
Nov 01 03:22:48.979923 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.979909    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc508d65-2f4b-4cf1-aec1-8d229baef624-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-f4djx\" (UID: \"fc508d65-2f4b-4cf1-aec1-8d229baef624\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx"
Nov 01 03:22:48.980072 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.979939    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-c29ts\" (UID: \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts"
Nov 01 03:22:48.980072 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.979922    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-c29ts\" (UID: \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts"
Nov 01 03:22:48.980072 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.979986    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85529ed7-e2d9-470f-a897-1711fd4abfa0-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xpl7k\" (UID: \"85529ed7-e2d9-470f-a897-1711fd4abfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k"
Nov 01 03:22:48.980072 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.980030    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-c29ts\" (UID: \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts"
Nov 01 03:22:48.980072 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.980060    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzkdx\" (UniqueName: \"kubernetes.io/projected/fc508d65-2f4b-4cf1-aec1-8d229baef624-kube-api-access-nzkdx\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-f4djx\" (UID: \"fc508d65-2f4b-4cf1-aec1-8d229baef624\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx"
Nov 01 03:22:48.980358 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.980222    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-c29ts\" (UID: \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts"
Nov 01 03:22:48.980358 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.980336    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85529ed7-e2d9-470f-a897-1711fd4abfa0-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xpl7k\" (UID: \"85529ed7-e2d9-470f-a897-1711fd4abfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k"
Nov 01 03:22:48.980464 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.980365    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-c29ts\" (UID: \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts"
Nov 01 03:22:48.980464 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.980455    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-c29ts\" (UID: \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts"
Nov 01 03:22:48.980600 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.980487    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-c29ts\" (UID: \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts"
Nov 01 03:22:48.980600 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.980529    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc508d65-2f4b-4cf1-aec1-8d229baef624-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-f4djx\" (UID: \"fc508d65-2f4b-4cf1-aec1-8d229baef624\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx"
Nov 01 03:22:48.980600 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.980587    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhnzk\" (UniqueName: \"kubernetes.io/projected/85529ed7-e2d9-470f-a897-1711fd4abfa0-kube-api-access-zhnzk\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xpl7k\" (UID: \"85529ed7-e2d9-470f-a897-1711fd4abfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k"
Nov 01 03:22:48.980761 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.980622    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85529ed7-e2d9-470f-a897-1711fd4abfa0-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xpl7k\" (UID: \"85529ed7-e2d9-470f-a897-1711fd4abfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k"
Nov 01 03:22:48.980761 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.980628    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc508d65-2f4b-4cf1-aec1-8d229baef624-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-f4djx\" (UID: \"fc508d65-2f4b-4cf1-aec1-8d229baef624\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx"
Nov 01 03:22:48.980761 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.980635    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-c29ts\" (UID: \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts"
Nov 01 03:22:48.980761 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.980647    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc508d65-2f4b-4cf1-aec1-8d229baef624-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-f4djx\" (UID: \"fc508d65-2f4b-4cf1-aec1-8d229baef624\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx"
Nov 01 03:22:48.981027 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.981004    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc508d65-2f4b-4cf1-aec1-8d229baef624-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-f4djx\" (UID: \"fc508d65-2f4b-4cf1-aec1-8d229baef624\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx"
Nov 01 03:22:48.981092 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.981068    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc508d65-2f4b-4cf1-aec1-8d229baef624-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-f4djx\" (UID: \"fc508d65-2f4b-4cf1-aec1-8d229baef624\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx"
Nov 01 03:22:48.981359 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.981333    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-c29ts\" (UID: \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts"
Nov 01 03:22:48.982028 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.982005    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc508d65-2f4b-4cf1-aec1-8d229baef624-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-f4djx\" (UID: \"fc508d65-2f4b-4cf1-aec1-8d229baef624\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx"
Nov 01 03:22:48.982853 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.982834    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85529ed7-e2d9-470f-a897-1711fd4abfa0-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xpl7k\" (UID: \"85529ed7-e2d9-470f-a897-1711fd4abfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k"
Nov 01 03:22:48.983206 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:48.983187    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-c29ts\" (UID: \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts"
Nov 01 03:22:49.038105 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:49.038039    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85529ed7-e2d9-470f-a897-1711fd4abfa0-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xpl7k\" (UID: \"85529ed7-e2d9-470f-a897-1711fd4abfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k"
Nov 01 03:22:49.038389 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:49.038364    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltsb7\" (UniqueName: \"kubernetes.io/projected/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-kube-api-access-ltsb7\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-c29ts\" (UID: \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts"
Nov 01 03:22:49.039893 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:49.039869    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-c29ts\" (UID: \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts"
Nov 01 03:22:49.040402 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:49.040376    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc508d65-2f4b-4cf1-aec1-8d229baef624-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-f4djx\" (UID: \"fc508d65-2f4b-4cf1-aec1-8d229baef624\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx"
Nov 01 03:22:49.040483 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:49.040449    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzkdx\" (UniqueName: \"kubernetes.io/projected/fc508d65-2f4b-4cf1-aec1-8d229baef624-kube-api-access-nzkdx\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-f4djx\" (UID: \"fc508d65-2f4b-4cf1-aec1-8d229baef624\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx"
Nov 01 03:22:49.042646 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:49.042618    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhnzk\" (UniqueName: \"kubernetes.io/projected/85529ed7-e2d9-470f-a897-1711fd4abfa0-kube-api-access-zhnzk\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xpl7k\" (UID: \"85529ed7-e2d9-470f-a897-1711fd4abfa0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k"
Nov 01 03:22:49.074204 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:49.074183    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx"
Nov 01 03:22:49.123989 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:22:49.123952    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:22:49.125335 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:49.125144    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k"
Nov 01 03:22:49.132361 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:49.132339    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts"
Nov 01 03:22:49.514306 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:49.514277    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx"]
Nov 01 03:22:49.548162 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:22:49.548137    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc508d65_2f4b_4cf1_aec1_8d229baef624.slice/crio-5d84ec1cbc8fcc73a9e88a2ad8bcd8041110d6f85eef08b441b1102707f3716a WatchSource:0}: Error finding container 5d84ec1cbc8fcc73a9e88a2ad8bcd8041110d6f85eef08b441b1102707f3716a: Status 404 returned error can't find the container with id 5d84ec1cbc8fcc73a9e88a2ad8bcd8041110d6f85eef08b441b1102707f3716a
Nov 01 03:22:49.550547 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:49.550514    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:22:49.550698 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:49.550614    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:22:49.550698 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:49.550665    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:22:49.566931 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:49.566900    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts"]
Nov 01 03:22:49.567016 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:22:49.566984    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba1754d_0939_4b6a_b6d0_69155eb4cfa0.slice/crio-af32df15583507f107ac567a95a663ecac427550458434bed52352e0f7b0bf0c WatchSource:0}: Error finding container af32df15583507f107ac567a95a663ecac427550458434bed52352e0f7b0bf0c: Status 404 returned error can't find the container with id af32df15583507f107ac567a95a663ecac427550458434bed52352e0f7b0bf0c
Nov 01 03:22:49.569919 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:49.569888    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:22:49.570018 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:49.569968    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:22:49.570110 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:49.570015    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:22:49.584411 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:49.584374    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts" event={"ID":"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0","Type":"ContainerStarted","Data":"af32df15583507f107ac567a95a663ecac427550458434bed52352e0f7b0bf0c"}
Nov 01 03:22:49.586127 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:49.586092    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx" event={"ID":"fc508d65-2f4b-4cf1-aec1-8d229baef624","Type":"ContainerStarted","Data":"5d84ec1cbc8fcc73a9e88a2ad8bcd8041110d6f85eef08b441b1102707f3716a"}
Nov 01 03:22:49.621708 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:22:49.621683    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85529ed7_e2d9_470f_a897_1711fd4abfa0.slice/crio-fb296b8c820b2b92da89c3db86b0c01eb0f77914a763b71d1970f5fc6894431e WatchSource:0}: Error finding container fb296b8c820b2b92da89c3db86b0c01eb0f77914a763b71d1970f5fc6894431e: Status 404 returned error can't find the container with id fb296b8c820b2b92da89c3db86b0c01eb0f77914a763b71d1970f5fc6894431e
Nov 01 03:22:49.624684 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:49.624653    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:22:49.624862 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:49.624727    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:22:49.624862 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:49.624762    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:22:49.625171 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:49.625143    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k"]
Nov 01 03:22:50.631278 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:50.631234    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k" event={"ID":"85529ed7-e2d9-470f-a897-1711fd4abfa0","Type":"ContainerStarted","Data":"9018f3823ac66b77bb260a122a93c861dad45b752d4a6b6c361e24e975bbcc51"}
Nov 01 03:22:50.633576 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:50.631882    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k" event={"ID":"85529ed7-e2d9-470f-a897-1711fd4abfa0","Type":"ContainerStarted","Data":"fb296b8c820b2b92da89c3db86b0c01eb0f77914a763b71d1970f5fc6894431e"}
Nov 01 03:22:50.648352 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:50.648311    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts" event={"ID":"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0","Type":"ContainerStarted","Data":"a77bf021a96e4721e04519717998f814ff214ba344930518070f2ec898d1903a"}
Nov 01 03:22:50.650024 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:50.649998    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx" event={"ID":"fc508d65-2f4b-4cf1-aec1-8d229baef624","Type":"ContainerStarted","Data":"12df7a95ac96d8046f62b0152f8172986d41576d330ef368406262d186b53926"}
Nov 01 03:22:50.755710 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:50.755662    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k" podStartSLOduration=2.755649296 podStartE2EDuration="2.755649296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:22:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:22:50.752653208 +0000 UTC m=+975.275659798" watchObservedRunningTime="2026-11-01 03:22:50.755649296 +0000 UTC m=+975.278655884"
Nov 01 03:22:50.903634 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:50.903382    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx" podStartSLOduration=2.903357887 podStartE2EDuration="2.903357887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:22:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:22:50.881203831 +0000 UTC m=+975.404210419" watchObservedRunningTime="2026-11-01 03:22:50.903357887 +0000 UTC m=+975.426364475"
Nov 01 03:22:51.080583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:51.077138    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx"
Nov 01 03:22:51.080583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:51.077267    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-f4djx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 01 03:22:51.080583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:51.077313    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx" podUID="fc508d65-2f4b-4cf1-aec1-8d229baef624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 01 03:22:51.130581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:51.127685    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k"
Nov 01 03:22:51.130581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:51.127824    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 01 03:22:51.130581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:51.127873    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k" podUID="85529ed7-e2d9-470f-a897-1711fd4abfa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 01 03:22:51.136575 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:51.134210    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts"
Nov 01 03:22:51.147580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:51.138243    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-c29ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 01 03:22:51.147580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:51.138293    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts" podUID="4ba1754d-0939-4b6a-b6d0-69155eb4cfa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 01 03:22:52.077577 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:52.077030    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-f4djx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 01 03:22:52.077577 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:52.077146    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx" podUID="fc508d65-2f4b-4cf1-aec1-8d229baef624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 01 03:22:52.132578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:52.131538    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 01 03:22:52.132578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:52.131656    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k" podUID="85529ed7-e2d9-470f-a897-1711fd4abfa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 01 03:22:52.140042 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:52.135124    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-c29ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 01 03:22:52.140042 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:52.135172    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts" podUID="4ba1754d-0939-4b6a-b6d0-69155eb4cfa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 01 03:22:52.376580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:52.366648    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf" podUID="40881383-61bb-4fc8-9c75-f8ffce4834c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:22:52.416580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:52.415516    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp" podUID="667240b0-64a7-45cd-84fe-9cd7e0d8a301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:22:52.460577 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:52.459636    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps" podUID="d658772c-b072-4513-a639-d19a94c78525" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:22:52.541582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:52.530020    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj" podUID="b0e32cb3-ac18-4c1f-819c-f1139a88bfea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:22:52.972353 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:52.972318    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:22:53.040687 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:53.040634    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t" podUID="13930c1e-2200-4412-b8d3-ae3e4126a851" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:22:53.083968 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:53.083933    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-f4djx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 01 03:22:53.106224 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:53.106184    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx" podUID="fc508d65-2f4b-4cf1-aec1-8d229baef624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 01 03:22:53.140963 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:53.140928    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 01 03:22:53.143161 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:53.143128    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k" podUID="85529ed7-e2d9-470f-a897-1711fd4abfa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 01 03:22:53.143359 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:53.143068    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-c29ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 01 03:22:53.143480 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:53.143461    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts" podUID="4ba1754d-0939-4b6a-b6d0-69155eb4cfa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 01 03:22:53.153660 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:53.153629    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n" podUID="d2511772-66f9-40a0-9166-87c4345485c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:22:53.479735 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:53.460199    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf" podUID="40881383-61bb-4fc8-9c75-f8ffce4834c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:22:53.479735 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:53.461156    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:22:53.479735 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:53.461107    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps" podUID="d658772c-b072-4513-a639-d19a94c78525" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:22:53.479735 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:53.463228    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:22:53.589581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:53.589328    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts" podStartSLOduration=5.589309108 podStartE2EDuration="5.589309108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:22:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:22:51.029012071 +0000 UTC m=+975.552018676" watchObservedRunningTime="2026-11-01 03:22:53.589309108 +0000 UTC m=+978.112315698"
Nov 01 03:22:53.706590 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:53.702079    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm" podUID="b2637bc6-3828-4b75-9783-5415519403a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:22:53.750659 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:53.745608    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm"]
Nov 01 03:22:53.766695 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:53.762818    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm"
Nov 01 03:22:53.909267 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:53.909235    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:22:53.939720 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:53.937256    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-workload-certs\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm\" (UID: \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm"
Nov 01 03:22:53.939954 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:53.939930    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-istio-data\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm\" (UID: \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm"
Nov 01 03:22:53.940091 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:53.940072    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-workload-socket\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm\" (UID: \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm"
Nov 01 03:22:53.940242 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:53.940224    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-credential-socket\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm\" (UID: \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm"
Nov 01 03:22:53.940435 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:53.940417    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-istio-podinfo\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm\" (UID: \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm"
Nov 01 03:22:53.940643 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:53.940625    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-istio-envoy\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm\" (UID: \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm"
Nov 01 03:22:53.940784 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:53.940770    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fglzl\" (UniqueName: \"kubernetes.io/projected/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-kube-api-access-fglzl\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm\" (UID: \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm"
Nov 01 03:22:53.940989 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:53.940971    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-istiod-ca-cert\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm\" (UID: \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm"
Nov 01 03:22:53.941129 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:53.941115    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-istio-token\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm\" (UID: \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm"
Nov 01 03:22:54.114323 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:54.114280    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-f4djx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 01 03:22:54.114817 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:54.114342    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx" podUID="fc508d65-2f4b-4cf1-aec1-8d229baef624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 01 03:22:54.131595 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:54.120615    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-istio-podinfo\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm\" (UID: \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm"
Nov 01 03:22:54.131595 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:54.120730    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-istio-envoy\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm\" (UID: \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm"
Nov 01 03:22:54.131595 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:54.120782    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fglzl\" (UniqueName: \"kubernetes.io/projected/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-kube-api-access-fglzl\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm\" (UID: \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm"
Nov 01 03:22:54.131595 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:54.120892    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-istiod-ca-cert\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm\" (UID: \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm"
Nov 01 03:22:54.131595 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:54.120941    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-istio-token\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm\" (UID: \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm"
Nov 01 03:22:54.131595 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:54.120985    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-workload-certs\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm\" (UID: \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm"
Nov 01 03:22:54.131595 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:54.121027    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-istio-data\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm\" (UID: \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm"
Nov 01 03:22:54.131595 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:54.121078    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-workload-socket\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm\" (UID: \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm"
Nov 01 03:22:54.131595 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:54.121111    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-credential-socket\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm\" (UID: \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm"
Nov 01 03:22:54.156035 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:54.156003    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-c29ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 01 03:22:54.156279 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:54.156252    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts" podUID="4ba1754d-0939-4b6a-b6d0-69155eb4cfa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 01 03:22:54.157862 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:54.157637    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-workload-certs\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm\" (UID: \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm"
Nov 01 03:22:54.179828 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:54.158386    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-istiod-ca-cert\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm\" (UID: \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm"
Nov 01 03:22:54.179828 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:54.158680    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-istio-data\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm\" (UID: \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm"
Nov 01 03:22:54.179828 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:54.158908    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-workload-socket\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm\" (UID: \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm"
Nov 01 03:22:54.179828 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:54.159152    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 01 03:22:54.179828 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:54.159195    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k" podUID="85529ed7-e2d9-470f-a897-1711fd4abfa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 01 03:22:54.192723 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:54.121551    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-credential-socket\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm\" (UID: \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm"
Nov 01 03:22:54.256377 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:54.240241    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-istio-podinfo\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm\" (UID: \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm"
Nov 01 03:22:54.256377 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:54.252567    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:22:54.256377 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:54.255959    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-istio-token\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm\" (UID: \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm"
Nov 01 03:22:54.256377 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:54.256257    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-istio-envoy\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm\" (UID: \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm"
Nov 01 03:22:54.309579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:54.294243    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fglzl\" (UniqueName: \"kubernetes.io/projected/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-kube-api-access-fglzl\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm\" (UID: \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm"
Nov 01 03:22:54.335578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:54.319521    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm"]
Nov 01 03:22:54.583632 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:54.582861    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm"
Nov 01 03:22:55.110589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:55.090664    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-f4djx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 01 03:22:55.110589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:55.108633    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx" podUID="fc508d65-2f4b-4cf1-aec1-8d229baef624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 01 03:22:55.135592 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:55.126050    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 01 03:22:55.135592 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:55.126117    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k" podUID="85529ed7-e2d9-470f-a897-1711fd4abfa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 01 03:22:55.143648 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:55.143614    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-c29ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 01 03:22:55.144360 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:55.144288    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts" podUID="4ba1754d-0939-4b6a-b6d0-69155eb4cfa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 01 03:22:55.757265 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:55.757171    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm" podUID="b2637bc6-3828-4b75-9783-5415519403a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:22:56.176638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:56.087102    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-f4djx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 01 03:22:56.176638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:56.087160    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx" podUID="fc508d65-2f4b-4cf1-aec1-8d229baef624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 01 03:22:56.176638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:56.155091    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 01 03:22:56.176638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:56.155165    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k" podUID="85529ed7-e2d9-470f-a897-1711fd4abfa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 01 03:22:56.176638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:56.157048    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-c29ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 01 03:22:56.176638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:56.157092    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts" podUID="4ba1754d-0939-4b6a-b6d0-69155eb4cfa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 01 03:22:56.222802 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:56.222337    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm"]
Nov 01 03:22:56.253857 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:56.253816    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:22:56.253975 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:56.253918    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:22:56.254031 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:56.253972    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:22:57.025050 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:57.025013    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:22:57.038811 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:57.038782    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:22:57.074713 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:57.074680    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-f4djx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 01 03:22:57.074849 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:57.074742    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx" podUID="fc508d65-2f4b-4cf1-aec1-8d229baef624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 01 03:22:57.125646 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:57.125612    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 01 03:22:57.125775 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:57.125667    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k" podUID="85529ed7-e2d9-470f-a897-1711fd4abfa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 01 03:22:57.133613 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:57.133584    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-c29ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 01 03:22:57.133740 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:57.133633    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts" podUID="4ba1754d-0939-4b6a-b6d0-69155eb4cfa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 01 03:22:57.221585 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:57.220598    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm" event={"ID":"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd","Type":"ContainerStarted","Data":"1c365ed9d508a7b0d892d0491d5646a1d5ebdb820ae17d2c13959dd219724c50"}
Nov 01 03:22:57.221585 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:57.220655    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm" event={"ID":"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd","Type":"ContainerStarted","Data":"631b0e1e28cdce8cf89b08a8c16cf739edc1804969d251e89fa0117fcb4f6c6c"}
Nov 01 03:22:57.239042 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:57.239012    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:22:57.351683 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:57.350505    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm" podStartSLOduration=4.350485987 podStartE2EDuration="4.350485987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:22:57.345745711 +0000 UTC m=+981.868752302" watchObservedRunningTime="2026-11-01 03:22:57.350485987 +0000 UTC m=+981.873492575"
Nov 01 03:22:57.596600 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:57.591580    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm"
Nov 01 03:22:57.596600 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:57.591712    4930 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 01 03:22:57.596600 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:57.591757    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm" podUID="accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 01 03:22:57.967163 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:57.967126    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:22:58.078022 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:58.077987    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-f4djx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 01 03:22:58.078310 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:58.078269    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx" podUID="fc508d65-2f4b-4cf1-aec1-8d229baef624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 01 03:22:58.130185 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:58.130150    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Nov 01 03:22:58.130589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:58.130546    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k" podUID="85529ed7-e2d9-470f-a897-1711fd4abfa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Nov 01 03:22:58.132957 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:58.132936    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-c29ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Nov 01 03:22:58.133100 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:58.133078    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts" podUID="4ba1754d-0939-4b6a-b6d0-69155eb4cfa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Nov 01 03:22:58.204706 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:58.200304    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-495d744b-wglnt"]
Nov 01 03:22:58.213032 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:58.213003    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495d744b-wglnt"
Nov 01 03:22:58.233377 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:58.233346    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqhpz\" (UniqueName: \"kubernetes.io/projected/80c056a8-53c1-4778-af88-3a8ac446a6d1-kube-api-access-pqhpz\") pod \"maas-api-revoke-sub-495d744b-wglnt\" (UID: \"80c056a8-53c1-4778-af88-3a8ac446a6d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495d744b-wglnt"
Nov 01 03:22:58.233912 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:58.233885    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80c056a8-53c1-4778-af88-3a8ac446a6d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-495d744b-wglnt\" (UID: \"80c056a8-53c1-4778-af88-3a8ac446a6d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495d744b-wglnt"
Nov 01 03:22:58.234140 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:58.234111    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80c056a8-53c1-4778-af88-3a8ac446a6d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-495d744b-wglnt\" (UID: \"80c056a8-53c1-4778-af88-3a8ac446a6d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495d744b-wglnt"
Nov 01 03:22:58.245969 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:58.245944    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-495d744b-wglnt"]
Nov 01 03:22:58.315652 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:58.315609    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:22:58.315826 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:58.315748    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:22:58.315892 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:58.315839    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:22:58.315948 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:58.315891    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:22:58.334924 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:58.334887    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80c056a8-53c1-4778-af88-3a8ac446a6d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-495d744b-wglnt\" (UID: \"80c056a8-53c1-4778-af88-3a8ac446a6d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495d744b-wglnt"
Nov 01 03:22:58.335135 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:58.335116    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80c056a8-53c1-4778-af88-3a8ac446a6d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-495d744b-wglnt\" (UID: \"80c056a8-53c1-4778-af88-3a8ac446a6d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495d744b-wglnt"
Nov 01 03:22:58.335270 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:58.335254    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqhpz\" (UniqueName: \"kubernetes.io/projected/80c056a8-53c1-4778-af88-3a8ac446a6d1-kube-api-access-pqhpz\") pod \"maas-api-revoke-sub-495d744b-wglnt\" (UID: \"80c056a8-53c1-4778-af88-3a8ac446a6d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495d744b-wglnt"
Nov 01 03:22:58.336274 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:58.336247    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80c056a8-53c1-4778-af88-3a8ac446a6d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-495d744b-wglnt\" (UID: \"80c056a8-53c1-4778-af88-3a8ac446a6d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495d744b-wglnt"
Nov 01 03:22:58.340257 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:58.340236    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80c056a8-53c1-4778-af88-3a8ac446a6d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-495d744b-wglnt\" (UID: \"80c056a8-53c1-4778-af88-3a8ac446a6d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495d744b-wglnt"
Nov 01 03:22:58.417901 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:58.417868    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqhpz\" (UniqueName: \"kubernetes.io/projected/80c056a8-53c1-4778-af88-3a8ac446a6d1-kube-api-access-pqhpz\") pod \"maas-api-revoke-sub-495d744b-wglnt\" (UID: \"80c056a8-53c1-4778-af88-3a8ac446a6d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-495d744b-wglnt"
Nov 01 03:22:58.539861 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:58.539770    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495d744b-wglnt"
Nov 01 03:22:58.597587 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:58.597348    4930 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 01 03:22:58.597587 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:58.597415    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm" podUID="accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 01 03:22:59.587354 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:59.587317    4930 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 01 03:22:59.587973 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:59.587938    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm" podUID="accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 01 03:22:59.809810 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:22:59.809592    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-495d744b-wglnt"]
Nov 01 03:23:00.074589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:00.068643    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj" podUID="b0e32cb3-ac18-4c1f-819c-f1139a88bfea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded"
Nov 01 03:23:00.125774 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:00.125635    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-f4djx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 03:23:00.125774 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:00.125695    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx" podUID="fc508d65-2f4b-4cf1-aec1-8d229baef624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:23:00.145583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:00.141633    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw" podUID="b6d688b8-b911-4a6a-a842-2b0fb7a75e83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:23:00.210241 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:00.209835    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 03:23:00.210241 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:00.209904    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k" podUID="85529ed7-e2d9-470f-a897-1711fd4abfa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:23:00.259027 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:00.258019    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-c29ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 03:23:00.259027 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:00.258078    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts" podUID="4ba1754d-0939-4b6a-b6d0-69155eb4cfa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:23:00.452703 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:00.451893    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495d744b-wglnt" event={"ID":"80c056a8-53c1-4778-af88-3a8ac446a6d1","Type":"ContainerStarted","Data":"13e611e45bc7a71f14948c2c0a1abb97d97093b7a02ce5e2fc4e2d56332fdf12"}
Nov 01 03:23:00.452703 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:00.451940    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495d744b-wglnt" event={"ID":"80c056a8-53c1-4778-af88-3a8ac446a6d1","Type":"ContainerStarted","Data":"0fe22f92a0c6a4f86fe9a6111322a1631e2936f54fbce456be3b72b0199b419e"}
Nov 01 03:23:00.583919 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:00.583884    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts"
Nov 01 03:23:00.585335 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:00.585313    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts"
Nov 01 03:23:00.591465 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:00.591410    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495d744b-wglnt" podStartSLOduration=2.591393201 podStartE2EDuration="2.591393201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:22:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:23:00.585850329 +0000 UTC m=+985.108856918" watchObservedRunningTime="2026-11-01 03:23:00.591393201 +0000 UTC m=+985.114399790"
Nov 01 03:23:00.598576 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:00.593263    4930 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 01 03:23:00.598576 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:00.594662    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm" podUID="accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 01 03:23:00.602838 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:00.602811    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts"
Nov 01 03:23:00.609212 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:00.609188    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dee007ba-w6t4t"]
Nov 01 03:23:00.614996 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:00.614977    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee007ba-w6t4t"
Nov 01 03:23:00.642185 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:00.642155    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx"
Nov 01 03:23:00.644069 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:00.644034    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx"
Nov 01 03:23:00.645493 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:00.645447    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx"
Nov 01 03:23:00.696632 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:00.695268    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dee007ba-w6t4t"]
Nov 01 03:23:00.734773 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:00.734739    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/505adf30-6661-40ae-969b-5936e9ae4879-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dee007ba-w6t4t\" (UID: \"505adf30-6661-40ae-969b-5936e9ae4879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee007ba-w6t4t"
Nov 01 03:23:00.736142 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:00.736117    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v8f7\" (UniqueName: \"kubernetes.io/projected/505adf30-6661-40ae-969b-5936e9ae4879-kube-api-access-5v8f7\") pod \"maas-api-revoke-sub-dee007ba-w6t4t\" (UID: \"505adf30-6661-40ae-969b-5936e9ae4879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee007ba-w6t4t"
Nov 01 03:23:00.736504 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:00.736486    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/505adf30-6661-40ae-969b-5936e9ae4879-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dee007ba-w6t4t\" (UID: \"505adf30-6661-40ae-969b-5936e9ae4879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee007ba-w6t4t"
Nov 01 03:23:00.838154 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:00.838107    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/505adf30-6661-40ae-969b-5936e9ae4879-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dee007ba-w6t4t\" (UID: \"505adf30-6661-40ae-969b-5936e9ae4879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee007ba-w6t4t"
Nov 01 03:23:00.838325 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:00.838178    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v8f7\" (UniqueName: \"kubernetes.io/projected/505adf30-6661-40ae-969b-5936e9ae4879-kube-api-access-5v8f7\") pod \"maas-api-revoke-sub-dee007ba-w6t4t\" (UID: \"505adf30-6661-40ae-969b-5936e9ae4879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee007ba-w6t4t"
Nov 01 03:23:00.838325 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:00.838246    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/505adf30-6661-40ae-969b-5936e9ae4879-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dee007ba-w6t4t\" (UID: \"505adf30-6661-40ae-969b-5936e9ae4879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee007ba-w6t4t"
Nov 01 03:23:00.839092 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:00.839058    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/505adf30-6661-40ae-969b-5936e9ae4879-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dee007ba-w6t4t\" (UID: \"505adf30-6661-40ae-969b-5936e9ae4879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee007ba-w6t4t"
Nov 01 03:23:00.840858 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:00.840833    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/505adf30-6661-40ae-969b-5936e9ae4879-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dee007ba-w6t4t\" (UID: \"505adf30-6661-40ae-969b-5936e9ae4879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee007ba-w6t4t"
Nov 01 03:23:00.927224 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:00.923631    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v8f7\" (UniqueName: \"kubernetes.io/projected/505adf30-6661-40ae-969b-5936e9ae4879-kube-api-access-5v8f7\") pod \"maas-api-revoke-sub-dee007ba-w6t4t\" (UID: \"505adf30-6661-40ae-969b-5936e9ae4879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee007ba-w6t4t"
Nov 01 03:23:00.948265 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:00.948183    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee007ba-w6t4t"
Nov 01 03:23:01.217370 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:01.217267    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 03:23:01.217971 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:01.217936    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k" podUID="85529ed7-e2d9-470f-a897-1711fd4abfa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:23:01.245684 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:01.238631    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k"
Nov 01 03:23:01.245684 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:01.244590    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k"
Nov 01 03:23:01.245684 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:01.244653    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d752352-2z46j"]
Nov 01 03:23:01.276619 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:01.256065    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d752352-2z46j"
Nov 01 03:23:01.332548 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:01.332476    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k"
Nov 01 03:23:01.340117 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:01.340088    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d752352-2z46j"]
Nov 01 03:23:01.427326 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:01.427289    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dn5d\" (UniqueName: \"kubernetes.io/projected/a57688db-f33f-4e22-a356-48b95564e4ae-kube-api-access-5dn5d\") pod \"maas-api-revoke-sub-6d752352-2z46j\" (UID: \"a57688db-f33f-4e22-a356-48b95564e4ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d752352-2z46j"
Nov 01 03:23:01.427681 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:01.427658    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a57688db-f33f-4e22-a356-48b95564e4ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d752352-2z46j\" (UID: \"a57688db-f33f-4e22-a356-48b95564e4ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d752352-2z46j"
Nov 01 03:23:01.427901 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:01.427884    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a57688db-f33f-4e22-a356-48b95564e4ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d752352-2z46j\" (UID: \"a57688db-f33f-4e22-a356-48b95564e4ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d752352-2z46j"
Nov 01 03:23:01.465653 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:01.465604    4930 generic.go:356] "Generic (PLEG): container finished" podID="80c056a8-53c1-4778-af88-3a8ac446a6d1" containerID="13e611e45bc7a71f14948c2c0a1abb97d97093b7a02ce5e2fc4e2d56332fdf12" exitCode=0
Nov 01 03:23:01.467160 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:01.467122    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495d744b-wglnt" event={"ID":"80c056a8-53c1-4778-af88-3a8ac446a6d1","Type":"ContainerDied","Data":"13e611e45bc7a71f14948c2c0a1abb97d97093b7a02ce5e2fc4e2d56332fdf12"}
Nov 01 03:23:01.529206 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:01.529111    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dn5d\" (UniqueName: \"kubernetes.io/projected/a57688db-f33f-4e22-a356-48b95564e4ae-kube-api-access-5dn5d\") pod \"maas-api-revoke-sub-6d752352-2z46j\" (UID: \"a57688db-f33f-4e22-a356-48b95564e4ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d752352-2z46j"
Nov 01 03:23:01.529206 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:01.529162    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a57688db-f33f-4e22-a356-48b95564e4ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d752352-2z46j\" (UID: \"a57688db-f33f-4e22-a356-48b95564e4ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d752352-2z46j"
Nov 01 03:23:01.529206 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:01.529204    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a57688db-f33f-4e22-a356-48b95564e4ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d752352-2z46j\" (UID: \"a57688db-f33f-4e22-a356-48b95564e4ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d752352-2z46j"
Nov 01 03:23:01.530252 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:01.530228    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a57688db-f33f-4e22-a356-48b95564e4ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d752352-2z46j\" (UID: \"a57688db-f33f-4e22-a356-48b95564e4ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d752352-2z46j"
Nov 01 03:23:01.538600 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:01.538542    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a57688db-f33f-4e22-a356-48b95564e4ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d752352-2z46j\" (UID: \"a57688db-f33f-4e22-a356-48b95564e4ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d752352-2z46j"
Nov 01 03:23:01.632604 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:01.621895    4930 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 01 03:23:01.632604 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:01.621961    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm" podUID="accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 01 03:23:01.632604 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:01.629178    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dn5d\" (UniqueName: \"kubernetes.io/projected/a57688db-f33f-4e22-a356-48b95564e4ae-kube-api-access-5dn5d\") pod \"maas-api-revoke-sub-6d752352-2z46j\" (UID: \"a57688db-f33f-4e22-a356-48b95564e4ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d752352-2z46j"
Nov 01 03:23:01.643771 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:01.643741    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dee007ba-w6t4t"]
Nov 01 03:23:01.712076 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:01.710982    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d752352-2z46j"
Nov 01 03:23:02.358899 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:02.358804    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d752352-2z46j"]
Nov 01 03:23:02.599341 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:02.599283    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d752352-2z46j" event={"ID":"a57688db-f33f-4e22-a356-48b95564e4ae","Type":"ContainerStarted","Data":"0e8ef37653bd1b22dbafea834478bbd3a047f3f00ac22cb7273619e021206756"}
Nov 01 03:23:02.621582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:02.610657    4930 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 01 03:23:02.621879 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:02.621844    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm" podUID="accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 01 03:23:02.646224 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:02.646188    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee007ba-w6t4t" event={"ID":"505adf30-6661-40ae-969b-5936e9ae4879","Type":"ContainerStarted","Data":"105bb53f1f5c178985a2ab52f829caa56ca33545699a3b97c4db8ef8ff3e80a5"}
Nov 01 03:23:02.666086 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:02.666056    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee007ba-w6t4t" event={"ID":"505adf30-6661-40ae-969b-5936e9ae4879","Type":"ContainerStarted","Data":"ca46fd46dd1f9631e55210d2feed3876dc22c7765dc945d811e8a51c59e0b856"}
Nov 01 03:23:02.778343 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:02.778181    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:23:02.821316 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:02.815835    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee007ba-w6t4t" podStartSLOduration=2.815814155 podStartE2EDuration="2.815814155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:23:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:23:02.814852212 +0000 UTC m=+987.337858801" watchObservedRunningTime="2026-11-01 03:23:02.815814155 +0000 UTC m=+987.338820745"
Nov 01 03:23:03.403492 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:03.403455    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495d744b-wglnt"
Nov 01 03:23:03.469426 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:03.469386    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80c056a8-53c1-4778-af88-3a8ac446a6d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80c056a8-53c1-4778-af88-3a8ac446a6d1-maas-api-xks-ca\") pod \"80c056a8-53c1-4778-af88-3a8ac446a6d1\" (UID: \"80c056a8-53c1-4778-af88-3a8ac446a6d1\") "
Nov 01 03:23:03.469832 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:03.469806    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80c056a8-53c1-4778-af88-3a8ac446a6d1-kube-api-access-pqhpz\" (UniqueName: \"kubernetes.io/projected/80c056a8-53c1-4778-af88-3a8ac446a6d1-kube-api-access-pqhpz\") pod \"80c056a8-53c1-4778-af88-3a8ac446a6d1\" (UID: \"80c056a8-53c1-4778-af88-3a8ac446a6d1\") "
Nov 01 03:23:03.470052 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:03.470032    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80c056a8-53c1-4778-af88-3a8ac446a6d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80c056a8-53c1-4778-af88-3a8ac446a6d1-maas-api-openshift-ca\") pod \"80c056a8-53c1-4778-af88-3a8ac446a6d1\" (UID: \"80c056a8-53c1-4778-af88-3a8ac446a6d1\") "
Nov 01 03:23:03.551646 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:03.551612    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80c056a8-53c1-4778-af88-3a8ac446a6d1-maas-api-openshift-ca" pod "80c056a8-53c1-4778-af88-3a8ac446a6d1" (UID: "80c056a8-53c1-4778-af88-3a8ac446a6d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:23:03.615602 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:03.585787    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80c056a8-53c1-4778-af88-3a8ac446a6d1-maas-api-xks-ca" pod "80c056a8-53c1-4778-af88-3a8ac446a6d1" (UID: "80c056a8-53c1-4778-af88-3a8ac446a6d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:23:03.615602 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:03.587242    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80c056a8-53c1-4778-af88-3a8ac446a6d1-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:23:03.615602 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:03.587272    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80c056a8-53c1-4778-af88-3a8ac446a6d1-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:23:03.615602 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:03.587675    4930 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Nov 01 03:23:03.615602 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:03.587734    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm" podUID="accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Nov 01 03:23:03.615602 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:03.594373    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80c056a8-53c1-4778-af88-3a8ac446a6d1-kube-api-access-pqhpz" pod "80c056a8-53c1-4778-af88-3a8ac446a6d1" (UID: "80c056a8-53c1-4778-af88-3a8ac446a6d1"). InnerVolumeSpecName "kube-api-access-pqhpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:23:03.697636 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:03.695647    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqhpz\" (UniqueName: \"kubernetes.io/projected/80c056a8-53c1-4778-af88-3a8ac446a6d1-kube-api-access-pqhpz\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:23:03.840578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:03.825706    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d752352-2z46j" event={"ID":"a57688db-f33f-4e22-a356-48b95564e4ae","Type":"ContainerStarted","Data":"fcdc66c8210b4f4612e8c0e01599464bcb45d1f599c5b44cdc41269df15fb66e"}
Nov 01 03:23:03.840578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:03.828835    4930 generic.go:356] "Generic (PLEG): container finished" podID="505adf30-6661-40ae-969b-5936e9ae4879" containerID="105bb53f1f5c178985a2ab52f829caa56ca33545699a3b97c4db8ef8ff3e80a5" exitCode=0
Nov 01 03:23:03.840578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:03.829026    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee007ba-w6t4t" event={"ID":"505adf30-6661-40ae-969b-5936e9ae4879","Type":"ContainerDied","Data":"105bb53f1f5c178985a2ab52f829caa56ca33545699a3b97c4db8ef8ff3e80a5"}
Nov 01 03:23:03.845577 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:03.843789    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495d744b-wglnt" event={"ID":"80c056a8-53c1-4778-af88-3a8ac446a6d1","Type":"ContainerDied","Data":"0fe22f92a0c6a4f86fe9a6111322a1631e2936f54fbce456be3b72b0199b419e"}
Nov 01 03:23:03.845577 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:03.843844    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fe22f92a0c6a4f86fe9a6111322a1631e2936f54fbce456be3b72b0199b419e"
Nov 01 03:23:03.845577 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:03.843921    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-495d744b-wglnt"
Nov 01 03:23:03.943646 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:03.943583    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d752352-2z46j" podStartSLOduration=2.9435499910000003 podStartE2EDuration="2.943549991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:23:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:23:03.941531162 +0000 UTC m=+988.464537750" watchObservedRunningTime="2026-11-01 03:23:03.943549991 +0000 UTC m=+988.466556578"
Nov 01 03:23:04.590879 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:04.590852    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm"
Nov 01 03:23:04.591306 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:04.591289    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm"
Nov 01 03:23:04.591970 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:04.591951    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm"
Nov 01 03:23:04.857916 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:04.857833    4930 generic.go:356] "Generic (PLEG): container finished" podID="a57688db-f33f-4e22-a356-48b95564e4ae" containerID="fcdc66c8210b4f4612e8c0e01599464bcb45d1f599c5b44cdc41269df15fb66e" exitCode=0
Nov 01 03:23:04.863364 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:04.863331    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d752352-2z46j" event={"ID":"a57688db-f33f-4e22-a356-48b95564e4ae","Type":"ContainerDied","Data":"fcdc66c8210b4f4612e8c0e01599464bcb45d1f599c5b44cdc41269df15fb66e"}
Nov 01 03:23:05.035249 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:05.035225    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee007ba-w6t4t"
Nov 01 03:23:05.124610 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:05.124507    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/505adf30-6661-40ae-969b-5936e9ae4879-kube-api-access-5v8f7\" (UniqueName: \"kubernetes.io/projected/505adf30-6661-40ae-969b-5936e9ae4879-kube-api-access-5v8f7\") pod \"505adf30-6661-40ae-969b-5936e9ae4879\" (UID: \"505adf30-6661-40ae-969b-5936e9ae4879\") "
Nov 01 03:23:05.124610 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:05.124545    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/505adf30-6661-40ae-969b-5936e9ae4879-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/505adf30-6661-40ae-969b-5936e9ae4879-maas-api-openshift-ca\") pod \"505adf30-6661-40ae-969b-5936e9ae4879\" (UID: \"505adf30-6661-40ae-969b-5936e9ae4879\") "
Nov 01 03:23:05.124610 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:05.124604    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/505adf30-6661-40ae-969b-5936e9ae4879-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/505adf30-6661-40ae-969b-5936e9ae4879-maas-api-xks-ca\") pod \"505adf30-6661-40ae-969b-5936e9ae4879\" (UID: \"505adf30-6661-40ae-969b-5936e9ae4879\") "
Nov 01 03:23:05.124937 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:05.124911    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/505adf30-6661-40ae-969b-5936e9ae4879-maas-api-openshift-ca" pod "505adf30-6661-40ae-969b-5936e9ae4879" (UID: "505adf30-6661-40ae-969b-5936e9ae4879"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:23:05.125089 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:05.125071    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/505adf30-6661-40ae-969b-5936e9ae4879-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:23:05.127597 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:05.126788    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/505adf30-6661-40ae-969b-5936e9ae4879-maas-api-xks-ca" pod "505adf30-6661-40ae-969b-5936e9ae4879" (UID: "505adf30-6661-40ae-969b-5936e9ae4879"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:23:05.127597 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:05.126880    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/505adf30-6661-40ae-969b-5936e9ae4879-kube-api-access-5v8f7" pod "505adf30-6661-40ae-969b-5936e9ae4879" (UID: "505adf30-6661-40ae-969b-5936e9ae4879"). InnerVolumeSpecName "kube-api-access-5v8f7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:23:05.226121 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:05.226056    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v8f7\" (UniqueName: \"kubernetes.io/projected/505adf30-6661-40ae-969b-5936e9ae4879-kube-api-access-5v8f7\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:23:05.226121 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:05.226094    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/505adf30-6661-40ae-969b-5936e9ae4879-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:23:05.871780 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:05.871749    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee007ba-w6t4t"
Nov 01 03:23:05.872896 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:05.871748    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee007ba-w6t4t" event={"ID":"505adf30-6661-40ae-969b-5936e9ae4879","Type":"ContainerDied","Data":"ca46fd46dd1f9631e55210d2feed3876dc22c7765dc945d811e8a51c59e0b856"}
Nov 01 03:23:05.872896 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:05.872872    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca46fd46dd1f9631e55210d2feed3876dc22c7765dc945d811e8a51c59e0b856"
Nov 01 03:23:06.077847 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:06.077818    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d752352-2z46j"
Nov 01 03:23:06.143215 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:06.143187    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a57688db-f33f-4e22-a356-48b95564e4ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a57688db-f33f-4e22-a356-48b95564e4ae-maas-api-xks-ca\") pod \"a57688db-f33f-4e22-a356-48b95564e4ae\" (UID: \"a57688db-f33f-4e22-a356-48b95564e4ae\") "
Nov 01 03:23:06.143395 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:06.143269    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a57688db-f33f-4e22-a356-48b95564e4ae-kube-api-access-5dn5d\" (UniqueName: \"kubernetes.io/projected/a57688db-f33f-4e22-a356-48b95564e4ae-kube-api-access-5dn5d\") pod \"a57688db-f33f-4e22-a356-48b95564e4ae\" (UID: \"a57688db-f33f-4e22-a356-48b95564e4ae\") "
Nov 01 03:23:06.143395 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:06.143308    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a57688db-f33f-4e22-a356-48b95564e4ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a57688db-f33f-4e22-a356-48b95564e4ae-maas-api-openshift-ca\") pod \"a57688db-f33f-4e22-a356-48b95564e4ae\" (UID: \"a57688db-f33f-4e22-a356-48b95564e4ae\") "
Nov 01 03:23:06.143739 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:06.143711    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a57688db-f33f-4e22-a356-48b95564e4ae-maas-api-openshift-ca" pod "a57688db-f33f-4e22-a356-48b95564e4ae" (UID: "a57688db-f33f-4e22-a356-48b95564e4ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:23:06.145217 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:06.145196    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a57688db-f33f-4e22-a356-48b95564e4ae-maas-api-xks-ca" pod "a57688db-f33f-4e22-a356-48b95564e4ae" (UID: "a57688db-f33f-4e22-a356-48b95564e4ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:23:06.145433 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:06.145404    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a57688db-f33f-4e22-a356-48b95564e4ae-kube-api-access-5dn5d" pod "a57688db-f33f-4e22-a356-48b95564e4ae" (UID: "a57688db-f33f-4e22-a356-48b95564e4ae"). InnerVolumeSpecName "kube-api-access-5dn5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:23:06.236969 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:06.236901    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4304e0c2-86r5h"]
Nov 01 03:23:06.237415 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:06.237399    4930 cpu_manager.go:414] "containerMap: removing container" podUID="505adf30-6661-40ae-969b-5936e9ae4879" containerName="revoke-keys"
Nov 01 03:23:06.237506 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:06.237415    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="505adf30-6661-40ae-969b-5936e9ae4879" containerName="revoke-keys"
Nov 01 03:23:06.237506 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:06.237431    4930 cpu_manager.go:414] "containerMap: removing container" podUID="80c056a8-53c1-4778-af88-3a8ac446a6d1" containerName="revoke-keys"
Nov 01 03:23:06.237506 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:06.237436    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80c056a8-53c1-4778-af88-3a8ac446a6d1" containerName="revoke-keys"
Nov 01 03:23:06.237506 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:06.237463    4930 cpu_manager.go:414] "containerMap: removing container" podUID="a57688db-f33f-4e22-a356-48b95564e4ae" containerName="revoke-keys"
Nov 01 03:23:06.237506 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:06.237468    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a57688db-f33f-4e22-a356-48b95564e4ae" containerName="revoke-keys"
Nov 01 03:23:06.237780 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:06.237567    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="80c056a8-53c1-4778-af88-3a8ac446a6d1" containerName="revoke-keys"
Nov 01 03:23:06.237780 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:06.237586    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="a57688db-f33f-4e22-a356-48b95564e4ae" containerName="revoke-keys"
Nov 01 03:23:06.237780 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:06.237601    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="505adf30-6661-40ae-969b-5936e9ae4879" containerName="revoke-keys"
Nov 01 03:23:06.241040 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:06.241021    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4304e0c2-86r5h"
Nov 01 03:23:06.244352 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:06.244330    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a57688db-f33f-4e22-a356-48b95564e4ae-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:23:06.244352 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:06.244353    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dn5d\" (UniqueName: \"kubernetes.io/projected/a57688db-f33f-4e22-a356-48b95564e4ae-kube-api-access-5dn5d\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:23:06.244490 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:06.244364    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a57688db-f33f-4e22-a356-48b95564e4ae-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:23:06.322974 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:06.322941    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4304e0c2-86r5h"]
Nov 01 03:23:06.345366 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:06.345326    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6da1345e-9204-4023-b610-f31e4c3c8250-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4304e0c2-86r5h\" (UID: \"6da1345e-9204-4023-b610-f31e4c3c8250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4304e0c2-86r5h"
Nov 01 03:23:06.345548 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:06.345487    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnjv8\" (UniqueName: \"kubernetes.io/projected/6da1345e-9204-4023-b610-f31e4c3c8250-kube-api-access-nnjv8\") pod \"maas-api-revoke-sub-4304e0c2-86r5h\" (UID: \"6da1345e-9204-4023-b610-f31e4c3c8250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4304e0c2-86r5h"
Nov 01 03:23:06.345632 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:06.345541    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6da1345e-9204-4023-b610-f31e4c3c8250-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4304e0c2-86r5h\" (UID: \"6da1345e-9204-4023-b610-f31e4c3c8250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4304e0c2-86r5h"
Nov 01 03:23:06.447155 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:06.447123    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6da1345e-9204-4023-b610-f31e4c3c8250-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4304e0c2-86r5h\" (UID: \"6da1345e-9204-4023-b610-f31e4c3c8250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4304e0c2-86r5h"
Nov 01 03:23:06.447306 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:06.447188    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnjv8\" (UniqueName: \"kubernetes.io/projected/6da1345e-9204-4023-b610-f31e4c3c8250-kube-api-access-nnjv8\") pod \"maas-api-revoke-sub-4304e0c2-86r5h\" (UID: \"6da1345e-9204-4023-b610-f31e4c3c8250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4304e0c2-86r5h"
Nov 01 03:23:06.447306 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:06.447214    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6da1345e-9204-4023-b610-f31e4c3c8250-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4304e0c2-86r5h\" (UID: \"6da1345e-9204-4023-b610-f31e4c3c8250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4304e0c2-86r5h"
Nov 01 03:23:06.447914 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:06.447887    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6da1345e-9204-4023-b610-f31e4c3c8250-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4304e0c2-86r5h\" (UID: \"6da1345e-9204-4023-b610-f31e4c3c8250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4304e0c2-86r5h"
Nov 01 03:23:06.449514 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:06.449485    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6da1345e-9204-4023-b610-f31e4c3c8250-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4304e0c2-86r5h\" (UID: \"6da1345e-9204-4023-b610-f31e4c3c8250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4304e0c2-86r5h"
Nov 01 03:23:06.508145 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:06.508074    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnjv8\" (UniqueName: \"kubernetes.io/projected/6da1345e-9204-4023-b610-f31e4c3c8250-kube-api-access-nnjv8\") pod \"maas-api-revoke-sub-4304e0c2-86r5h\" (UID: \"6da1345e-9204-4023-b610-f31e4c3c8250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4304e0c2-86r5h"
Nov 01 03:23:06.558327 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:06.558292    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4304e0c2-86r5h"
Nov 01 03:23:06.881415 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:06.881389    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d752352-2z46j"
Nov 01 03:23:06.881798 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:06.881420    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d752352-2z46j" event={"ID":"a57688db-f33f-4e22-a356-48b95564e4ae","Type":"ContainerDied","Data":"0e8ef37653bd1b22dbafea834478bbd3a047f3f00ac22cb7273619e021206756"}
Nov 01 03:23:06.881798 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:06.881464    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e8ef37653bd1b22dbafea834478bbd3a047f3f00ac22cb7273619e021206756"
Nov 01 03:23:06.948154 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:06.944348    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4304e0c2-86r5h"]
Nov 01 03:23:07.540816 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:07.540739    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:23:07.892698 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:07.892662    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4304e0c2-86r5h" event={"ID":"6da1345e-9204-4023-b610-f31e4c3c8250","Type":"ContainerStarted","Data":"3abc7e643fba1b0d2a6c5daa16d67ea024db19e35cd0561072b3a5acccb3ddad"}
Nov 01 03:23:07.893092 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:07.892718    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4304e0c2-86r5h" event={"ID":"6da1345e-9204-4023-b610-f31e4c3c8250","Type":"ContainerStarted","Data":"0f0a9165d2adab2c5a5a2160edecb39395b84e2d6f70118de78d9f1c1fff4819"}
Nov 01 03:23:08.011384 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:08.011334    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4304e0c2-86r5h" podStartSLOduration=2.011321526 podStartE2EDuration="2.011321526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:23:08.010564506 +0000 UTC m=+992.533571085" watchObservedRunningTime="2026-11-01 03:23:08.011321526 +0000 UTC m=+992.534328115"
Nov 01 03:23:08.909913 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:08.909880    4930 generic.go:356] "Generic (PLEG): container finished" podID="6da1345e-9204-4023-b610-f31e4c3c8250" containerID="3abc7e643fba1b0d2a6c5daa16d67ea024db19e35cd0561072b3a5acccb3ddad" exitCode=0
Nov 01 03:23:08.910416 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:08.909967    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4304e0c2-86r5h" event={"ID":"6da1345e-9204-4023-b610-f31e4c3c8250","Type":"ContainerDied","Data":"3abc7e643fba1b0d2a6c5daa16d67ea024db19e35cd0561072b3a5acccb3ddad"}
Nov 01 03:23:10.068544 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:10.068515    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4304e0c2-86r5h"
Nov 01 03:23:10.207489 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:10.207413    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6da1345e-9204-4023-b610-f31e4c3c8250-kube-api-access-nnjv8\" (UniqueName: \"kubernetes.io/projected/6da1345e-9204-4023-b610-f31e4c3c8250-kube-api-access-nnjv8\") pod \"6da1345e-9204-4023-b610-f31e4c3c8250\" (UID: \"6da1345e-9204-4023-b610-f31e4c3c8250\") "
Nov 01 03:23:10.207489 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:10.207482    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6da1345e-9204-4023-b610-f31e4c3c8250-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6da1345e-9204-4023-b610-f31e4c3c8250-maas-api-xks-ca\") pod \"6da1345e-9204-4023-b610-f31e4c3c8250\" (UID: \"6da1345e-9204-4023-b610-f31e4c3c8250\") "
Nov 01 03:23:10.207687 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:10.207577    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6da1345e-9204-4023-b610-f31e4c3c8250-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6da1345e-9204-4023-b610-f31e4c3c8250-maas-api-openshift-ca\") pod \"6da1345e-9204-4023-b610-f31e4c3c8250\" (UID: \"6da1345e-9204-4023-b610-f31e4c3c8250\") "
Nov 01 03:23:10.208145 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:10.207962    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6da1345e-9204-4023-b610-f31e4c3c8250-maas-api-openshift-ca" pod "6da1345e-9204-4023-b610-f31e4c3c8250" (UID: "6da1345e-9204-4023-b610-f31e4c3c8250"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:23:10.208145 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:10.208119    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6da1345e-9204-4023-b610-f31e4c3c8250-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:23:10.209672 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:10.209645    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6da1345e-9204-4023-b610-f31e4c3c8250-maas-api-xks-ca" pod "6da1345e-9204-4023-b610-f31e4c3c8250" (UID: "6da1345e-9204-4023-b610-f31e4c3c8250"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:23:10.210069 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:10.210045    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6da1345e-9204-4023-b610-f31e4c3c8250-kube-api-access-nnjv8" pod "6da1345e-9204-4023-b610-f31e4c3c8250" (UID: "6da1345e-9204-4023-b610-f31e4c3c8250"). InnerVolumeSpecName "kube-api-access-nnjv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:23:10.309016 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:10.308980    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnjv8\" (UniqueName: \"kubernetes.io/projected/6da1345e-9204-4023-b610-f31e4c3c8250-kube-api-access-nnjv8\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:23:10.309016 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:10.309012    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6da1345e-9204-4023-b610-f31e4c3c8250-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:23:10.925300 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:10.925271    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4304e0c2-86r5h"
Nov 01 03:23:10.925506 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:10.925304    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4304e0c2-86r5h" event={"ID":"6da1345e-9204-4023-b610-f31e4c3c8250","Type":"ContainerDied","Data":"0f0a9165d2adab2c5a5a2160edecb39395b84e2d6f70118de78d9f1c1fff4819"}
Nov 01 03:23:10.925506 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:10.925352    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f0a9165d2adab2c5a5a2160edecb39395b84e2d6f70118de78d9f1c1fff4819"
Nov 01 03:23:11.367438 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:11.367407    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-034045af-xq7ns"]
Nov 01 03:23:11.367953 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:11.367935    4930 cpu_manager.go:414] "containerMap: removing container" podUID="6da1345e-9204-4023-b610-f31e4c3c8250" containerName="revoke-keys"
Nov 01 03:23:11.367993 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:11.367954    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6da1345e-9204-4023-b610-f31e4c3c8250" containerName="revoke-keys"
Nov 01 03:23:11.368064 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:11.368055    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="6da1345e-9204-4023-b610-f31e4c3c8250" containerName="revoke-keys"
Nov 01 03:23:11.371527 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:11.371502    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034045af-xq7ns"
Nov 01 03:23:11.424907 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:11.424878    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:23:11.427513 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:11.427491    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:23:11.474664 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:11.474641    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:23:11.474797 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:11.474693    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:23:11.481259 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:11.481240    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-034045af-xq7ns"]
Nov 01 03:23:11.522519 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:11.522497    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8805a580-8cc7-483b-b266-d6db7193c90f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-034045af-xq7ns\" (UID: \"8805a580-8cc7-483b-b266-d6db7193c90f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034045af-xq7ns"
Nov 01 03:23:11.522655 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:11.522551    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lvtg\" (UniqueName: \"kubernetes.io/projected/8805a580-8cc7-483b-b266-d6db7193c90f-kube-api-access-5lvtg\") pod \"maas-api-revoke-sub-034045af-xq7ns\" (UID: \"8805a580-8cc7-483b-b266-d6db7193c90f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034045af-xq7ns"
Nov 01 03:23:11.522715 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:11.522696    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8805a580-8cc7-483b-b266-d6db7193c90f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-034045af-xq7ns\" (UID: \"8805a580-8cc7-483b-b266-d6db7193c90f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034045af-xq7ns"
Nov 01 03:23:11.624308 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:11.624224    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8805a580-8cc7-483b-b266-d6db7193c90f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-034045af-xq7ns\" (UID: \"8805a580-8cc7-483b-b266-d6db7193c90f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034045af-xq7ns"
Nov 01 03:23:11.624461 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:11.624395    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8805a580-8cc7-483b-b266-d6db7193c90f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-034045af-xq7ns\" (UID: \"8805a580-8cc7-483b-b266-d6db7193c90f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034045af-xq7ns"
Nov 01 03:23:11.624543 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:11.624478    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lvtg\" (UniqueName: \"kubernetes.io/projected/8805a580-8cc7-483b-b266-d6db7193c90f-kube-api-access-5lvtg\") pod \"maas-api-revoke-sub-034045af-xq7ns\" (UID: \"8805a580-8cc7-483b-b266-d6db7193c90f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034045af-xq7ns"
Nov 01 03:23:11.625061 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:11.625030    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8805a580-8cc7-483b-b266-d6db7193c90f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-034045af-xq7ns\" (UID: \"8805a580-8cc7-483b-b266-d6db7193c90f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034045af-xq7ns"
Nov 01 03:23:11.626907 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:11.626885    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8805a580-8cc7-483b-b266-d6db7193c90f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-034045af-xq7ns\" (UID: \"8805a580-8cc7-483b-b266-d6db7193c90f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034045af-xq7ns"
Nov 01 03:23:11.686924 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:11.686886    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lvtg\" (UniqueName: \"kubernetes.io/projected/8805a580-8cc7-483b-b266-d6db7193c90f-kube-api-access-5lvtg\") pod \"maas-api-revoke-sub-034045af-xq7ns\" (UID: \"8805a580-8cc7-483b-b266-d6db7193c90f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-034045af-xq7ns"
Nov 01 03:23:11.986551 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:11.986476    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034045af-xq7ns"
Nov 01 03:23:12.360772 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:12.360741    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-034045af-xq7ns"]
Nov 01 03:23:12.362079 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:23:12.362032    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8805a580_8cc7_483b_b266_d6db7193c90f.slice/crio-c70fe2cc01553dbebfa7f6c11fdeaeb0b61c59c6c2386353baf902d455c91ec3 WatchSource:0}: Error finding container c70fe2cc01553dbebfa7f6c11fdeaeb0b61c59c6c2386353baf902d455c91ec3: Status 404 returned error can't find the container with id c70fe2cc01553dbebfa7f6c11fdeaeb0b61c59c6c2386353baf902d455c91ec3
Nov 01 03:23:12.453234 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:12.453202    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:23:12.944607 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:12.944509    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034045af-xq7ns" event={"ID":"8805a580-8cc7-483b-b266-d6db7193c90f","Type":"ContainerStarted","Data":"72bfcfe9560db808d211c0fcfc775b7cc25ed1d3cd2777d909c4668294e1561a"}
Nov 01 03:23:12.944607 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:12.944600    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034045af-xq7ns" event={"ID":"8805a580-8cc7-483b-b266-d6db7193c90f","Type":"ContainerStarted","Data":"c70fe2cc01553dbebfa7f6c11fdeaeb0b61c59c6c2386353baf902d455c91ec3"}
Nov 01 03:23:13.062659 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:13.062606    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034045af-xq7ns" podStartSLOduration=2.062585881 podStartE2EDuration="2.062585881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:23:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:23:13.059840698 +0000 UTC m=+997.582847288" watchObservedRunningTime="2026-11-01 03:23:13.062585881 +0000 UTC m=+997.585592466"
Nov 01 03:23:13.295513 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:13.295481    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1a9ce86-9jznn"]
Nov 01 03:23:13.299442 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:13.299421    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a9ce86-9jznn"
Nov 01 03:23:13.365677 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:13.365646    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1a9ce86-9jznn"]
Nov 01 03:23:13.448422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:13.448394    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p74vb\" (UniqueName: \"kubernetes.io/projected/eefc364f-9d31-456c-b95c-0fcebe66bfc5-kube-api-access-p74vb\") pod \"maas-api-revoke-sub-b1a9ce86-9jznn\" (UID: \"eefc364f-9d31-456c-b95c-0fcebe66bfc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a9ce86-9jznn"
Nov 01 03:23:13.448700 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:13.448678    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eefc364f-9d31-456c-b95c-0fcebe66bfc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1a9ce86-9jznn\" (UID: \"eefc364f-9d31-456c-b95c-0fcebe66bfc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a9ce86-9jznn"
Nov 01 03:23:13.448791 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:13.448722    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eefc364f-9d31-456c-b95c-0fcebe66bfc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1a9ce86-9jznn\" (UID: \"eefc364f-9d31-456c-b95c-0fcebe66bfc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a9ce86-9jznn"
Nov 01 03:23:13.550487 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:13.550403    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eefc364f-9d31-456c-b95c-0fcebe66bfc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1a9ce86-9jznn\" (UID: \"eefc364f-9d31-456c-b95c-0fcebe66bfc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a9ce86-9jznn"
Nov 01 03:23:13.550487 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:13.550447    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eefc364f-9d31-456c-b95c-0fcebe66bfc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1a9ce86-9jznn\" (UID: \"eefc364f-9d31-456c-b95c-0fcebe66bfc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a9ce86-9jznn"
Nov 01 03:23:13.550487 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:13.550482    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p74vb\" (UniqueName: \"kubernetes.io/projected/eefc364f-9d31-456c-b95c-0fcebe66bfc5-kube-api-access-p74vb\") pod \"maas-api-revoke-sub-b1a9ce86-9jznn\" (UID: \"eefc364f-9d31-456c-b95c-0fcebe66bfc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a9ce86-9jznn"
Nov 01 03:23:13.551106 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:13.551085    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eefc364f-9d31-456c-b95c-0fcebe66bfc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1a9ce86-9jznn\" (UID: \"eefc364f-9d31-456c-b95c-0fcebe66bfc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a9ce86-9jznn"
Nov 01 03:23:13.552728 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:13.552698    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eefc364f-9d31-456c-b95c-0fcebe66bfc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1a9ce86-9jznn\" (UID: \"eefc364f-9d31-456c-b95c-0fcebe66bfc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a9ce86-9jznn"
Nov 01 03:23:13.612280 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:13.612245    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p74vb\" (UniqueName: \"kubernetes.io/projected/eefc364f-9d31-456c-b95c-0fcebe66bfc5-kube-api-access-p74vb\") pod \"maas-api-revoke-sub-b1a9ce86-9jznn\" (UID: \"eefc364f-9d31-456c-b95c-0fcebe66bfc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a9ce86-9jznn"
Nov 01 03:23:13.614727 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:13.614703    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a9ce86-9jznn"
Nov 01 03:23:13.953914 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:13.953879    4930 generic.go:356] "Generic (PLEG): container finished" podID="8805a580-8cc7-483b-b266-d6db7193c90f" containerID="72bfcfe9560db808d211c0fcfc775b7cc25ed1d3cd2777d909c4668294e1561a" exitCode=0
Nov 01 03:23:13.954077 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:13.953967    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034045af-xq7ns" event={"ID":"8805a580-8cc7-483b-b266-d6db7193c90f","Type":"ContainerDied","Data":"72bfcfe9560db808d211c0fcfc775b7cc25ed1d3cd2777d909c4668294e1561a"}
Nov 01 03:23:13.990127 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:23:13.990094    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeefc364f_9d31_456c_b95c_0fcebe66bfc5.slice/crio-5b5a53558f2e1692d5986ccdca58ebb2fc1d759701dbd09d54c6b9664e4da818 WatchSource:0}: Error finding container 5b5a53558f2e1692d5986ccdca58ebb2fc1d759701dbd09d54c6b9664e4da818: Status 404 returned error can't find the container with id 5b5a53558f2e1692d5986ccdca58ebb2fc1d759701dbd09d54c6b9664e4da818
Nov 01 03:23:13.990702 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:13.990677    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1a9ce86-9jznn"]
Nov 01 03:23:14.963094 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:14.963066    4930 generic.go:356] "Generic (PLEG): container finished" podID="eefc364f-9d31-456c-b95c-0fcebe66bfc5" containerID="ff4960e4b031b593cf7d2916a85c0b1486b8f95962d8756fb58568fe65aa89b6" exitCode=0
Nov 01 03:23:14.963398 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:14.963145    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a9ce86-9jznn" event={"ID":"eefc364f-9d31-456c-b95c-0fcebe66bfc5","Type":"ContainerDied","Data":"ff4960e4b031b593cf7d2916a85c0b1486b8f95962d8756fb58568fe65aa89b6"}
Nov 01 03:23:14.963398 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:14.963234    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a9ce86-9jznn" event={"ID":"eefc364f-9d31-456c-b95c-0fcebe66bfc5","Type":"ContainerStarted","Data":"5b5a53558f2e1692d5986ccdca58ebb2fc1d759701dbd09d54c6b9664e4da818"}
Nov 01 03:23:15.139703 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:15.139679    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034045af-xq7ns"
Nov 01 03:23:15.271551 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:15.271472    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8805a580-8cc7-483b-b266-d6db7193c90f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8805a580-8cc7-483b-b266-d6db7193c90f-maas-api-openshift-ca\") pod \"8805a580-8cc7-483b-b266-d6db7193c90f\" (UID: \"8805a580-8cc7-483b-b266-d6db7193c90f\") "
Nov 01 03:23:15.271770 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:15.271615    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8805a580-8cc7-483b-b266-d6db7193c90f-kube-api-access-5lvtg\" (UniqueName: \"kubernetes.io/projected/8805a580-8cc7-483b-b266-d6db7193c90f-kube-api-access-5lvtg\") pod \"8805a580-8cc7-483b-b266-d6db7193c90f\" (UID: \"8805a580-8cc7-483b-b266-d6db7193c90f\") "
Nov 01 03:23:15.271770 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:15.271723    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8805a580-8cc7-483b-b266-d6db7193c90f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8805a580-8cc7-483b-b266-d6db7193c90f-maas-api-xks-ca\") pod \"8805a580-8cc7-483b-b266-d6db7193c90f\" (UID: \"8805a580-8cc7-483b-b266-d6db7193c90f\") "
Nov 01 03:23:15.271885 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:15.271867    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8805a580-8cc7-483b-b266-d6db7193c90f-maas-api-openshift-ca" pod "8805a580-8cc7-483b-b266-d6db7193c90f" (UID: "8805a580-8cc7-483b-b266-d6db7193c90f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:23:15.272137 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:15.272117    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8805a580-8cc7-483b-b266-d6db7193c90f-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:23:15.274652 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:15.274621    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8805a580-8cc7-483b-b266-d6db7193c90f-maas-api-xks-ca" pod "8805a580-8cc7-483b-b266-d6db7193c90f" (UID: "8805a580-8cc7-483b-b266-d6db7193c90f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:23:15.274803 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:15.274640    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8805a580-8cc7-483b-b266-d6db7193c90f-kube-api-access-5lvtg" pod "8805a580-8cc7-483b-b266-d6db7193c90f" (UID: "8805a580-8cc7-483b-b266-d6db7193c90f"). InnerVolumeSpecName "kube-api-access-5lvtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:23:15.373422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:15.373392    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8805a580-8cc7-483b-b266-d6db7193c90f-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:23:15.373422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:15.373420    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lvtg\" (UniqueName: \"kubernetes.io/projected/8805a580-8cc7-483b-b266-d6db7193c90f-kube-api-access-5lvtg\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:23:15.973967 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:15.973883    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034045af-xq7ns"
Nov 01 03:23:15.973967 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:15.973919    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-034045af-xq7ns" event={"ID":"8805a580-8cc7-483b-b266-d6db7193c90f","Type":"ContainerDied","Data":"c70fe2cc01553dbebfa7f6c11fdeaeb0b61c59c6c2386353baf902d455c91ec3"}
Nov 01 03:23:15.974437 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:15.973982    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c70fe2cc01553dbebfa7f6c11fdeaeb0b61c59c6c2386353baf902d455c91ec3"
Nov 01 03:23:16.236377 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:16.236321    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a9ce86-9jznn"
Nov 01 03:23:16.295297 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:16.295265    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eefc364f-9d31-456c-b95c-0fcebe66bfc5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eefc364f-9d31-456c-b95c-0fcebe66bfc5-maas-api-xks-ca\") pod \"eefc364f-9d31-456c-b95c-0fcebe66bfc5\" (UID: \"eefc364f-9d31-456c-b95c-0fcebe66bfc5\") "
Nov 01 03:23:16.295498 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:16.295312    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eefc364f-9d31-456c-b95c-0fcebe66bfc5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eefc364f-9d31-456c-b95c-0fcebe66bfc5-maas-api-openshift-ca\") pod \"eefc364f-9d31-456c-b95c-0fcebe66bfc5\" (UID: \"eefc364f-9d31-456c-b95c-0fcebe66bfc5\") "
Nov 01 03:23:16.295740 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:16.295717    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eefc364f-9d31-456c-b95c-0fcebe66bfc5-maas-api-openshift-ca" pod "eefc364f-9d31-456c-b95c-0fcebe66bfc5" (UID: "eefc364f-9d31-456c-b95c-0fcebe66bfc5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:23:16.295851 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:16.295832    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eefc364f-9d31-456c-b95c-0fcebe66bfc5-kube-api-access-p74vb\" (UniqueName: \"kubernetes.io/projected/eefc364f-9d31-456c-b95c-0fcebe66bfc5-kube-api-access-p74vb\") pod \"eefc364f-9d31-456c-b95c-0fcebe66bfc5\" (UID: \"eefc364f-9d31-456c-b95c-0fcebe66bfc5\") "
Nov 01 03:23:16.296200 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:16.296181    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eefc364f-9d31-456c-b95c-0fcebe66bfc5-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:23:16.298520 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:16.298491    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eefc364f-9d31-456c-b95c-0fcebe66bfc5-kube-api-access-p74vb" pod "eefc364f-9d31-456c-b95c-0fcebe66bfc5" (UID: "eefc364f-9d31-456c-b95c-0fcebe66bfc5"). InnerVolumeSpecName "kube-api-access-p74vb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:23:16.299570 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:16.299534    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eefc364f-9d31-456c-b95c-0fcebe66bfc5-maas-api-xks-ca" pod "eefc364f-9d31-456c-b95c-0fcebe66bfc5" (UID: "eefc364f-9d31-456c-b95c-0fcebe66bfc5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:23:16.419963 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:16.419913    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p74vb\" (UniqueName: \"kubernetes.io/projected/eefc364f-9d31-456c-b95c-0fcebe66bfc5-kube-api-access-p74vb\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:23:16.419963 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:16.419955    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eefc364f-9d31-456c-b95c-0fcebe66bfc5-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:23:17.022107 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:17.022070    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a9ce86-9jznn" event={"ID":"eefc364f-9d31-456c-b95c-0fcebe66bfc5","Type":"ContainerDied","Data":"5b5a53558f2e1692d5986ccdca58ebb2fc1d759701dbd09d54c6b9664e4da818"}
Nov 01 03:23:17.022878 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:17.022859    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b5a53558f2e1692d5986ccdca58ebb2fc1d759701dbd09d54c6b9664e4da818"
Nov 01 03:23:17.023005 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:17.022808    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1a9ce86-9jznn"
Nov 01 03:23:17.380193 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:17.380161    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:23:18.763959 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:18.763922    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:23:19.123155 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:23:19.123118    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:23:22.137542 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:22.137505    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:23:26.987217 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:26.987177    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:23:28.588150 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:28.588111    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f2040c8-csxkb"]
Nov 01 03:23:28.588926 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:28.588904    4930 cpu_manager.go:414] "containerMap: removing container" podUID="8805a580-8cc7-483b-b266-d6db7193c90f" containerName="revoke-keys"
Nov 01 03:23:28.588993 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:28.588928    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8805a580-8cc7-483b-b266-d6db7193c90f" containerName="revoke-keys"
Nov 01 03:23:28.588993 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:28.588943    4930 cpu_manager.go:414] "containerMap: removing container" podUID="eefc364f-9d31-456c-b95c-0fcebe66bfc5" containerName="revoke-keys"
Nov 01 03:23:28.588993 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:28.588951    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eefc364f-9d31-456c-b95c-0fcebe66bfc5" containerName="revoke-keys"
Nov 01 03:23:28.589124 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:28.589113    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="8805a580-8cc7-483b-b266-d6db7193c90f" containerName="revoke-keys"
Nov 01 03:23:28.589159 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:28.589126    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="eefc364f-9d31-456c-b95c-0fcebe66bfc5" containerName="revoke-keys"
Nov 01 03:23:28.593233 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:28.593215    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2040c8-csxkb"
Nov 01 03:23:28.645451 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:28.645384    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:23:28.649989 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:28.649962    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f2040c8-csxkb"]
Nov 01 03:23:28.685224 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:28.685202    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfdfd\" (UniqueName: \"kubernetes.io/projected/5370e16f-3d6e-4e90-9f29-dcafa5110b74-kube-api-access-bfdfd\") pod \"maas-api-revoke-sub-6f2040c8-csxkb\" (UID: \"5370e16f-3d6e-4e90-9f29-dcafa5110b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2040c8-csxkb"
Nov 01 03:23:28.685338 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:28.685258    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5370e16f-3d6e-4e90-9f29-dcafa5110b74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f2040c8-csxkb\" (UID: \"5370e16f-3d6e-4e90-9f29-dcafa5110b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2040c8-csxkb"
Nov 01 03:23:28.685338 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:28.685280    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5370e16f-3d6e-4e90-9f29-dcafa5110b74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f2040c8-csxkb\" (UID: \"5370e16f-3d6e-4e90-9f29-dcafa5110b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2040c8-csxkb"
Nov 01 03:23:28.694126 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:28.694105    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:23:28.694230 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:28.694136    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:23:28.694843 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:28.694828    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:23:28.786912 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:28.786888    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfdfd\" (UniqueName: \"kubernetes.io/projected/5370e16f-3d6e-4e90-9f29-dcafa5110b74-kube-api-access-bfdfd\") pod \"maas-api-revoke-sub-6f2040c8-csxkb\" (UID: \"5370e16f-3d6e-4e90-9f29-dcafa5110b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2040c8-csxkb"
Nov 01 03:23:28.787019 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:28.786943    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5370e16f-3d6e-4e90-9f29-dcafa5110b74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f2040c8-csxkb\" (UID: \"5370e16f-3d6e-4e90-9f29-dcafa5110b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2040c8-csxkb"
Nov 01 03:23:28.787019 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:28.786972    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5370e16f-3d6e-4e90-9f29-dcafa5110b74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f2040c8-csxkb\" (UID: \"5370e16f-3d6e-4e90-9f29-dcafa5110b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2040c8-csxkb"
Nov 01 03:23:28.787770 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:28.787751    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5370e16f-3d6e-4e90-9f29-dcafa5110b74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f2040c8-csxkb\" (UID: \"5370e16f-3d6e-4e90-9f29-dcafa5110b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2040c8-csxkb"
Nov 01 03:23:28.789377 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:28.789343    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5370e16f-3d6e-4e90-9f29-dcafa5110b74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f2040c8-csxkb\" (UID: \"5370e16f-3d6e-4e90-9f29-dcafa5110b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2040c8-csxkb"
Nov 01 03:23:28.847780 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:28.847758    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfdfd\" (UniqueName: \"kubernetes.io/projected/5370e16f-3d6e-4e90-9f29-dcafa5110b74-kube-api-access-bfdfd\") pod \"maas-api-revoke-sub-6f2040c8-csxkb\" (UID: \"5370e16f-3d6e-4e90-9f29-dcafa5110b74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2040c8-csxkb"
Nov 01 03:23:28.908739 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:28.908693    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2040c8-csxkb"
Nov 01 03:23:29.297789 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:23:29.297755    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5370e16f_3d6e_4e90_9f29_dcafa5110b74.slice/crio-1b288d2638d41e439d9d8bb02d37d892cb56e464b5a1e82701c1f45ff1350853 WatchSource:0}: Error finding container 1b288d2638d41e439d9d8bb02d37d892cb56e464b5a1e82701c1f45ff1350853: Status 404 returned error can't find the container with id 1b288d2638d41e439d9d8bb02d37d892cb56e464b5a1e82701c1f45ff1350853
Nov 01 03:23:29.297937 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:29.297911    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f2040c8-csxkb"]
Nov 01 03:23:30.114324 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:30.114283    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2040c8-csxkb" event={"ID":"5370e16f-3d6e-4e90-9f29-dcafa5110b74","Type":"ContainerStarted","Data":"7525db34171186f94600ea1777d93416508732f6d925370498af6fa7f760af8d"}
Nov 01 03:23:30.114835 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:30.114806    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2040c8-csxkb" event={"ID":"5370e16f-3d6e-4e90-9f29-dcafa5110b74","Type":"ContainerStarted","Data":"1b288d2638d41e439d9d8bb02d37d892cb56e464b5a1e82701c1f45ff1350853"}
Nov 01 03:23:30.230387 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:30.230330    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2040c8-csxkb" podStartSLOduration=2.230312599 podStartE2EDuration="2.230312599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:23:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:23:30.229786335 +0000 UTC m=+1014.752792950" watchObservedRunningTime="2026-11-01 03:23:30.230312599 +0000 UTC m=+1014.753319188"
Nov 01 03:23:31.122678 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:31.122644    4930 generic.go:356] "Generic (PLEG): container finished" podID="5370e16f-3d6e-4e90-9f29-dcafa5110b74" containerID="7525db34171186f94600ea1777d93416508732f6d925370498af6fa7f760af8d" exitCode=0
Nov 01 03:23:31.123094 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:31.122725    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2040c8-csxkb" event={"ID":"5370e16f-3d6e-4e90-9f29-dcafa5110b74","Type":"ContainerDied","Data":"7525db34171186f94600ea1777d93416508732f6d925370498af6fa7f760af8d"}
Nov 01 03:23:31.242463 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:31.242424    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:23:31.856683 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:31.856650    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:23:32.284803 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:32.284771    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2040c8-csxkb"
Nov 01 03:23:32.428631 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:32.428530    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5370e16f-3d6e-4e90-9f29-dcafa5110b74-kube-api-access-bfdfd\" (UniqueName: \"kubernetes.io/projected/5370e16f-3d6e-4e90-9f29-dcafa5110b74-kube-api-access-bfdfd\") pod \"5370e16f-3d6e-4e90-9f29-dcafa5110b74\" (UID: \"5370e16f-3d6e-4e90-9f29-dcafa5110b74\") "
Nov 01 03:23:32.428631 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:32.428585    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5370e16f-3d6e-4e90-9f29-dcafa5110b74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5370e16f-3d6e-4e90-9f29-dcafa5110b74-maas-api-openshift-ca\") pod \"5370e16f-3d6e-4e90-9f29-dcafa5110b74\" (UID: \"5370e16f-3d6e-4e90-9f29-dcafa5110b74\") "
Nov 01 03:23:32.428820 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:32.428706    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5370e16f-3d6e-4e90-9f29-dcafa5110b74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5370e16f-3d6e-4e90-9f29-dcafa5110b74-maas-api-xks-ca\") pod \"5370e16f-3d6e-4e90-9f29-dcafa5110b74\" (UID: \"5370e16f-3d6e-4e90-9f29-dcafa5110b74\") "
Nov 01 03:23:32.429023 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:32.428991    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5370e16f-3d6e-4e90-9f29-dcafa5110b74-maas-api-openshift-ca" pod "5370e16f-3d6e-4e90-9f29-dcafa5110b74" (UID: "5370e16f-3d6e-4e90-9f29-dcafa5110b74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:23:32.429258 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:32.429236    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5370e16f-3d6e-4e90-9f29-dcafa5110b74-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:23:32.430768 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:32.430737    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5370e16f-3d6e-4e90-9f29-dcafa5110b74-maas-api-xks-ca" pod "5370e16f-3d6e-4e90-9f29-dcafa5110b74" (UID: "5370e16f-3d6e-4e90-9f29-dcafa5110b74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:23:32.430841 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:32.430738    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5370e16f-3d6e-4e90-9f29-dcafa5110b74-kube-api-access-bfdfd" pod "5370e16f-3d6e-4e90-9f29-dcafa5110b74" (UID: "5370e16f-3d6e-4e90-9f29-dcafa5110b74"). InnerVolumeSpecName "kube-api-access-bfdfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:23:32.530316 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:32.530285    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5370e16f-3d6e-4e90-9f29-dcafa5110b74-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:23:32.530316 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:32.530314    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfdfd\" (UniqueName: \"kubernetes.io/projected/5370e16f-3d6e-4e90-9f29-dcafa5110b74-kube-api-access-bfdfd\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:23:33.137828 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:33.137792    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2040c8-csxkb"
Nov 01 03:23:33.137828 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:33.137804    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2040c8-csxkb" event={"ID":"5370e16f-3d6e-4e90-9f29-dcafa5110b74","Type":"ContainerDied","Data":"1b288d2638d41e439d9d8bb02d37d892cb56e464b5a1e82701c1f45ff1350853"}
Nov 01 03:23:33.138128 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:33.137860    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b288d2638d41e439d9d8bb02d37d892cb56e464b5a1e82701c1f45ff1350853"
Nov 01 03:23:34.988580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:34.987871    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"]
Nov 01 03:23:34.989098 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:34.988631    4930 cpu_manager.go:414] "containerMap: removing container" podUID="5370e16f-3d6e-4e90-9f29-dcafa5110b74" containerName="revoke-keys"
Nov 01 03:23:34.989098 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:34.988651    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5370e16f-3d6e-4e90-9f29-dcafa5110b74" containerName="revoke-keys"
Nov 01 03:23:34.989098 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:34.988829    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="5370e16f-3d6e-4e90-9f29-dcafa5110b74" containerName="revoke-keys"
Nov 01 03:23:34.995569 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:34.995236    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:23:35.061269 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:35.059166    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75d142b3-7172-4b57-bc07-a2f404c4656e-tls-certs\") pod \"distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm\" (UID: \"75d142b3-7172-4b57-bc07-a2f404c4656e\") " pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:23:35.061269 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:35.059295    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75d142b3-7172-4b57-bc07-a2f404c4656e-dshm\") pod \"distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm\" (UID: \"75d142b3-7172-4b57-bc07-a2f404c4656e\") " pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:23:35.061269 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:35.059373    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75d142b3-7172-4b57-bc07-a2f404c4656e-model-cache\") pod \"distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm\" (UID: \"75d142b3-7172-4b57-bc07-a2f404c4656e\") " pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:23:35.061269 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:35.059405    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75d142b3-7172-4b57-bc07-a2f404c4656e-tmp-dir\") pod \"distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm\" (UID: \"75d142b3-7172-4b57-bc07-a2f404c4656e\") " pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:23:35.061269 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:35.059448    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctrm7\" (UniqueName: \"kubernetes.io/projected/75d142b3-7172-4b57-bc07-a2f404c4656e-kube-api-access-ctrm7\") pod \"distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm\" (UID: \"75d142b3-7172-4b57-bc07-a2f404c4656e\") " pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:23:35.061269 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:35.059510    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75d142b3-7172-4b57-bc07-a2f404c4656e-home\") pod \"distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm\" (UID: \"75d142b3-7172-4b57-bc07-a2f404c4656e\") " pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:23:35.098786 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:35.098719    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:23:35.102297 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:35.102268    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-c8cb9a\"/\"distinct-w2-c8cb9a-kserve-self-signed-certs\""
Nov 01 03:23:35.106777 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:35.106752    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"]
Nov 01 03:23:35.192549 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:35.192496    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75d142b3-7172-4b57-bc07-a2f404c4656e-model-cache\") pod \"distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm\" (UID: \"75d142b3-7172-4b57-bc07-a2f404c4656e\") " pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:23:35.192549 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:35.192572    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75d142b3-7172-4b57-bc07-a2f404c4656e-tmp-dir\") pod \"distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm\" (UID: \"75d142b3-7172-4b57-bc07-a2f404c4656e\") " pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:23:35.192846 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:35.192617    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctrm7\" (UniqueName: \"kubernetes.io/projected/75d142b3-7172-4b57-bc07-a2f404c4656e-kube-api-access-ctrm7\") pod \"distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm\" (UID: \"75d142b3-7172-4b57-bc07-a2f404c4656e\") " pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:23:35.192846 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:35.192679    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75d142b3-7172-4b57-bc07-a2f404c4656e-home\") pod \"distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm\" (UID: \"75d142b3-7172-4b57-bc07-a2f404c4656e\") " pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:23:35.192846 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:35.192803    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75d142b3-7172-4b57-bc07-a2f404c4656e-tls-certs\") pod \"distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm\" (UID: \"75d142b3-7172-4b57-bc07-a2f404c4656e\") " pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:23:35.193626 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:35.192891    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75d142b3-7172-4b57-bc07-a2f404c4656e-dshm\") pod \"distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm\" (UID: \"75d142b3-7172-4b57-bc07-a2f404c4656e\") " pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:23:35.206505 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:35.206468    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75d142b3-7172-4b57-bc07-a2f404c4656e-model-cache\") pod \"distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm\" (UID: \"75d142b3-7172-4b57-bc07-a2f404c4656e\") " pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:23:35.206704 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:35.206682    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75d142b3-7172-4b57-bc07-a2f404c4656e-home\") pod \"distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm\" (UID: \"75d142b3-7172-4b57-bc07-a2f404c4656e\") " pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:23:35.206855 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:35.206829    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75d142b3-7172-4b57-bc07-a2f404c4656e-tmp-dir\") pod \"distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm\" (UID: \"75d142b3-7172-4b57-bc07-a2f404c4656e\") " pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:23:35.261009 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:35.260924    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75d142b3-7172-4b57-bc07-a2f404c4656e-dshm\") pod \"distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm\" (UID: \"75d142b3-7172-4b57-bc07-a2f404c4656e\") " pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:23:35.266412 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:35.266352    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75d142b3-7172-4b57-bc07-a2f404c4656e-tls-certs\") pod \"distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm\" (UID: \"75d142b3-7172-4b57-bc07-a2f404c4656e\") " pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:23:35.274628 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:35.274569    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctrm7\" (UniqueName: \"kubernetes.io/projected/75d142b3-7172-4b57-bc07-a2f404c4656e-kube-api-access-ctrm7\") pod \"distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm\" (UID: \"75d142b3-7172-4b57-bc07-a2f404c4656e\") " pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:23:35.311955 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:35.311920    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:23:35.726366 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:23:35.726319    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75d142b3_7172_4b57_bc07_a2f404c4656e.slice/crio-7c26a1d58e68e1187ca042c85fb853f07b13328c7cc01cbcca3188291f6b471b WatchSource:0}: Error finding container 7c26a1d58e68e1187ca042c85fb853f07b13328c7cc01cbcca3188291f6b471b: Status 404 returned error can't find the container with id 7c26a1d58e68e1187ca042c85fb853f07b13328c7cc01cbcca3188291f6b471b
Nov 01 03:23:35.782452 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:35.782418    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"]
Nov 01 03:23:36.216065 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:36.216029    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm" event={"ID":"75d142b3-7172-4b57-bc07-a2f404c4656e","Type":"ContainerStarted","Data":"4f89b4379fe0a3729d8ba889940d964e9964876362cdfa6d6a25741d85491dd3"}
Nov 01 03:23:36.216462 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:36.216080    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm" event={"ID":"75d142b3-7172-4b57-bc07-a2f404c4656e","Type":"ContainerStarted","Data":"7c26a1d58e68e1187ca042c85fb853f07b13328c7cc01cbcca3188291f6b471b"}
Nov 01 03:23:36.313052 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:36.313026    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:23:36.337173 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:36.337121    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm" podStartSLOduration=2.337109451 podStartE2EDuration="2.337109451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:23:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:23:36.333401836 +0000 UTC m=+1020.856408432" watchObservedRunningTime="2026-11-01 03:23:36.337109451 +0000 UTC m=+1020.860116050"
Nov 01 03:23:36.944701 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:36.944660    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:23:41.392075 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:41.392037    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:23:41.753271 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:41.753185    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:23:45.312589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:45.312543    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:23:45.325474 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:45.325450    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:23:46.308361 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:46.308331    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:23:46.533814 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:46.533780    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:23:49.123656 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:23:49.123535    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:23:51.436113 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:51.436070    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:23:56.237530 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:23:56.237486    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:24:01.076704 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:01.076660    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:24:02.133072 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:02.133032    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"]
Nov 01 03:24:02.137416 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:02.137393    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:24:02.231421 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:02.231367    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf6d6ac-7399-4c2b-bf41-5423abbedef2-tls-certs\") pod \"distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5\" (UID: \"baf6d6ac-7399-4c2b-bf41-5423abbedef2\") " pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:24:02.231605 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:02.231581    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baf6d6ac-7399-4c2b-bf41-5423abbedef2-home\") pod \"distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5\" (UID: \"baf6d6ac-7399-4c2b-bf41-5423abbedef2\") " pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:24:02.231666 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:02.231644    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf6d6ac-7399-4c2b-bf41-5423abbedef2-dshm\") pod \"distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5\" (UID: \"baf6d6ac-7399-4c2b-bf41-5423abbedef2\") " pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:24:02.231780 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:02.231761    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf6d6ac-7399-4c2b-bf41-5423abbedef2-tmp-dir\") pod \"distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5\" (UID: \"baf6d6ac-7399-4c2b-bf41-5423abbedef2\") " pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:24:02.231846 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:02.231807    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf6d6ac-7399-4c2b-bf41-5423abbedef2-model-cache\") pod \"distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5\" (UID: \"baf6d6ac-7399-4c2b-bf41-5423abbedef2\") " pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:24:02.231950 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:02.231933    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljxsz\" (UniqueName: \"kubernetes.io/projected/baf6d6ac-7399-4c2b-bf41-5423abbedef2-kube-api-access-ljxsz\") pod \"distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5\" (UID: \"baf6d6ac-7399-4c2b-bf41-5423abbedef2\") " pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:24:02.241837 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:02.241813    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-19faa5\"/\"distinct-w6-19faa5-kserve-self-signed-certs\""
Nov 01 03:24:02.246415 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:02.246387    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"]
Nov 01 03:24:02.246618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:02.246586    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:24:02.333064 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:02.333019    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baf6d6ac-7399-4c2b-bf41-5423abbedef2-home\") pod \"distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5\" (UID: \"baf6d6ac-7399-4c2b-bf41-5423abbedef2\") " pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:24:02.333257 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:02.333098    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf6d6ac-7399-4c2b-bf41-5423abbedef2-dshm\") pod \"distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5\" (UID: \"baf6d6ac-7399-4c2b-bf41-5423abbedef2\") " pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:24:02.333257 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:02.333133    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf6d6ac-7399-4c2b-bf41-5423abbedef2-tmp-dir\") pod \"distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5\" (UID: \"baf6d6ac-7399-4c2b-bf41-5423abbedef2\") " pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:24:02.333257 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:02.333157    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf6d6ac-7399-4c2b-bf41-5423abbedef2-model-cache\") pod \"distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5\" (UID: \"baf6d6ac-7399-4c2b-bf41-5423abbedef2\") " pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:24:02.333257 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:02.333216    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljxsz\" (UniqueName: \"kubernetes.io/projected/baf6d6ac-7399-4c2b-bf41-5423abbedef2-kube-api-access-ljxsz\") pod \"distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5\" (UID: \"baf6d6ac-7399-4c2b-bf41-5423abbedef2\") " pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:24:02.333434 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:02.333289    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf6d6ac-7399-4c2b-bf41-5423abbedef2-tls-certs\") pod \"distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5\" (UID: \"baf6d6ac-7399-4c2b-bf41-5423abbedef2\") " pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:24:02.333434 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:02.333405    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baf6d6ac-7399-4c2b-bf41-5423abbedef2-home\") pod \"distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5\" (UID: \"baf6d6ac-7399-4c2b-bf41-5423abbedef2\") " pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:24:02.333541 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:02.333511    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf6d6ac-7399-4c2b-bf41-5423abbedef2-tmp-dir\") pod \"distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5\" (UID: \"baf6d6ac-7399-4c2b-bf41-5423abbedef2\") " pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:24:02.333745 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:02.333726    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf6d6ac-7399-4c2b-bf41-5423abbedef2-model-cache\") pod \"distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5\" (UID: \"baf6d6ac-7399-4c2b-bf41-5423abbedef2\") " pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:24:02.335399 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:02.335381    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf6d6ac-7399-4c2b-bf41-5423abbedef2-dshm\") pod \"distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5\" (UID: \"baf6d6ac-7399-4c2b-bf41-5423abbedef2\") " pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:24:02.335835 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:02.335815    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf6d6ac-7399-4c2b-bf41-5423abbedef2-tls-certs\") pod \"distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5\" (UID: \"baf6d6ac-7399-4c2b-bf41-5423abbedef2\") " pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:24:02.395963 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:02.395910    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljxsz\" (UniqueName: \"kubernetes.io/projected/baf6d6ac-7399-4c2b-bf41-5423abbedef2-kube-api-access-ljxsz\") pod \"distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5\" (UID: \"baf6d6ac-7399-4c2b-bf41-5423abbedef2\") " pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:24:02.450617 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:02.450596    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:24:02.852887 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:02.852851    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"]
Nov 01 03:24:02.853204 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:24:02.853172    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbaf6d6ac_7399_4c2b_bf41_5423abbedef2.slice/crio-76b1d42a1fa93249cbd18ac86e6bee0ac381ec71301951315257dc7cb641d076 WatchSource:0}: Error finding container 76b1d42a1fa93249cbd18ac86e6bee0ac381ec71301951315257dc7cb641d076: Status 404 returned error can't find the container with id 76b1d42a1fa93249cbd18ac86e6bee0ac381ec71301951315257dc7cb641d076
Nov 01 03:24:03.420611 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:03.420547    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" event={"ID":"baf6d6ac-7399-4c2b-bf41-5423abbedef2","Type":"ContainerStarted","Data":"cc9372d0920554e876a589302dd84a247bb6e71f92b186a7071aaf60da8f1986"}
Nov 01 03:24:03.420611 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:03.420617    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" event={"ID":"baf6d6ac-7399-4c2b-bf41-5423abbedef2","Type":"ContainerStarted","Data":"76b1d42a1fa93249cbd18ac86e6bee0ac381ec71301951315257dc7cb641d076"}
Nov 01 03:24:03.450938 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:03.450913    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:24:05.932307 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:05.932271    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:24:07.291414 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:07.291377    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:24:10.841000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:10.840966    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:24:12.451295 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:12.451261    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:24:12.463867 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:12.463834    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:24:12.499486 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:12.499454    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:24:12.586764 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:12.586715    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" podStartSLOduration=10.586702117 podStartE2EDuration="10.586702117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:24:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:24:03.545244111 +0000 UTC m=+1048.068250700" watchObservedRunningTime="2026-11-01 03:24:12.586702117 +0000 UTC m=+1057.109708752"
Nov 01 03:24:12.791222 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:12.791137    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:24:15.643615 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:15.643546    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:24:19.121508 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:24:19.121470    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:24:20.563855 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:20.563813    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:24:21.828307 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:21.828261    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:24:25.452823 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:25.452786    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:24:25.678171 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:25.678138    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2"]
Nov 01 03:24:25.682071 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:25.682049    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2"
Nov 01 03:24:25.734390 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:25.734330    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-c22dd8-openshift-default-dockercfg-64gt8\""
Nov 01 03:24:25.788169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:25.788143    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2"]
Nov 01 03:24:25.819026 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:25.818961    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb6a2bc2-9530-44d6-aa35-608f182ddb04-workload-socket\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2\" (UID: \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2"
Nov 01 03:24:25.819154 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:25.819032    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb6a2bc2-9530-44d6-aa35-608f182ddb04-istio-envoy\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2\" (UID: \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2"
Nov 01 03:24:25.819154 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:25.819071    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb6a2bc2-9530-44d6-aa35-608f182ddb04-credential-socket\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2\" (UID: \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2"
Nov 01 03:24:25.819154 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:25.819089    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7mbc\" (UniqueName: \"kubernetes.io/projected/eb6a2bc2-9530-44d6-aa35-608f182ddb04-kube-api-access-w7mbc\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2\" (UID: \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2"
Nov 01 03:24:25.819154 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:25.819149    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb6a2bc2-9530-44d6-aa35-608f182ddb04-istio-token\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2\" (UID: \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2"
Nov 01 03:24:25.819287 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:25.819170    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb6a2bc2-9530-44d6-aa35-608f182ddb04-istio-podinfo\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2\" (UID: \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2"
Nov 01 03:24:25.819287 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:25.819226    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb6a2bc2-9530-44d6-aa35-608f182ddb04-istio-data\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2\" (UID: \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2"
Nov 01 03:24:25.819353 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:25.819336    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb6a2bc2-9530-44d6-aa35-608f182ddb04-workload-certs\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2\" (UID: \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2"
Nov 01 03:24:25.819386 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:25.819363    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb6a2bc2-9530-44d6-aa35-608f182ddb04-istiod-ca-cert\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2\" (UID: \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2"
Nov 01 03:24:25.920865 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:25.920837    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb6a2bc2-9530-44d6-aa35-608f182ddb04-istio-data\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2\" (UID: \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2"
Nov 01 03:24:25.921014 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:25.920942    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb6a2bc2-9530-44d6-aa35-608f182ddb04-workload-certs\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2\" (UID: \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2"
Nov 01 03:24:25.921014 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:25.920973    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb6a2bc2-9530-44d6-aa35-608f182ddb04-istiod-ca-cert\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2\" (UID: \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2"
Nov 01 03:24:25.921133 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:25.921075    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb6a2bc2-9530-44d6-aa35-608f182ddb04-workload-socket\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2\" (UID: \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2"
Nov 01 03:24:25.921133 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:25.921103    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb6a2bc2-9530-44d6-aa35-608f182ddb04-istio-envoy\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2\" (UID: \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2"
Nov 01 03:24:25.921240 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:25.921154    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb6a2bc2-9530-44d6-aa35-608f182ddb04-credential-socket\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2\" (UID: \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2"
Nov 01 03:24:25.921240 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:25.921182    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7mbc\" (UniqueName: \"kubernetes.io/projected/eb6a2bc2-9530-44d6-aa35-608f182ddb04-kube-api-access-w7mbc\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2\" (UID: \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2"
Nov 01 03:24:25.921240 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:25.921224    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb6a2bc2-9530-44d6-aa35-608f182ddb04-istio-token\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2\" (UID: \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2"
Nov 01 03:24:25.921388 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:25.921240    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb6a2bc2-9530-44d6-aa35-608f182ddb04-istio-data\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2\" (UID: \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2"
Nov 01 03:24:25.921388 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:25.921260    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb6a2bc2-9530-44d6-aa35-608f182ddb04-istio-podinfo\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2\" (UID: \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2"
Nov 01 03:24:25.921388 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:25.921291    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb6a2bc2-9530-44d6-aa35-608f182ddb04-workload-certs\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2\" (UID: \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2"
Nov 01 03:24:25.921715 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:25.921689    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb6a2bc2-9530-44d6-aa35-608f182ddb04-workload-socket\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2\" (UID: \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2"
Nov 01 03:24:25.921715 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:25.921705    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb6a2bc2-9530-44d6-aa35-608f182ddb04-credential-socket\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2\" (UID: \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2"
Nov 01 03:24:25.921874 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:25.921722    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb6a2bc2-9530-44d6-aa35-608f182ddb04-istiod-ca-cert\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2\" (UID: \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2"
Nov 01 03:24:25.923416 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:25.923398    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb6a2bc2-9530-44d6-aa35-608f182ddb04-istio-envoy\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2\" (UID: \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2"
Nov 01 03:24:25.923736 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:25.923718    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb6a2bc2-9530-44d6-aa35-608f182ddb04-istio-podinfo\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2\" (UID: \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2"
Nov 01 03:24:25.983122 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:25.983093    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb6a2bc2-9530-44d6-aa35-608f182ddb04-istio-token\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2\" (UID: \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2"
Nov 01 03:24:25.983216 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:25.983190    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7mbc\" (UniqueName: \"kubernetes.io/projected/eb6a2bc2-9530-44d6-aa35-608f182ddb04-kube-api-access-w7mbc\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2\" (UID: \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2"
Nov 01 03:24:25.995993 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:25.995941    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2"
Nov 01 03:24:26.446543 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:26.446506    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2"]
Nov 01 03:24:26.480520 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:24:26.480492    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb6a2bc2_9530_44d6_aa35_608f182ddb04.slice/crio-c8063bdd7bc8d83fcda82627d1d9282af265570b62f698367aa73352044b2747 WatchSource:0}: Error finding container c8063bdd7bc8d83fcda82627d1d9282af265570b62f698367aa73352044b2747: Status 404 returned error can't find the container with id c8063bdd7bc8d83fcda82627d1d9282af265570b62f698367aa73352044b2747
Nov 01 03:24:26.482856 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:26.482825    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:24:26.482940 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:26.482888    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:24:26.482940 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:26.482918    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:24:26.600054 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:26.600015    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2" event={"ID":"eb6a2bc2-9530-44d6-aa35-608f182ddb04","Type":"ContainerStarted","Data":"acde80c7095a104624d04842e1217d560af7e8823b60c08afdc64c9af86a7e67"}
Nov 01 03:24:26.600239 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:26.600070    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2" event={"ID":"eb6a2bc2-9530-44d6-aa35-608f182ddb04","Type":"ContainerStarted","Data":"c8063bdd7bc8d83fcda82627d1d9282af265570b62f698367aa73352044b2747"}
Nov 01 03:24:26.725960 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:26.725868    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2" podStartSLOduration=1.7258544489999998 podStartE2EDuration="1.725854449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:24:26.72012702 +0000 UTC m=+1071.243133608" watchObservedRunningTime="2026-11-01 03:24:26.725854449 +0000 UTC m=+1071.248861037"
Nov 01 03:24:26.996873 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:26.996777    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2"
Nov 01 03:24:27.997838 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:27.997799    4930 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 01 03:24:27.998223 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:27.997866    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2" podUID="eb6a2bc2-9530-44d6-aa35-608f182ddb04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 01 03:24:28.996757 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:28.996720    4930 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 01 03:24:28.996935 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:28.996786    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2" podUID="eb6a2bc2-9530-44d6-aa35-608f182ddb04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 01 03:24:29.997128 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:29.997090    4930 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 01 03:24:29.997631 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:29.997170    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2" podUID="eb6a2bc2-9530-44d6-aa35-608f182ddb04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 01 03:24:30.506590 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:30.506526    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:24:30.997227 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:30.997189    4930 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 01 03:24:30.997644 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:30.997245    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2" podUID="eb6a2bc2-9530-44d6-aa35-608f182ddb04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 01 03:24:31.628326 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:31.628273    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:24:31.996322 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:31.996236    4930 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 01 03:24:31.996322 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:31.996291    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2" podUID="eb6a2bc2-9530-44d6-aa35-608f182ddb04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 01 03:24:32.996345 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:32.996309    4930 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 01 03:24:32.996774 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:32.996366    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2" podUID="eb6a2bc2-9530-44d6-aa35-608f182ddb04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 01 03:24:34.001319 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:34.001286    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2"
Nov 01 03:24:34.001846 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:34.001800    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2"
Nov 01 03:24:34.002461 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:34.002442    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2"
Nov 01 03:24:35.241736 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:35.241700    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:24:37.530389 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:37.530350    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"]
Nov 01 03:24:37.541682 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:37.541663    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"
Nov 01 03:24:37.595308 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:37.595272    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"]
Nov 01 03:24:37.601149 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:37.601117    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ce834cc-d34d-46cc-a081-4d2cc261becb-model-cache\") pod \"distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7\" (UID: \"1ce834cc-d34d-46cc-a081-4d2cc261becb\") " pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"
Nov 01 03:24:37.601346 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:37.601326    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ce834cc-d34d-46cc-a081-4d2cc261becb-dshm\") pod \"distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7\" (UID: \"1ce834cc-d34d-46cc-a081-4d2cc261becb\") " pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"
Nov 01 03:24:37.601488 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:37.601470    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ce834cc-d34d-46cc-a081-4d2cc261becb-tmp-dir\") pod \"distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7\" (UID: \"1ce834cc-d34d-46cc-a081-4d2cc261becb\") " pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"
Nov 01 03:24:37.601699 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:37.601674    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ce834cc-d34d-46cc-a081-4d2cc261becb-tls-certs\") pod \"distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7\" (UID: \"1ce834cc-d34d-46cc-a081-4d2cc261becb\") " pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"
Nov 01 03:24:37.601782 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:37.601765    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wz65\" (UniqueName: \"kubernetes.io/projected/1ce834cc-d34d-46cc-a081-4d2cc261becb-kube-api-access-9wz65\") pod \"distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7\" (UID: \"1ce834cc-d34d-46cc-a081-4d2cc261becb\") " pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"
Nov 01 03:24:37.601929 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:37.601882    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ce834cc-d34d-46cc-a081-4d2cc261becb-home\") pod \"distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7\" (UID: \"1ce834cc-d34d-46cc-a081-4d2cc261becb\") " pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"
Nov 01 03:24:37.616314 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:37.616284    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:24:37.639024 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:37.638996    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ded2d7\"/\"distinct-2-w0-ded2d7-kserve-self-signed-certs\""
Nov 01 03:24:37.710712 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:37.710675    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wz65\" (UniqueName: \"kubernetes.io/projected/1ce834cc-d34d-46cc-a081-4d2cc261becb-kube-api-access-9wz65\") pod \"distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7\" (UID: \"1ce834cc-d34d-46cc-a081-4d2cc261becb\") " pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"
Nov 01 03:24:37.710926 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:37.710761    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ce834cc-d34d-46cc-a081-4d2cc261becb-home\") pod \"distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7\" (UID: \"1ce834cc-d34d-46cc-a081-4d2cc261becb\") " pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"
Nov 01 03:24:37.710926 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:37.710803    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ce834cc-d34d-46cc-a081-4d2cc261becb-model-cache\") pod \"distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7\" (UID: \"1ce834cc-d34d-46cc-a081-4d2cc261becb\") " pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"
Nov 01 03:24:37.710926 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:37.710827    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ce834cc-d34d-46cc-a081-4d2cc261becb-dshm\") pod \"distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7\" (UID: \"1ce834cc-d34d-46cc-a081-4d2cc261becb\") " pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"
Nov 01 03:24:37.710926 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:37.710871    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ce834cc-d34d-46cc-a081-4d2cc261becb-tmp-dir\") pod \"distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7\" (UID: \"1ce834cc-d34d-46cc-a081-4d2cc261becb\") " pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"
Nov 01 03:24:37.711161 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:37.711086    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ce834cc-d34d-46cc-a081-4d2cc261becb-tls-certs\") pod \"distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7\" (UID: \"1ce834cc-d34d-46cc-a081-4d2cc261becb\") " pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"
Nov 01 03:24:37.712057 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:37.712029    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ce834cc-d34d-46cc-a081-4d2cc261becb-tmp-dir\") pod \"distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7\" (UID: \"1ce834cc-d34d-46cc-a081-4d2cc261becb\") " pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"
Nov 01 03:24:37.712493 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:37.712296    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ce834cc-d34d-46cc-a081-4d2cc261becb-home\") pod \"distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7\" (UID: \"1ce834cc-d34d-46cc-a081-4d2cc261becb\") " pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"
Nov 01 03:24:37.720475 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:37.712925    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ce834cc-d34d-46cc-a081-4d2cc261becb-model-cache\") pod \"distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7\" (UID: \"1ce834cc-d34d-46cc-a081-4d2cc261becb\") " pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"
Nov 01 03:24:37.720475 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:37.716548    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ce834cc-d34d-46cc-a081-4d2cc261becb-tls-certs\") pod \"distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7\" (UID: \"1ce834cc-d34d-46cc-a081-4d2cc261becb\") " pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"
Nov 01 03:24:37.720475 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:37.719116    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ce834cc-d34d-46cc-a081-4d2cc261becb-dshm\") pod \"distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7\" (UID: \"1ce834cc-d34d-46cc-a081-4d2cc261becb\") " pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"
Nov 01 03:24:37.772882 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:37.772847    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wz65\" (UniqueName: \"kubernetes.io/projected/1ce834cc-d34d-46cc-a081-4d2cc261becb-kube-api-access-9wz65\") pod \"distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7\" (UID: \"1ce834cc-d34d-46cc-a081-4d2cc261becb\") " pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"
Nov 01 03:24:37.858217 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:37.858183    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"
Nov 01 03:24:38.303578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:38.302012    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"]
Nov 01 03:24:38.742579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:38.741146    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7" event={"ID":"1ce834cc-d34d-46cc-a081-4d2cc261becb","Type":"ContainerStarted","Data":"d0936fbc459faadef3f261b6ac7a19503cf768b4d884dcf5a86bdb6cbd4e3c20"}
Nov 01 03:24:38.742579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:38.741199    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7" event={"ID":"1ce834cc-d34d-46cc-a081-4d2cc261becb","Type":"ContainerStarted","Data":"c7294e5ec3b16e452c74a01d4da0ac9e6d5514d1952036997130d71067901e2b"}
Nov 01 03:24:38.879579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:38.871637    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"
Nov 01 03:24:38.879579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:38.873151    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7" podStartSLOduration=1.873133054 podStartE2EDuration="1.873133054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:24:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:24:38.860990303 +0000 UTC m=+1083.383996892" watchObservedRunningTime="2026-11-01 03:24:38.873133054 +0000 UTC m=+1083.396139643"
Nov 01 03:24:40.182581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:40.177664    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:24:41.723823 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:41.723793    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:24:44.948401 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:44.948363    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:24:45.278171 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:45.278084    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:24:47.859145 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:47.859109    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"
Nov 01 03:24:47.871600 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:47.871576    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"
Nov 01 03:24:48.831516 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:48.831491    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"
Nov 01 03:24:49.121684 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:24:49.121647    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:24:49.764320 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:49.764288    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:24:54.549299 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:54.549267    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:24:54.640341 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:54.640263    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw"]
Nov 01 03:24:54.644459 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:54.644437    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw"
Nov 01 03:24:54.668937 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:54.668908    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cd5db87-0172-46ce-b856-cfc6366d7cb2-istio-envoy\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw\" (UID: \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw"
Nov 01 03:24:54.669093 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:54.669071    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cd5db87-0172-46ce-b856-cfc6366d7cb2-istio-podinfo\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw\" (UID: \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw"
Nov 01 03:24:54.669161 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:54.669140    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd5db87-0172-46ce-b856-cfc6366d7cb2-workload-socket\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw\" (UID: \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw"
Nov 01 03:24:54.669224 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:54.669212    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cd5db87-0172-46ce-b856-cfc6366d7cb2-istio-data\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw\" (UID: \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw"
Nov 01 03:24:54.669281 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:54.669244    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cd5db87-0172-46ce-b856-cfc6366d7cb2-workload-certs\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw\" (UID: \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw"
Nov 01 03:24:54.669281 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:54.669275    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cd5db87-0172-46ce-b856-cfc6366d7cb2-istiod-ca-cert\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw\" (UID: \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw"
Nov 01 03:24:54.669410 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:54.669372    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cd5db87-0172-46ce-b856-cfc6366d7cb2-istio-token\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw\" (UID: \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw"
Nov 01 03:24:54.669477 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:54.669460    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd5db87-0172-46ce-b856-cfc6366d7cb2-credential-socket\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw\" (UID: \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw"
Nov 01 03:24:54.669531 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:54.669520    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzhdm\" (UniqueName: \"kubernetes.io/projected/9cd5db87-0172-46ce-b856-cfc6366d7cb2-kube-api-access-dzhdm\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw\" (UID: \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw"
Nov 01 03:24:54.771111 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:54.771071    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cd5db87-0172-46ce-b856-cfc6366d7cb2-istio-podinfo\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw\" (UID: \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw"
Nov 01 03:24:54.771294 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:54.771123    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd5db87-0172-46ce-b856-cfc6366d7cb2-workload-socket\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw\" (UID: \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw"
Nov 01 03:24:54.771294 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:54.771149    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cd5db87-0172-46ce-b856-cfc6366d7cb2-istio-data\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw\" (UID: \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw"
Nov 01 03:24:54.771294 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:54.771171    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cd5db87-0172-46ce-b856-cfc6366d7cb2-workload-certs\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw\" (UID: \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw"
Nov 01 03:24:54.771294 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:54.771186    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cd5db87-0172-46ce-b856-cfc6366d7cb2-istiod-ca-cert\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw\" (UID: \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw"
Nov 01 03:24:54.771294 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:54.771230    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cd5db87-0172-46ce-b856-cfc6366d7cb2-istio-token\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw\" (UID: \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw"
Nov 01 03:24:54.771647 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:54.771308    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd5db87-0172-46ce-b856-cfc6366d7cb2-credential-socket\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw\" (UID: \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw"
Nov 01 03:24:54.771716 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:54.771659    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cd5db87-0172-46ce-b856-cfc6366d7cb2-workload-certs\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw\" (UID: \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw"
Nov 01 03:24:54.771716 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:54.771699    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzhdm\" (UniqueName: \"kubernetes.io/projected/9cd5db87-0172-46ce-b856-cfc6366d7cb2-kube-api-access-dzhdm\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw\" (UID: \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw"
Nov 01 03:24:54.771827 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:54.771630    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cd5db87-0172-46ce-b856-cfc6366d7cb2-istio-data\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw\" (UID: \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw"
Nov 01 03:24:54.771983 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:54.771960    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cd5db87-0172-46ce-b856-cfc6366d7cb2-istio-envoy\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw\" (UID: \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw"
Nov 01 03:24:54.772088 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:54.772068    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cd5db87-0172-46ce-b856-cfc6366d7cb2-istiod-ca-cert\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw\" (UID: \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw"
Nov 01 03:24:54.772483 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:54.772459    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd5db87-0172-46ce-b856-cfc6366d7cb2-credential-socket\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw\" (UID: \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw"
Nov 01 03:24:54.772616 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:54.772591    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd5db87-0172-46ce-b856-cfc6366d7cb2-workload-socket\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw\" (UID: \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw"
Nov 01 03:24:54.774006 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:54.773982    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cd5db87-0172-46ce-b856-cfc6366d7cb2-istio-podinfo\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw\" (UID: \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw"
Nov 01 03:24:54.774195 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:54.774175    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cd5db87-0172-46ce-b856-cfc6366d7cb2-istio-envoy\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw\" (UID: \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw"
Nov 01 03:24:54.832753 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:54.832693    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cd5db87-0172-46ce-b856-cfc6366d7cb2-istio-token\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw\" (UID: \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw"
Nov 01 03:24:54.832892 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:54.832777    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzhdm\" (UniqueName: \"kubernetes.io/projected/9cd5db87-0172-46ce-b856-cfc6366d7cb2-kube-api-access-dzhdm\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw\" (UID: \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw"
Nov 01 03:24:54.861840 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:54.861797    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw"]
Nov 01 03:24:54.960673 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:54.960640    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw"
Nov 01 03:24:55.405116 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:24:55.405086    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cd5db87_0172_46ce_b856_cfc6366d7cb2.slice/crio-cc45f2cd577517a9ce54704373293c5877fb25528b3ff8707deb8f737f75edcd WatchSource:0}: Error finding container cc45f2cd577517a9ce54704373293c5877fb25528b3ff8707deb8f737f75edcd: Status 404 returned error can't find the container with id cc45f2cd577517a9ce54704373293c5877fb25528b3ff8707deb8f737f75edcd
Nov 01 03:24:55.405256 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:55.405115    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw"]
Nov 01 03:24:55.408113 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:55.408083    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:24:55.408192 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:55.408145    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:24:55.408192 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:55.408177    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:24:55.881794 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:55.881751    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw" event={"ID":"9cd5db87-0172-46ce-b856-cfc6366d7cb2","Type":"ContainerStarted","Data":"42ae3b9e3c4d81ad83023657e8fe369041adb8c52cff478cdfa0e186410cd13a"}
Nov 01 03:24:55.882204 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:55.881812    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw" event={"ID":"9cd5db87-0172-46ce-b856-cfc6366d7cb2","Type":"ContainerStarted","Data":"cc45f2cd577517a9ce54704373293c5877fb25528b3ff8707deb8f737f75edcd"}
Nov 01 03:24:55.969580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:55.967637    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw"
Nov 01 03:24:56.966582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:56.965769    4930 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 01 03:24:56.966582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:56.965843    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw" podUID="9cd5db87-0172-46ce-b856-cfc6366d7cb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 01 03:24:57.962948 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:57.962910    4930 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 01 03:24:57.963340 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:57.963307    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw" podUID="9cd5db87-0172-46ce-b856-cfc6366d7cb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 01 03:24:58.978584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:58.975874    4930 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 01 03:24:58.978584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:58.975931    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw" podUID="9cd5db87-0172-46ce-b856-cfc6366d7cb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 01 03:24:59.689400 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:59.689342    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:24:59.980632 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:59.977898    4930 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 01 03:24:59.980632 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:24:59.977960    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw" podUID="9cd5db87-0172-46ce-b856-cfc6366d7cb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 01 03:25:00.972582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:00.962875    4930 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 01 03:25:00.972582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:00.962933    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw" podUID="9cd5db87-0172-46ce-b856-cfc6366d7cb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 01 03:25:01.961921 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:01.961884    4930 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Nov 01 03:25:01.962340 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:01.961946    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw" podUID="9cd5db87-0172-46ce-b856-cfc6366d7cb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Nov 01 03:25:02.965482 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:02.965448    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw"
Nov 01 03:25:02.965984 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:02.965843    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw"
Nov 01 03:25:02.966408 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:02.966387    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw"
Nov 01 03:25:03.193224 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:03.193165    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw" podStartSLOduration=9.193139828 podStartE2EDuration="9.193139828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:24:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:24:56.071006645 +0000 UTC m=+1100.594013247" watchObservedRunningTime="2026-11-01 03:25:03.193139828 +0000 UTC m=+1107.716146418"
Nov 01 03:25:04.650618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:04.650544    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:25:05.830743 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:05.830709    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:25:05.979251 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:05.979211    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:25:09.065868 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:09.065835    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3efb826a-g5vr2"]
Nov 01 03:25:09.114770 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:09.114745    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3efb826a-g5vr2"]
Nov 01 03:25:09.566873 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:09.566835    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:25:09.638182 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:09.638148    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ffd51c9-c42a-4168-96ae-2a596c836aab" path="/var/lib/kubelet/pods/7ffd51c9-c42a-4168-96ae-2a596c836aab/volumes"
Nov 01 03:25:10.384614 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.384580    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr"]
Nov 01 03:25:10.389963 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.389946    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr"
Nov 01 03:25:10.422190 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.422163    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-workload-certs\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr\" (UID: \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr"
Nov 01 03:25:10.422317 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.422199    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnd8l\" (UniqueName: \"kubernetes.io/projected/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-kube-api-access-dnd8l\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr\" (UID: \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr"
Nov 01 03:25:10.422379 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.422295    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-workload-socket\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr\" (UID: \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr"
Nov 01 03:25:10.422437 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.422374    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-istiod-ca-cert\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr\" (UID: \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr"
Nov 01 03:25:10.422437 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.422409    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-istio-podinfo\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr\" (UID: \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr"
Nov 01 03:25:10.422531 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.422451    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-istio-data\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr\" (UID: \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr"
Nov 01 03:25:10.422531 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.422499    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-istio-token\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr\" (UID: \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr"
Nov 01 03:25:10.422749 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.422732    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-credential-socket\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr\" (UID: \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr"
Nov 01 03:25:10.422822 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.422763    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-istio-envoy\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr\" (UID: \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr"
Nov 01 03:25:10.524136 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.524101    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-credential-socket\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr\" (UID: \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr"
Nov 01 03:25:10.524263 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.524157    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-istio-envoy\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr\" (UID: \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr"
Nov 01 03:25:10.524263 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.524217    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-workload-certs\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr\" (UID: \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr"
Nov 01 03:25:10.524263 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.524236    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnd8l\" (UniqueName: \"kubernetes.io/projected/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-kube-api-access-dnd8l\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr\" (UID: \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr"
Nov 01 03:25:10.524443 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.524287    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-workload-socket\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr\" (UID: \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr"
Nov 01 03:25:10.524507 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.524441    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-credential-socket\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr\" (UID: \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr"
Nov 01 03:25:10.524507 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.524485    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-istiod-ca-cert\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr\" (UID: \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr"
Nov 01 03:25:10.524655 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.524514    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-istio-podinfo\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr\" (UID: \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr"
Nov 01 03:25:10.524655 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.524566    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-istio-data\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr\" (UID: \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr"
Nov 01 03:25:10.524655 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.524621    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-istio-token\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr\" (UID: \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr"
Nov 01 03:25:10.524807 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.524674    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-workload-socket\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr\" (UID: \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr"
Nov 01 03:25:10.524807 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.524621    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-workload-certs\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr\" (UID: \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr"
Nov 01 03:25:10.525034 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.525013    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-istio-data\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr\" (UID: \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr"
Nov 01 03:25:10.525213 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.525180    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-istiod-ca-cert\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr\" (UID: \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr"
Nov 01 03:25:10.526883 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.526861    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-istio-envoy\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr\" (UID: \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr"
Nov 01 03:25:10.527600 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.527572    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-istio-podinfo\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr\" (UID: \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr"
Nov 01 03:25:10.533731 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.533710    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr"]
Nov 01 03:25:10.586440 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.586397    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnd8l\" (UniqueName: \"kubernetes.io/projected/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-kube-api-access-dnd8l\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr\" (UID: \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr"
Nov 01 03:25:10.586605 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.586400    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-istio-token\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr\" (UID: \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr"
Nov 01 03:25:10.704503 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.704441    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr"
Nov 01 03:25:10.756408 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.756373    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss"]
Nov 01 03:25:10.762700 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.762675    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss"
Nov 01 03:25:10.824134 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.824083    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss"]
Nov 01 03:25:10.828412 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.828384    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6522cd84-3c6c-4280-9cb7-02eb8916b24d-istio-podinfo\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss\" (UID: \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss"
Nov 01 03:25:10.828548 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.828441    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg4fl\" (UniqueName: \"kubernetes.io/projected/6522cd84-3c6c-4280-9cb7-02eb8916b24d-kube-api-access-kg4fl\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss\" (UID: \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss"
Nov 01 03:25:10.828646 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.828544    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6522cd84-3c6c-4280-9cb7-02eb8916b24d-istiod-ca-cert\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss\" (UID: \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss"
Nov 01 03:25:10.828646 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.828620    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6522cd84-3c6c-4280-9cb7-02eb8916b24d-workload-socket\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss\" (UID: \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss"
Nov 01 03:25:10.828646 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.828639    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6522cd84-3c6c-4280-9cb7-02eb8916b24d-workload-certs\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss\" (UID: \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss"
Nov 01 03:25:10.828808 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.828720    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6522cd84-3c6c-4280-9cb7-02eb8916b24d-credential-socket\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss\" (UID: \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss"
Nov 01 03:25:10.828808 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.828748    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6522cd84-3c6c-4280-9cb7-02eb8916b24d-istio-token\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss\" (UID: \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss"
Nov 01 03:25:10.828808 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.828784    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6522cd84-3c6c-4280-9cb7-02eb8916b24d-istio-data\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss\" (UID: \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss"
Nov 01 03:25:10.828808 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.828802    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6522cd84-3c6c-4280-9cb7-02eb8916b24d-istio-envoy\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss\" (UID: \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss"
Nov 01 03:25:10.929425 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.929396    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6522cd84-3c6c-4280-9cb7-02eb8916b24d-istio-podinfo\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss\" (UID: \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss"
Nov 01 03:25:10.929622 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.929444    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg4fl\" (UniqueName: \"kubernetes.io/projected/6522cd84-3c6c-4280-9cb7-02eb8916b24d-kube-api-access-kg4fl\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss\" (UID: \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss"
Nov 01 03:25:10.929622 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.929485    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6522cd84-3c6c-4280-9cb7-02eb8916b24d-istiod-ca-cert\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss\" (UID: \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss"
Nov 01 03:25:10.929622 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.929513    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6522cd84-3c6c-4280-9cb7-02eb8916b24d-workload-socket\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss\" (UID: \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss"
Nov 01 03:25:10.929622 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.929530    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6522cd84-3c6c-4280-9cb7-02eb8916b24d-workload-certs\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss\" (UID: \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss"
Nov 01 03:25:10.929866 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.929641    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6522cd84-3c6c-4280-9cb7-02eb8916b24d-credential-socket\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss\" (UID: \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss"
Nov 01 03:25:10.929866 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.929673    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6522cd84-3c6c-4280-9cb7-02eb8916b24d-istio-token\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss\" (UID: \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss"
Nov 01 03:25:10.929866 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.929730    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6522cd84-3c6c-4280-9cb7-02eb8916b24d-istio-data\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss\" (UID: \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss"
Nov 01 03:25:10.930027 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.929876    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6522cd84-3c6c-4280-9cb7-02eb8916b24d-istio-envoy\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss\" (UID: \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss"
Nov 01 03:25:10.930082 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.930022    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6522cd84-3c6c-4280-9cb7-02eb8916b24d-workload-socket\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss\" (UID: \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss"
Nov 01 03:25:10.930121 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.930098    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6522cd84-3c6c-4280-9cb7-02eb8916b24d-credential-socket\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss\" (UID: \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss"
Nov 01 03:25:10.930204 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.930171    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6522cd84-3c6c-4280-9cb7-02eb8916b24d-workload-certs\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss\" (UID: \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss"
Nov 01 03:25:10.930251 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.930203    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6522cd84-3c6c-4280-9cb7-02eb8916b24d-istio-data\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss\" (UID: \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss"
Nov 01 03:25:10.930377 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.930360    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6522cd84-3c6c-4280-9cb7-02eb8916b24d-istiod-ca-cert\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss\" (UID: \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss"
Nov 01 03:25:10.932109 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.932087    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6522cd84-3c6c-4280-9cb7-02eb8916b24d-istio-envoy\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss\" (UID: \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss"
Nov 01 03:25:10.932198 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.932092    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6522cd84-3c6c-4280-9cb7-02eb8916b24d-istio-podinfo\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss\" (UID: \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss"
Nov 01 03:25:10.990678 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.990613    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6522cd84-3c6c-4280-9cb7-02eb8916b24d-istio-token\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss\" (UID: \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss"
Nov 01 03:25:10.990799 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:10.990693    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg4fl\" (UniqueName: \"kubernetes.io/projected/6522cd84-3c6c-4280-9cb7-02eb8916b24d-kube-api-access-kg4fl\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss\" (UID: \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss"
Nov 01 03:25:11.086590 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:11.086542    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss"
Nov 01 03:25:11.146201 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:11.146171    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr"]
Nov 01 03:25:11.148700 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:11.148665    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:25:11.148814 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:11.148746    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:25:11.148814 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:11.148782    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:25:11.472845 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:25:11.472807    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6522cd84_3c6c_4280_9cb7_02eb8916b24d.slice/crio-f73be621b68afb71cee1ddedd185a5670ca5137e650a6c8cff258d72d9838668 WatchSource:0}: Error finding container f73be621b68afb71cee1ddedd185a5670ca5137e650a6c8cff258d72d9838668: Status 404 returned error can't find the container with id f73be621b68afb71cee1ddedd185a5670ca5137e650a6c8cff258d72d9838668
Nov 01 03:25:11.475531 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:11.475505    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss"]
Nov 01 03:25:11.475764 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:11.475730    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:25:11.475824 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:11.475809    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:25:11.475869 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:11.475854    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:25:12.026130 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:12.026096    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss" event={"ID":"6522cd84-3c6c-4280-9cb7-02eb8916b24d","Type":"ContainerStarted","Data":"b09efc044d9f444cbc90815da3059fbc3c8c18a835c2f06233f51cc908b7a3ed"}
Nov 01 03:25:12.026339 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:12.026157    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss" event={"ID":"6522cd84-3c6c-4280-9cb7-02eb8916b24d","Type":"ContainerStarted","Data":"f73be621b68afb71cee1ddedd185a5670ca5137e650a6c8cff258d72d9838668"}
Nov 01 03:25:12.027979 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:12.027951    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr" event={"ID":"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8","Type":"ContainerStarted","Data":"dbfcbf6fc66767eb265f4df5b42d750a5c8403f2ed14262eef96a0a9a97f651c"}
Nov 01 03:25:12.028076 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:12.027999    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr" event={"ID":"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8","Type":"ContainerStarted","Data":"6ed3f3636286b90c34e710dc54ec90876ada3a97a455a76542cdf865617c1336"}
Nov 01 03:25:12.094580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:12.094287    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss"
Nov 01 03:25:12.094580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:12.094479    4930 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 01 03:25:12.094580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:12.094532    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss" podUID="6522cd84-3c6c-4280-9cb7-02eb8916b24d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 01 03:25:12.151719 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:12.151652    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss" podStartSLOduration=2.151632929 podStartE2EDuration="2.151632929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:25:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:25:12.147371392 +0000 UTC m=+1116.670378020" watchObservedRunningTime="2026-11-01 03:25:12.151632929 +0000 UTC m=+1116.674639519"
Nov 01 03:25:12.305638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:12.301845    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr" podStartSLOduration=2.301825103 podStartE2EDuration="2.301825103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:25:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:25:12.285898021 +0000 UTC m=+1116.808904611" watchObservedRunningTime="2026-11-01 03:25:12.301825103 +0000 UTC m=+1116.824831692"
Nov 01 03:25:12.707919 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:12.707886    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr"
Nov 01 03:25:12.712179 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:12.711782    4930 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 01 03:25:12.712179 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:12.711837    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr" podUID="f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 01 03:25:13.094586 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:13.093946    4930 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 01 03:25:13.094586 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:13.094005    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss" podUID="6522cd84-3c6c-4280-9cb7-02eb8916b24d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 01 03:25:13.706580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:13.706198    4930 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 01 03:25:13.706580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:13.706257    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr" podUID="f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 01 03:25:14.090795 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:14.090761    4930 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 01 03:25:14.091383 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:14.091355    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss" podUID="6522cd84-3c6c-4280-9cb7-02eb8916b24d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 01 03:25:14.684288 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:14.684246    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:25:14.709979 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:14.709937    4930 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 01 03:25:14.710288 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:14.710257    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr" podUID="f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 01 03:25:15.103576 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:15.090931    4930 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 01 03:25:15.103576 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:15.090991    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss" podUID="6522cd84-3c6c-4280-9cb7-02eb8916b24d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 01 03:25:15.711578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:15.710863    4930 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 01 03:25:15.711578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:15.710920    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr" podUID="f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 01 03:25:16.092578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:16.088931    4930 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 01 03:25:16.092578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:16.088991    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss" podUID="6522cd84-3c6c-4280-9cb7-02eb8916b24d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 01 03:25:16.706580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:16.705936    4930 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 01 03:25:16.706580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:16.705998    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr" podUID="f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 01 03:25:17.101585 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:17.093849    4930 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 01 03:25:17.101585 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:17.093922    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss" podUID="6522cd84-3c6c-4280-9cb7-02eb8916b24d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 01 03:25:17.706521 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:17.706477    4930 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 01 03:25:17.706725 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:17.706574    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr" podUID="f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 01 03:25:18.087983 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:18.087948    4930 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 01 03:25:18.088179 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:18.088011    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss" podUID="6522cd84-3c6c-4280-9cb7-02eb8916b24d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 01 03:25:18.708658 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:18.708619    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr"
Nov 01 03:25:18.709137 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:18.709117    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr"
Nov 01 03:25:18.709734 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:18.709711    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr"
Nov 01 03:25:19.092809 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:19.092780    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss"
Nov 01 03:25:19.093391 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:19.093364    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss"
Nov 01 03:25:19.094633 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:19.094610    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss"
Nov 01 03:25:19.132411 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:25:19.132377    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:25:19.557395 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:19.557315    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:25:24.439130 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:24.439092    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:25:25.519208 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:25.519175    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll"]
Nov 01 03:25:25.523508 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:25.523490    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll"
Nov 01 03:25:25.626898 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:25.626865    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll"]
Nov 01 03:25:25.676414 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:25.676378    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-istio-token\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll\" (UID: \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll"
Nov 01 03:25:25.676625 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:25.676531    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmdrs\" (UniqueName: \"kubernetes.io/projected/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-kube-api-access-xmdrs\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll\" (UID: \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll"
Nov 01 03:25:25.676713 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:25.676641    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-istio-envoy\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll\" (UID: \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll"
Nov 01 03:25:25.676777 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:25.676714    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-istio-podinfo\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll\" (UID: \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll"
Nov 01 03:25:25.676777 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:25.676742    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-credential-socket\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll\" (UID: \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll"
Nov 01 03:25:25.676894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:25.676798    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-istio-data\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll\" (UID: \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll"
Nov 01 03:25:25.676894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:25.676831    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-workload-certs\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll\" (UID: \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll"
Nov 01 03:25:25.676894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:25.676850    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-istiod-ca-cert\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll\" (UID: \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll"
Nov 01 03:25:25.676894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:25.676878    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-workload-socket\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll\" (UID: \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll"
Nov 01 03:25:25.778814 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:25.778722    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmdrs\" (UniqueName: \"kubernetes.io/projected/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-kube-api-access-xmdrs\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll\" (UID: \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll"
Nov 01 03:25:25.778974 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:25.778823    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-istio-envoy\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll\" (UID: \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll"
Nov 01 03:25:25.778974 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:25.778899    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-istio-podinfo\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll\" (UID: \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll"
Nov 01 03:25:25.778974 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:25.778952    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-credential-socket\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll\" (UID: \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll"
Nov 01 03:25:25.779163 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:25.779040    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-istio-data\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll\" (UID: \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll"
Nov 01 03:25:25.779163 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:25.779092    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-workload-certs\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll\" (UID: \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll"
Nov 01 03:25:25.779163 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:25.779132    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-istiod-ca-cert\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll\" (UID: \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll"
Nov 01 03:25:25.779315 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:25.779183    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-workload-socket\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll\" (UID: \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll"
Nov 01 03:25:25.779315 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:25.779217    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-istio-token\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll\" (UID: \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll"
Nov 01 03:25:25.779425 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:25.779400    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-credential-socket\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll\" (UID: \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll"
Nov 01 03:25:25.779608 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:25.779548    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-workload-certs\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll\" (UID: \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll"
Nov 01 03:25:25.779714 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:25.779659    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-istio-data\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll\" (UID: \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll"
Nov 01 03:25:25.779813 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:25.779787    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-workload-socket\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll\" (UID: \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll"
Nov 01 03:25:25.780012 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:25.779982    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-istiod-ca-cert\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll\" (UID: \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll"
Nov 01 03:25:25.781837 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:25.781817    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-istio-envoy\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll\" (UID: \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll"
Nov 01 03:25:25.781972 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:25.781948    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-istio-podinfo\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll\" (UID: \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll"
Nov 01 03:25:25.842116 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:25.842083    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-istio-token\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll\" (UID: \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll"
Nov 01 03:25:25.842324 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:25.842301    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmdrs\" (UniqueName: \"kubernetes.io/projected/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-kube-api-access-xmdrs\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll\" (UID: \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll"
Nov 01 03:25:26.138536 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:26.138494    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll"
Nov 01 03:25:26.573649 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:25:26.573611    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0405e5bf_ff6e_467d_9f4c_9bd36005c3ee.slice/crio-5fd6986d23225df3408eee472e112643b562dd0c5cc246a3bf11a323ba520ece WatchSource:0}: Error finding container 5fd6986d23225df3408eee472e112643b562dd0c5cc246a3bf11a323ba520ece: Status 404 returned error can't find the container with id 5fd6986d23225df3408eee472e112643b562dd0c5cc246a3bf11a323ba520ece
Nov 01 03:25:26.575832 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:26.575808    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll"]
Nov 01 03:25:26.576795 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:26.576741    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:25:26.576874 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:26.576852    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:25:26.576922 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:26.576911    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:25:27.200076 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:27.200042    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll" event={"ID":"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee","Type":"ContainerStarted","Data":"6f0ca25fbe6521eb5afe7e04a2a8d2422747e1d80f90124ec2776a591e3a8351"}
Nov 01 03:25:27.200301 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:27.200093    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll" event={"ID":"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee","Type":"ContainerStarted","Data":"5fd6986d23225df3408eee472e112643b562dd0c5cc246a3bf11a323ba520ece"}
Nov 01 03:25:27.321239 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:27.321185    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll" podStartSLOduration=2.321168187 podStartE2EDuration="2.321168187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:25:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:25:27.319390739 +0000 UTC m=+1131.842397345" watchObservedRunningTime="2026-11-01 03:25:27.321168187 +0000 UTC m=+1131.844174774"
Nov 01 03:25:27.707034 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:27.704286    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18318018-p94kg"]
Nov 01 03:25:27.712201 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:27.710924    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18318018-p94kg"
Nov 01 03:25:27.764943 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:27.764911    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:25:27.765315 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:27.765297    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:25:27.765550 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:27.765535    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:25:27.786581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:27.769166    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:25:27.786825 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:27.781269    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18318018-p94kg"]
Nov 01 03:25:27.806291 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:27.806250    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14655c03-e09f-4917-b51a-fd2a5b01e119-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18318018-p94kg\" (UID: \"14655c03-e09f-4917-b51a-fd2a5b01e119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18318018-p94kg"
Nov 01 03:25:27.806623 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:27.806597    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qszz4\" (UniqueName: \"kubernetes.io/projected/14655c03-e09f-4917-b51a-fd2a5b01e119-kube-api-access-qszz4\") pod \"maas-api-revoke-sub-18318018-p94kg\" (UID: \"14655c03-e09f-4917-b51a-fd2a5b01e119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18318018-p94kg"
Nov 01 03:25:27.806856 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:27.806837    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14655c03-e09f-4917-b51a-fd2a5b01e119-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18318018-p94kg\" (UID: \"14655c03-e09f-4917-b51a-fd2a5b01e119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18318018-p94kg"
Nov 01 03:25:27.908392 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:27.908345    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qszz4\" (UniqueName: \"kubernetes.io/projected/14655c03-e09f-4917-b51a-fd2a5b01e119-kube-api-access-qszz4\") pod \"maas-api-revoke-sub-18318018-p94kg\" (UID: \"14655c03-e09f-4917-b51a-fd2a5b01e119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18318018-p94kg"
Nov 01 03:25:27.908598 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:27.908440    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14655c03-e09f-4917-b51a-fd2a5b01e119-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18318018-p94kg\" (UID: \"14655c03-e09f-4917-b51a-fd2a5b01e119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18318018-p94kg"
Nov 01 03:25:27.908598 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:27.908576    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14655c03-e09f-4917-b51a-fd2a5b01e119-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18318018-p94kg\" (UID: \"14655c03-e09f-4917-b51a-fd2a5b01e119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18318018-p94kg"
Nov 01 03:25:27.909445 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:27.909420    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14655c03-e09f-4917-b51a-fd2a5b01e119-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18318018-p94kg\" (UID: \"14655c03-e09f-4917-b51a-fd2a5b01e119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18318018-p94kg"
Nov 01 03:25:27.911909 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:27.911884    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14655c03-e09f-4917-b51a-fd2a5b01e119-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18318018-p94kg\" (UID: \"14655c03-e09f-4917-b51a-fd2a5b01e119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18318018-p94kg"
Nov 01 03:25:28.025661 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:28.024795    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qszz4\" (UniqueName: \"kubernetes.io/projected/14655c03-e09f-4917-b51a-fd2a5b01e119-kube-api-access-qszz4\") pod \"maas-api-revoke-sub-18318018-p94kg\" (UID: \"14655c03-e09f-4917-b51a-fd2a5b01e119\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18318018-p94kg"
Nov 01 03:25:28.033920 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:28.033853    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18318018-p94kg"
Nov 01 03:25:28.152673 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:28.147166    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll"
Nov 01 03:25:28.157680 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:28.157647    4930 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 01 03:25:28.157900 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:28.157870    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll" podUID="0405e5bf-ff6e-467d-9f4c-9bd36005c3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 01 03:25:28.551908 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:28.551861    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18318018-p94kg"]
Nov 01 03:25:29.143578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:29.139871    4930 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 01 03:25:29.143578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:29.139929    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll" podUID="0405e5bf-ff6e-467d-9f4c-9bd36005c3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 01 03:25:29.254568 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:29.254458    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18318018-p94kg" event={"ID":"14655c03-e09f-4917-b51a-fd2a5b01e119","Type":"ContainerStarted","Data":"75cbc6f855f445fa673d0120006dbc6bc57936825d128b1e989c73370c4b43e4"}
Nov 01 03:25:29.254857 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:29.254837    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18318018-p94kg" event={"ID":"14655c03-e09f-4917-b51a-fd2a5b01e119","Type":"ContainerStarted","Data":"60146afff549dad6e485d20c18770f2b930f7c7c9c082a7f3ef2cf685e91c499"}
Nov 01 03:25:29.383582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:29.380413    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18318018-p94kg" podStartSLOduration=2.380377654 podStartE2EDuration="2.380377654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:25:29.378092543 +0000 UTC m=+1133.901099135" watchObservedRunningTime="2026-11-01 03:25:29.380377654 +0000 UTC m=+1133.903384244"
Nov 01 03:25:29.406320 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:29.406287    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:25:30.145049 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:30.144579    4930 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 01 03:25:30.145049 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:30.144676    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll" podUID="0405e5bf-ff6e-467d-9f4c-9bd36005c3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 01 03:25:30.290580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:30.289751    4930 generic.go:356] "Generic (PLEG): container finished" podID="14655c03-e09f-4917-b51a-fd2a5b01e119" containerID="75cbc6f855f445fa673d0120006dbc6bc57936825d128b1e989c73370c4b43e4" exitCode=0
Nov 01 03:25:30.290580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:30.289925    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18318018-p94kg" event={"ID":"14655c03-e09f-4917-b51a-fd2a5b01e119","Type":"ContainerDied","Data":"75cbc6f855f445fa673d0120006dbc6bc57936825d128b1e989c73370c4b43e4"}
Nov 01 03:25:31.139810 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:31.139768    4930 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 01 03:25:31.140002 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:31.139841    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll" podUID="0405e5bf-ff6e-467d-9f4c-9bd36005c3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 01 03:25:31.552730 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:31.551889    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:25:31.735124 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:31.735101    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18318018-p94kg"
Nov 01 03:25:31.848131 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:31.848097    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14655c03-e09f-4917-b51a-fd2a5b01e119-kube-api-access-qszz4\" (UniqueName: \"kubernetes.io/projected/14655c03-e09f-4917-b51a-fd2a5b01e119-kube-api-access-qszz4\") pod \"14655c03-e09f-4917-b51a-fd2a5b01e119\" (UID: \"14655c03-e09f-4917-b51a-fd2a5b01e119\") "
Nov 01 03:25:31.860620 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:31.849058    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14655c03-e09f-4917-b51a-fd2a5b01e119-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14655c03-e09f-4917-b51a-fd2a5b01e119-maas-api-openshift-ca\") pod \"14655c03-e09f-4917-b51a-fd2a5b01e119\" (UID: \"14655c03-e09f-4917-b51a-fd2a5b01e119\") "
Nov 01 03:25:31.860957 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:31.860933    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14655c03-e09f-4917-b51a-fd2a5b01e119-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14655c03-e09f-4917-b51a-fd2a5b01e119-maas-api-xks-ca\") pod \"14655c03-e09f-4917-b51a-fd2a5b01e119\" (UID: \"14655c03-e09f-4917-b51a-fd2a5b01e119\") "
Nov 01 03:25:31.866164 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:31.859153    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14655c03-e09f-4917-b51a-fd2a5b01e119-maas-api-openshift-ca" pod "14655c03-e09f-4917-b51a-fd2a5b01e119" (UID: "14655c03-e09f-4917-b51a-fd2a5b01e119"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:25:31.866335 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:31.866128    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14655c03-e09f-4917-b51a-fd2a5b01e119-kube-api-access-qszz4" pod "14655c03-e09f-4917-b51a-fd2a5b01e119" (UID: "14655c03-e09f-4917-b51a-fd2a5b01e119"). InnerVolumeSpecName "kube-api-access-qszz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:25:31.878596 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:31.875684    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14655c03-e09f-4917-b51a-fd2a5b01e119-maas-api-xks-ca" pod "14655c03-e09f-4917-b51a-fd2a5b01e119" (UID: "14655c03-e09f-4917-b51a-fd2a5b01e119"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:25:31.993082 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:31.992976    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14655c03-e09f-4917-b51a-fd2a5b01e119-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:25:31.993306 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:31.993287    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14655c03-e09f-4917-b51a-fd2a5b01e119-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:25:31.993435 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:31.993422    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qszz4\" (UniqueName: \"kubernetes.io/projected/14655c03-e09f-4917-b51a-fd2a5b01e119-kube-api-access-qszz4\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:25:32.091403 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:32.091027    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:25:32.150589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:32.146642    4930 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 01 03:25:32.150589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:32.146700    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll" podUID="0405e5bf-ff6e-467d-9f4c-9bd36005c3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 01 03:25:32.313638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:32.313535    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18318018-p94kg" event={"ID":"14655c03-e09f-4917-b51a-fd2a5b01e119","Type":"ContainerDied","Data":"60146afff549dad6e485d20c18770f2b930f7c7c9c082a7f3ef2cf685e91c499"}
Nov 01 03:25:32.313638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:32.313609    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60146afff549dad6e485d20c18770f2b930f7c7c9c082a7f3ef2cf685e91c499"
Nov 01 03:25:32.313857 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:32.313678    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18318018-p94kg"
Nov 01 03:25:33.138976 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:33.138943    4930 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 01 03:25:33.139354 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:33.139002    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll" podUID="0405e5bf-ff6e-467d-9f4c-9bd36005c3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 01 03:25:34.145139 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:34.145109    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll"
Nov 01 03:25:34.145621 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:34.145530    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll"
Nov 01 03:25:34.146169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:34.146150    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll"
Nov 01 03:25:34.357189 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:34.357154    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:25:39.054715 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:39.054677    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28a6d1f1-zsn8r"]
Nov 01 03:25:39.079633 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:39.079595    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28a6d1f1-zsn8r"]
Nov 01 03:25:39.105678 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:39.105647    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:25:39.256727 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:39.256691    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:25:39.620730 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:39.620700    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbd0fcba-9a80-42ce-abff-fee5fe08b065" path="/var/lib/kubelet/pods/fbd0fcba-9a80-42ce-abff-fee5fe08b065/volumes"
Nov 01 03:25:42.074240 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:42.074200    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:25:42.197622 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:42.197584    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99b75fdf-68cgr"]
Nov 01 03:25:42.198142 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:42.198129    4930 cpu_manager.go:414] "containerMap: removing container" podUID="14655c03-e09f-4917-b51a-fd2a5b01e119" containerName="revoke-keys"
Nov 01 03:25:42.198186 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:42.198144    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14655c03-e09f-4917-b51a-fd2a5b01e119" containerName="revoke-keys"
Nov 01 03:25:42.198260 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:42.198251    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="14655c03-e09f-4917-b51a-fd2a5b01e119" containerName="revoke-keys"
Nov 01 03:25:42.203602 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:42.203572    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99b75fdf-68cgr"
Nov 01 03:25:42.260289 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:42.260244    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99b75fdf-68cgr"]
Nov 01 03:25:42.303182 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:42.303143    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:25:42.303182 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:42.303161    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:25:42.303182 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:42.303147    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:25:42.303479 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:42.303231    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:25:42.337666 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:42.337636    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48777d1-2695-4350-9a34-07d3319086c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99b75fdf-68cgr\" (UID: \"d48777d1-2695-4350-9a34-07d3319086c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99b75fdf-68cgr"
Nov 01 03:25:42.337872 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:42.337836    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48777d1-2695-4350-9a34-07d3319086c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99b75fdf-68cgr\" (UID: \"d48777d1-2695-4350-9a34-07d3319086c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99b75fdf-68cgr"
Nov 01 03:25:42.337946 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:42.337913    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2ntr\" (UniqueName: \"kubernetes.io/projected/d48777d1-2695-4350-9a34-07d3319086c3-kube-api-access-v2ntr\") pod \"maas-api-revoke-sub-99b75fdf-68cgr\" (UID: \"d48777d1-2695-4350-9a34-07d3319086c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99b75fdf-68cgr"
Nov 01 03:25:42.441507 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:42.441470    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48777d1-2695-4350-9a34-07d3319086c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99b75fdf-68cgr\" (UID: \"d48777d1-2695-4350-9a34-07d3319086c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99b75fdf-68cgr"
Nov 01 03:25:42.442675 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:42.442647    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2ntr\" (UniqueName: \"kubernetes.io/projected/d48777d1-2695-4350-9a34-07d3319086c3-kube-api-access-v2ntr\") pod \"maas-api-revoke-sub-99b75fdf-68cgr\" (UID: \"d48777d1-2695-4350-9a34-07d3319086c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99b75fdf-68cgr"
Nov 01 03:25:42.443007 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:42.442974    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48777d1-2695-4350-9a34-07d3319086c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99b75fdf-68cgr\" (UID: \"d48777d1-2695-4350-9a34-07d3319086c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99b75fdf-68cgr"
Nov 01 03:25:42.455682 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:42.442501    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48777d1-2695-4350-9a34-07d3319086c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99b75fdf-68cgr\" (UID: \"d48777d1-2695-4350-9a34-07d3319086c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99b75fdf-68cgr"
Nov 01 03:25:42.487451 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:42.487415    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48777d1-2695-4350-9a34-07d3319086c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99b75fdf-68cgr\" (UID: \"d48777d1-2695-4350-9a34-07d3319086c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99b75fdf-68cgr"
Nov 01 03:25:42.532604 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:42.523321    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2ntr\" (UniqueName: \"kubernetes.io/projected/d48777d1-2695-4350-9a34-07d3319086c3-kube-api-access-v2ntr\") pod \"maas-api-revoke-sub-99b75fdf-68cgr\" (UID: \"d48777d1-2695-4350-9a34-07d3319086c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99b75fdf-68cgr"
Nov 01 03:25:42.832047 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:42.830826    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99b75fdf-68cgr"
Nov 01 03:25:43.361626 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:25:43.359958    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd48777d1_2695_4350_9a34_07d3319086c3.slice/crio-c76a0883995a30c48f12fdbea796078b290940e2d7e2c4a587531024e43c2aef WatchSource:0}: Error finding container c76a0883995a30c48f12fdbea796078b290940e2d7e2c4a587531024e43c2aef: Status 404 returned error can't find the container with id c76a0883995a30c48f12fdbea796078b290940e2d7e2c4a587531024e43c2aef
Nov 01 03:25:43.362881 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:43.362851    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99b75fdf-68cgr"]
Nov 01 03:25:43.500578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:43.495655    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99b75fdf-68cgr" event={"ID":"d48777d1-2695-4350-9a34-07d3319086c3","Type":"ContainerStarted","Data":"c76a0883995a30c48f12fdbea796078b290940e2d7e2c4a587531024e43c2aef"}
Nov 01 03:25:43.968574 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:43.968102    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:25:44.560587 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:44.560405    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99b75fdf-68cgr" event={"ID":"d48777d1-2695-4350-9a34-07d3319086c3","Type":"ContainerStarted","Data":"e0e1d464c7abb6214395efa5bf6573b3d8ca32c8cbf6e090b7d3a557585cf000"}
Nov 01 03:25:44.688449 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:44.688389    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99b75fdf-68cgr" podStartSLOduration=2.688375763 podStartE2EDuration="2.688375763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:25:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:25:44.685855085 +0000 UTC m=+1149.208861674" watchObservedRunningTime="2026-11-01 03:25:44.688375763 +0000 UTC m=+1149.211382350"
Nov 01 03:25:45.571696 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:45.571653    4930 generic.go:356] "Generic (PLEG): container finished" podID="d48777d1-2695-4350-9a34-07d3319086c3" containerID="e0e1d464c7abb6214395efa5bf6573b3d8ca32c8cbf6e090b7d3a557585cf000" exitCode=0
Nov 01 03:25:45.572147 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:45.571738    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99b75fdf-68cgr" event={"ID":"d48777d1-2695-4350-9a34-07d3319086c3","Type":"ContainerDied","Data":"e0e1d464c7abb6214395efa5bf6573b3d8ca32c8cbf6e090b7d3a557585cf000"}
Nov 01 03:25:46.739651 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:46.739625    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99b75fdf-68cgr"
Nov 01 03:25:46.764960 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:46.764932    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d48777d1-2695-4350-9a34-07d3319086c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48777d1-2695-4350-9a34-07d3319086c3-maas-api-xks-ca\") pod \"d48777d1-2695-4350-9a34-07d3319086c3\" (UID: \"d48777d1-2695-4350-9a34-07d3319086c3\") "
Nov 01 03:25:46.765079 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:46.764983    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d48777d1-2695-4350-9a34-07d3319086c3-kube-api-access-v2ntr\" (UniqueName: \"kubernetes.io/projected/d48777d1-2695-4350-9a34-07d3319086c3-kube-api-access-v2ntr\") pod \"d48777d1-2695-4350-9a34-07d3319086c3\" (UID: \"d48777d1-2695-4350-9a34-07d3319086c3\") "
Nov 01 03:25:46.765177 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:46.765158    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d48777d1-2695-4350-9a34-07d3319086c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48777d1-2695-4350-9a34-07d3319086c3-maas-api-openshift-ca\") pod \"d48777d1-2695-4350-9a34-07d3319086c3\" (UID: \"d48777d1-2695-4350-9a34-07d3319086c3\") "
Nov 01 03:25:46.765616 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:46.765546    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d48777d1-2695-4350-9a34-07d3319086c3-maas-api-openshift-ca" pod "d48777d1-2695-4350-9a34-07d3319086c3" (UID: "d48777d1-2695-4350-9a34-07d3319086c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:25:46.765773 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:46.765754    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d48777d1-2695-4350-9a34-07d3319086c3-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:25:46.767077 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:46.767051    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d48777d1-2695-4350-9a34-07d3319086c3-kube-api-access-v2ntr" pod "d48777d1-2695-4350-9a34-07d3319086c3" (UID: "d48777d1-2695-4350-9a34-07d3319086c3"). InnerVolumeSpecName "kube-api-access-v2ntr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:25:46.767202 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:46.767122    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d48777d1-2695-4350-9a34-07d3319086c3-maas-api-xks-ca" pod "d48777d1-2695-4350-9a34-07d3319086c3" (UID: "d48777d1-2695-4350-9a34-07d3319086c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:25:46.867509 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:46.867467    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d48777d1-2695-4350-9a34-07d3319086c3-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:25:46.867509 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:46.867506    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2ntr\" (UniqueName: \"kubernetes.io/projected/d48777d1-2695-4350-9a34-07d3319086c3-kube-api-access-v2ntr\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:25:47.592760 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:47.592731    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99b75fdf-68cgr"
Nov 01 03:25:47.592962 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:47.592762    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99b75fdf-68cgr" event={"ID":"d48777d1-2695-4350-9a34-07d3319086c3","Type":"ContainerDied","Data":"c76a0883995a30c48f12fdbea796078b290940e2d7e2c4a587531024e43c2aef"}
Nov 01 03:25:47.592962 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:47.592815    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c76a0883995a30c48f12fdbea796078b290940e2d7e2c4a587531024e43c2aef"
Nov 01 03:25:48.837236 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:48.837200    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:25:49.121672 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:25:49.121592    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:25:53.689368 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:53.689318    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:25:54.052391 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:54.052308    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69b8ffed-9nl8q"]
Nov 01 03:25:54.101833 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:54.101804    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69b8ffed-9nl8q"]
Nov 01 03:25:55.621418 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:55.621384    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6398465-9388-479e-85c9-b44b1797deff" path="/var/lib/kubelet/pods/b6398465-9388-479e-85c9-b44b1797deff/volumes"
Nov 01 03:25:55.789893 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:55.789846    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n"]
Nov 01 03:25:55.791569 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:55.791524    4930 cpu_manager.go:414] "containerMap: removing container" podUID="d48777d1-2695-4350-9a34-07d3319086c3" containerName="revoke-keys"
Nov 01 03:25:55.791747 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:55.791733    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d48777d1-2695-4350-9a34-07d3319086c3" containerName="revoke-keys"
Nov 01 03:25:55.792182 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:55.792166    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="d48777d1-2695-4350-9a34-07d3319086c3" containerName="revoke-keys"
Nov 01 03:25:55.799664 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:55.799639    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n"
Nov 01 03:25:55.879617 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:55.879517    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f4df86a-a104-4c26-9b8e-9b556b23afcf-istio-token\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n\" (UID: \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n"
Nov 01 03:25:55.879764 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:55.879688    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f4df86a-a104-4c26-9b8e-9b556b23afcf-istio-data\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n\" (UID: \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n"
Nov 01 03:25:55.879764 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:55.879753    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4df86a-a104-4c26-9b8e-9b556b23afcf-workload-socket\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n\" (UID: \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n"
Nov 01 03:25:55.879886 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:55.879870    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f4df86a-a104-4c26-9b8e-9b556b23afcf-istio-podinfo\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n\" (UID: \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n"
Nov 01 03:25:55.880030 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:55.880005    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f4df86a-a104-4c26-9b8e-9b556b23afcf-istio-envoy\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n\" (UID: \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n"
Nov 01 03:25:55.880073 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:55.880061    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4df86a-a104-4c26-9b8e-9b556b23afcf-credential-socket\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n\" (UID: \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n"
Nov 01 03:25:55.880142 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:55.880124    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f4df86a-a104-4c26-9b8e-9b556b23afcf-istiod-ca-cert\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n\" (UID: \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n"
Nov 01 03:25:55.880263 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:55.880242    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxrnj\" (UniqueName: \"kubernetes.io/projected/7f4df86a-a104-4c26-9b8e-9b556b23afcf-kube-api-access-cxrnj\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n\" (UID: \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n"
Nov 01 03:25:55.880327 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:55.880311    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f4df86a-a104-4c26-9b8e-9b556b23afcf-workload-certs\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n\" (UID: \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n"
Nov 01 03:25:55.926078 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:55.926043    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n"]
Nov 01 03:25:55.981998 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:55.981963    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f4df86a-a104-4c26-9b8e-9b556b23afcf-istio-envoy\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n\" (UID: \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n"
Nov 01 03:25:55.982149 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:55.982008    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4df86a-a104-4c26-9b8e-9b556b23afcf-credential-socket\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n\" (UID: \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n"
Nov 01 03:25:55.982149 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:55.982030    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f4df86a-a104-4c26-9b8e-9b556b23afcf-istiod-ca-cert\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n\" (UID: \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n"
Nov 01 03:25:55.982149 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:55.982061    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxrnj\" (UniqueName: \"kubernetes.io/projected/7f4df86a-a104-4c26-9b8e-9b556b23afcf-kube-api-access-cxrnj\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n\" (UID: \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n"
Nov 01 03:25:55.982149 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:55.982098    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f4df86a-a104-4c26-9b8e-9b556b23afcf-workload-certs\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n\" (UID: \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n"
Nov 01 03:25:55.982149 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:55.982142    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f4df86a-a104-4c26-9b8e-9b556b23afcf-istio-token\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n\" (UID: \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n"
Nov 01 03:25:55.982429 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:55.982226    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f4df86a-a104-4c26-9b8e-9b556b23afcf-istio-data\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n\" (UID: \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n"
Nov 01 03:25:55.982429 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:55.982287    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4df86a-a104-4c26-9b8e-9b556b23afcf-workload-socket\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n\" (UID: \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n"
Nov 01 03:25:55.982429 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:55.982384    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f4df86a-a104-4c26-9b8e-9b556b23afcf-istio-podinfo\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n\" (UID: \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n"
Nov 01 03:25:55.982613 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:55.982527    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4df86a-a104-4c26-9b8e-9b556b23afcf-credential-socket\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n\" (UID: \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n"
Nov 01 03:25:55.982840 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:55.982813    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f4df86a-a104-4c26-9b8e-9b556b23afcf-istio-data\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n\" (UID: \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n"
Nov 01 03:25:55.982981 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:55.982852    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f4df86a-a104-4c26-9b8e-9b556b23afcf-workload-certs\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n\" (UID: \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n"
Nov 01 03:25:55.982981 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:55.982868    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4df86a-a104-4c26-9b8e-9b556b23afcf-workload-socket\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n\" (UID: \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n"
Nov 01 03:25:55.983072 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:55.983001    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f4df86a-a104-4c26-9b8e-9b556b23afcf-istiod-ca-cert\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n\" (UID: \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n"
Nov 01 03:25:55.984761 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:55.984738    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f4df86a-a104-4c26-9b8e-9b556b23afcf-istio-envoy\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n\" (UID: \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n"
Nov 01 03:25:55.984883 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:55.984866    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f4df86a-a104-4c26-9b8e-9b556b23afcf-istio-podinfo\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n\" (UID: \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n"
Nov 01 03:25:56.046134 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:56.046098    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxrnj\" (UniqueName: \"kubernetes.io/projected/7f4df86a-a104-4c26-9b8e-9b556b23afcf-kube-api-access-cxrnj\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n\" (UID: \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n"
Nov 01 03:25:56.046305 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:56.046145    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f4df86a-a104-4c26-9b8e-9b556b23afcf-istio-token\") pod \"e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n\" (UID: \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\") " pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n"
Nov 01 03:25:56.113753 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:56.113722    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n"
Nov 01 03:25:56.576885 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:56.576851    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n"]
Nov 01 03:25:56.577049 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:25:56.577028    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f4df86a_a104_4c26_9b8e_9b556b23afcf.slice/crio-a18676800dd2d3bc0cdeee4ba49e92b1b101a1d3328d9d757898cf1fdd6769d9 WatchSource:0}: Error finding container a18676800dd2d3bc0cdeee4ba49e92b1b101a1d3328d9d757898cf1fdd6769d9: Status 404 returned error can't find the container with id a18676800dd2d3bc0cdeee4ba49e92b1b101a1d3328d9d757898cf1fdd6769d9
Nov 01 03:25:56.579780 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:56.579740    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:25:56.579872 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:56.579854    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:25:56.579912 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:56.579901    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:25:56.674842 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:56.674812    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n" event={"ID":"7f4df86a-a104-4c26-9b8e-9b556b23afcf","Type":"ContainerStarted","Data":"179e253cd8413cfc95c8c24785d42b0588a3e9eaa91dc1d70d189a7512de9171"}
Nov 01 03:25:56.675236 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:56.674866    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n" event={"ID":"7f4df86a-a104-4c26-9b8e-9b556b23afcf","Type":"ContainerStarted","Data":"a18676800dd2d3bc0cdeee4ba49e92b1b101a1d3328d9d757898cf1fdd6769d9"}
Nov 01 03:25:56.803423 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:56.803391    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t"]
Nov 01 03:25:56.807814 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:56.807791    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t"
Nov 01 03:25:56.828551 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:56.828460    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n" podStartSLOduration=1.828447362 podStartE2EDuration="1.828447362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:25:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:25:56.8272085 +0000 UTC m=+1161.350215088" watchObservedRunningTime="2026-11-01 03:25:56.828447362 +0000 UTC m=+1161.351453951"
Nov 01 03:25:56.871801 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:56.871772    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t"]
Nov 01 03:25:56.899074 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:56.899026    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23ed3df4-075f-459b-be45-69da508ea513-istio-token\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t\" (UID: \"23ed3df4-075f-459b-be45-69da508ea513\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t"
Nov 01 03:25:56.899242 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:56.899107    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ed3df4-075f-459b-be45-69da508ea513-workload-certs\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t\" (UID: \"23ed3df4-075f-459b-be45-69da508ea513\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t"
Nov 01 03:25:56.899242 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:56.899157    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ed3df4-075f-459b-be45-69da508ea513-workload-socket\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t\" (UID: \"23ed3df4-075f-459b-be45-69da508ea513\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t"
Nov 01 03:25:56.899242 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:56.899203    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzkt4\" (UniqueName: \"kubernetes.io/projected/23ed3df4-075f-459b-be45-69da508ea513-kube-api-access-tzkt4\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t\" (UID: \"23ed3df4-075f-459b-be45-69da508ea513\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t"
Nov 01 03:25:56.899404 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:56.899302    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23ed3df4-075f-459b-be45-69da508ea513-istio-envoy\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t\" (UID: \"23ed3df4-075f-459b-be45-69da508ea513\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t"
Nov 01 03:25:56.899404 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:56.899328    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23ed3df4-075f-459b-be45-69da508ea513-istiod-ca-cert\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t\" (UID: \"23ed3df4-075f-459b-be45-69da508ea513\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t"
Nov 01 03:25:56.899404 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:56.899358    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23ed3df4-075f-459b-be45-69da508ea513-istio-data\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t\" (UID: \"23ed3df4-075f-459b-be45-69da508ea513\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t"
Nov 01 03:25:56.899713 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:56.899405    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ed3df4-075f-459b-be45-69da508ea513-credential-socket\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t\" (UID: \"23ed3df4-075f-459b-be45-69da508ea513\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t"
Nov 01 03:25:56.900436 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:56.900404    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23ed3df4-075f-459b-be45-69da508ea513-istio-podinfo\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t\" (UID: \"23ed3df4-075f-459b-be45-69da508ea513\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t"
Nov 01 03:25:57.001592 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:57.001527    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23ed3df4-075f-459b-be45-69da508ea513-istio-envoy\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t\" (UID: \"23ed3df4-075f-459b-be45-69da508ea513\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t"
Nov 01 03:25:57.001781 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:57.001632    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23ed3df4-075f-459b-be45-69da508ea513-istiod-ca-cert\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t\" (UID: \"23ed3df4-075f-459b-be45-69da508ea513\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t"
Nov 01 03:25:57.001781 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:57.001665    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23ed3df4-075f-459b-be45-69da508ea513-istio-data\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t\" (UID: \"23ed3df4-075f-459b-be45-69da508ea513\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t"
Nov 01 03:25:57.001781 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:57.001710    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ed3df4-075f-459b-be45-69da508ea513-credential-socket\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t\" (UID: \"23ed3df4-075f-459b-be45-69da508ea513\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t"
Nov 01 03:25:57.001957 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:57.001886    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23ed3df4-075f-459b-be45-69da508ea513-istio-podinfo\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t\" (UID: \"23ed3df4-075f-459b-be45-69da508ea513\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t"
Nov 01 03:25:57.001957 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:57.001929    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23ed3df4-075f-459b-be45-69da508ea513-istio-token\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t\" (UID: \"23ed3df4-075f-459b-be45-69da508ea513\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t"
Nov 01 03:25:57.002074 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:57.001988    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ed3df4-075f-459b-be45-69da508ea513-workload-certs\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t\" (UID: \"23ed3df4-075f-459b-be45-69da508ea513\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t"
Nov 01 03:25:57.002074 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:57.002034    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ed3df4-075f-459b-be45-69da508ea513-workload-socket\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t\" (UID: \"23ed3df4-075f-459b-be45-69da508ea513\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t"
Nov 01 03:25:57.002074 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:57.002070    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzkt4\" (UniqueName: \"kubernetes.io/projected/23ed3df4-075f-459b-be45-69da508ea513-kube-api-access-tzkt4\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t\" (UID: \"23ed3df4-075f-459b-be45-69da508ea513\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t"
Nov 01 03:25:57.002410 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:57.002383    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23ed3df4-075f-459b-be45-69da508ea513-istio-data\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t\" (UID: \"23ed3df4-075f-459b-be45-69da508ea513\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t"
Nov 01 03:25:57.002699 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:57.002675    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ed3df4-075f-459b-be45-69da508ea513-credential-socket\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t\" (UID: \"23ed3df4-075f-459b-be45-69da508ea513\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t"
Nov 01 03:25:57.003462 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:57.003422    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23ed3df4-075f-459b-be45-69da508ea513-istiod-ca-cert\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t\" (UID: \"23ed3df4-075f-459b-be45-69da508ea513\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t"
Nov 01 03:25:57.003592 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:57.003493    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ed3df4-075f-459b-be45-69da508ea513-workload-socket\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t\" (UID: \"23ed3df4-075f-459b-be45-69da508ea513\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t"
Nov 01 03:25:57.003719 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:57.003697    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ed3df4-075f-459b-be45-69da508ea513-workload-certs\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t\" (UID: \"23ed3df4-075f-459b-be45-69da508ea513\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t"
Nov 01 03:25:57.004355 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:57.004329    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23ed3df4-075f-459b-be45-69da508ea513-istio-envoy\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t\" (UID: \"23ed3df4-075f-459b-be45-69da508ea513\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t"
Nov 01 03:25:57.005262 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:57.005241    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23ed3df4-075f-459b-be45-69da508ea513-istio-podinfo\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t\" (UID: \"23ed3df4-075f-459b-be45-69da508ea513\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t"
Nov 01 03:25:57.063406 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:57.063375    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzkt4\" (UniqueName: \"kubernetes.io/projected/23ed3df4-075f-459b-be45-69da508ea513-kube-api-access-tzkt4\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t\" (UID: \"23ed3df4-075f-459b-be45-69da508ea513\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t"
Nov 01 03:25:57.064016 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:57.063996    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23ed3df4-075f-459b-be45-69da508ea513-istio-token\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t\" (UID: \"23ed3df4-075f-459b-be45-69da508ea513\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t"
Nov 01 03:25:57.114276 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:57.114238    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n"
Nov 01 03:25:57.120070 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:57.120039    4930 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 01 03:25:57.120230 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:57.120097    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n" podUID="7f4df86a-a104-4c26-9b8e-9b556b23afcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 01 03:25:57.123673 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:57.123293    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t"
Nov 01 03:25:57.596437 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:57.596405    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t"]
Nov 01 03:25:57.598830 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:25:57.598803    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23ed3df4_075f_459b_be45_69da508ea513.slice/crio-5667d2e9ab92d40e137bf5e30c9dc66f280425826b44cbdac6a4186789b572a6 WatchSource:0}: Error finding container 5667d2e9ab92d40e137bf5e30c9dc66f280425826b44cbdac6a4186789b572a6: Status 404 returned error can't find the container with id 5667d2e9ab92d40e137bf5e30c9dc66f280425826b44cbdac6a4186789b572a6
Nov 01 03:25:57.601922 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:57.601885    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:25:57.602038 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:57.601973    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:25:57.602038 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:57.602019    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:25:57.695708 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:57.695674    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t" event={"ID":"23ed3df4-075f-459b-be45-69da508ea513","Type":"ContainerStarted","Data":"5667d2e9ab92d40e137bf5e30c9dc66f280425826b44cbdac6a4186789b572a6"}
Nov 01 03:25:58.115089 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:58.115056    4930 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 01 03:25:58.115269 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:58.115115    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n" podUID="7f4df86a-a104-4c26-9b8e-9b556b23afcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 01 03:25:58.641019 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:58.640989    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:25:58.758450 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:58.758407    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t" event={"ID":"23ed3df4-075f-459b-be45-69da508ea513","Type":"ContainerStarted","Data":"b2780b594982117c704a8352f5ac248d50a664e116cbd6c9c3e888b35d2b2082"}
Nov 01 03:25:59.014896 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:59.014788    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t" podStartSLOduration=3.014769765 podStartE2EDuration="3.014769765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:25:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:25:58.948548539 +0000 UTC m=+1163.471555120" watchObservedRunningTime="2026-11-01 03:25:59.014769765 +0000 UTC m=+1163.537776723"
Nov 01 03:25:59.114200 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:59.114163    4930 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 01 03:25:59.115757 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:59.115670    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n" podUID="7f4df86a-a104-4c26-9b8e-9b556b23afcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 01 03:25:59.125254 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:59.124501    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t"
Nov 01 03:25:59.126476 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:59.126452    4930 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 01 03:25:59.126609 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:25:59.126492    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t" podUID="23ed3df4-075f-459b-be45-69da508ea513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 01 03:26:00.119581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:00.117890    4930 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 01 03:26:00.119581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:00.117964    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n" podUID="7f4df86a-a104-4c26-9b8e-9b556b23afcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 01 03:26:00.124835 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:00.124808    4930 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 01 03:26:00.124949 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:00.124852    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t" podUID="23ed3df4-075f-459b-be45-69da508ea513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 01 03:26:01.120794 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:01.118901    4930 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 01 03:26:01.120794 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:01.118962    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n" podUID="7f4df86a-a104-4c26-9b8e-9b556b23afcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 01 03:26:01.124079 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:01.124041    4930 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 01 03:26:01.124242 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:01.124123    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t" podUID="23ed3df4-075f-459b-be45-69da508ea513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 01 03:26:02.115039 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:02.115002    4930 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 01 03:26:02.115312 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:02.115279    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n" podUID="7f4df86a-a104-4c26-9b8e-9b556b23afcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 01 03:26:02.125175 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:02.124957    4930 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 01 03:26:02.125175 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:02.125014    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t" podUID="23ed3df4-075f-459b-be45-69da508ea513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 01 03:26:03.114391 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:03.114351    4930 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 01 03:26:03.114617 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:03.114417    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n" podUID="7f4df86a-a104-4c26-9b8e-9b556b23afcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 01 03:26:03.124693 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:03.124668    4930 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 01 03:26:03.124823 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:03.124708    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t" podUID="23ed3df4-075f-459b-be45-69da508ea513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 01 03:26:03.631105 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:03.631065    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:26:04.125002 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:04.124946    4930 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Nov 01 03:26:04.125198 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:04.125022    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t" podUID="23ed3df4-075f-459b-be45-69da508ea513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Nov 01 03:26:04.235089 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:04.232739    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n"
Nov 01 03:26:04.235089 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:04.233471    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n"
Nov 01 03:26:04.235515 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:04.235495    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n"
Nov 01 03:26:04.520044 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:04.519945    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:26:05.218767 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:05.218726    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t"
Nov 01 03:26:05.219965 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:05.219942    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t"
Nov 01 03:26:05.220538 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:05.220516    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t"
Nov 01 03:26:08.448935 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:08.448890    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:26:09.377118 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:09.377078    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e202228-gh4tp"]
Nov 01 03:26:09.383457 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:09.383409    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e202228-gh4tp"
Nov 01 03:26:09.441928 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:09.441873    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e202228-gh4tp"]
Nov 01 03:26:09.482707 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:09.482675    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:26:09.483267 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:09.483246    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:26:09.483498 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:09.483480    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:26:09.483722 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:09.483706    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:26:09.521584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:09.505435    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxxlr\" (UniqueName: \"kubernetes.io/projected/eedbb6ba-9513-45df-804b-58a710c0f294-kube-api-access-cxxlr\") pod \"maas-api-revoke-sub-4e202228-gh4tp\" (UID: \"eedbb6ba-9513-45df-804b-58a710c0f294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e202228-gh4tp"
Nov 01 03:26:09.551687 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:09.551643    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eedbb6ba-9513-45df-804b-58a710c0f294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e202228-gh4tp\" (UID: \"eedbb6ba-9513-45df-804b-58a710c0f294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e202228-gh4tp"
Nov 01 03:26:09.552116 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:09.552082    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eedbb6ba-9513-45df-804b-58a710c0f294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e202228-gh4tp\" (UID: \"eedbb6ba-9513-45df-804b-58a710c0f294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e202228-gh4tp"
Nov 01 03:26:09.656461 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:09.656368    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eedbb6ba-9513-45df-804b-58a710c0f294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e202228-gh4tp\" (UID: \"eedbb6ba-9513-45df-804b-58a710c0f294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e202228-gh4tp"
Nov 01 03:26:09.678585 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:09.678287    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eedbb6ba-9513-45df-804b-58a710c0f294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e202228-gh4tp\" (UID: \"eedbb6ba-9513-45df-804b-58a710c0f294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e202228-gh4tp"
Nov 01 03:26:09.678957 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:09.678931    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eedbb6ba-9513-45df-804b-58a710c0f294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e202228-gh4tp\" (UID: \"eedbb6ba-9513-45df-804b-58a710c0f294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e202228-gh4tp"
Nov 01 03:26:09.679372 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:09.679348    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxxlr\" (UniqueName: \"kubernetes.io/projected/eedbb6ba-9513-45df-804b-58a710c0f294-kube-api-access-cxxlr\") pod \"maas-api-revoke-sub-4e202228-gh4tp\" (UID: \"eedbb6ba-9513-45df-804b-58a710c0f294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e202228-gh4tp"
Nov 01 03:26:09.682672 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:09.682640    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eedbb6ba-9513-45df-804b-58a710c0f294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e202228-gh4tp\" (UID: \"eedbb6ba-9513-45df-804b-58a710c0f294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e202228-gh4tp"
Nov 01 03:26:09.748152 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:09.748115    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxxlr\" (UniqueName: \"kubernetes.io/projected/eedbb6ba-9513-45df-804b-58a710c0f294-kube-api-access-cxxlr\") pod \"maas-api-revoke-sub-4e202228-gh4tp\" (UID: \"eedbb6ba-9513-45df-804b-58a710c0f294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e202228-gh4tp"
Nov 01 03:26:10.008632 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:10.007707    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e202228-gh4tp"
Nov 01 03:26:10.055265 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:10.055229    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2361ef33-v6j2n"]
Nov 01 03:26:10.114575 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:10.114170    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2361ef33-v6j2n"]
Nov 01 03:26:10.767233 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:10.767195    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e202228-gh4tp"]
Nov 01 03:26:10.773934 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:26:10.773901    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeedbb6ba_9513_45df_804b_58a710c0f294.slice/crio-1cc1aab162edd4a1f094f0d219e5db7f9dde693cbc5f413c952dffb3874b43d1 WatchSource:0}: Error finding container 1cc1aab162edd4a1f094f0d219e5db7f9dde693cbc5f413c952dffb3874b43d1: Status 404 returned error can't find the container with id 1cc1aab162edd4a1f094f0d219e5db7f9dde693cbc5f413c952dffb3874b43d1
Nov 01 03:26:11.015429 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:11.015400    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e202228-gh4tp" event={"ID":"eedbb6ba-9513-45df-804b-58a710c0f294","Type":"ContainerStarted","Data":"1cc1aab162edd4a1f094f0d219e5db7f9dde693cbc5f413c952dffb3874b43d1"}
Nov 01 03:26:11.621296 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:11.621264    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f875872-7339-495a-8d7b-9462041bcec4" path="/var/lib/kubelet/pods/4f875872-7339-495a-8d7b-9462041bcec4/volumes"
Nov 01 03:26:12.024046 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:12.024015    4930 generic.go:356] "Generic (PLEG): container finished" podID="eedbb6ba-9513-45df-804b-58a710c0f294" containerID="63346684a0712acb2706b4cfa16013fff396dda94465ed609e094b2e1cf9bd4a" exitCode=0
Nov 01 03:26:12.024399 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:12.024097    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e202228-gh4tp" event={"ID":"eedbb6ba-9513-45df-804b-58a710c0f294","Type":"ContainerDied","Data":"63346684a0712acb2706b4cfa16013fff396dda94465ed609e094b2e1cf9bd4a"}
Nov 01 03:26:13.194756 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:13.194730    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e202228-gh4tp"
Nov 01 03:26:13.261917 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:13.261889    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eedbb6ba-9513-45df-804b-58a710c0f294-kube-api-access-cxxlr\" (UniqueName: \"kubernetes.io/projected/eedbb6ba-9513-45df-804b-58a710c0f294-kube-api-access-cxxlr\") pod \"eedbb6ba-9513-45df-804b-58a710c0f294\" (UID: \"eedbb6ba-9513-45df-804b-58a710c0f294\") "
Nov 01 03:26:13.262033 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:13.261934    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eedbb6ba-9513-45df-804b-58a710c0f294-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eedbb6ba-9513-45df-804b-58a710c0f294-maas-api-openshift-ca\") pod \"eedbb6ba-9513-45df-804b-58a710c0f294\" (UID: \"eedbb6ba-9513-45df-804b-58a710c0f294\") "
Nov 01 03:26:13.262085 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:13.262065    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eedbb6ba-9513-45df-804b-58a710c0f294-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eedbb6ba-9513-45df-804b-58a710c0f294-maas-api-xks-ca\") pod \"eedbb6ba-9513-45df-804b-58a710c0f294\" (UID: \"eedbb6ba-9513-45df-804b-58a710c0f294\") "
Nov 01 03:26:13.262380 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:13.262347    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eedbb6ba-9513-45df-804b-58a710c0f294-maas-api-openshift-ca" pod "eedbb6ba-9513-45df-804b-58a710c0f294" (UID: "eedbb6ba-9513-45df-804b-58a710c0f294"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:26:13.262752 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:13.262730    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eedbb6ba-9513-45df-804b-58a710c0f294-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:26:13.264008 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:13.263983    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eedbb6ba-9513-45df-804b-58a710c0f294-maas-api-xks-ca" pod "eedbb6ba-9513-45df-804b-58a710c0f294" (UID: "eedbb6ba-9513-45df-804b-58a710c0f294"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:26:13.264117 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:13.264099    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eedbb6ba-9513-45df-804b-58a710c0f294-kube-api-access-cxxlr" pod "eedbb6ba-9513-45df-804b-58a710c0f294" (UID: "eedbb6ba-9513-45df-804b-58a710c0f294"). InnerVolumeSpecName "kube-api-access-cxxlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:26:13.329471 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:13.329395    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:26:13.364192 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:13.364162    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxxlr\" (UniqueName: \"kubernetes.io/projected/eedbb6ba-9513-45df-804b-58a710c0f294-kube-api-access-cxxlr\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:26:13.364192 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:13.364196    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eedbb6ba-9513-45df-804b-58a710c0f294-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:26:14.042149 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:14.042074    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e202228-gh4tp"
Nov 01 03:26:14.042292 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:14.042073    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e202228-gh4tp" event={"ID":"eedbb6ba-9513-45df-804b-58a710c0f294","Type":"ContainerDied","Data":"1cc1aab162edd4a1f094f0d219e5db7f9dde693cbc5f413c952dffb3874b43d1"}
Nov 01 03:26:14.042292 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:14.042255    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cc1aab162edd4a1f094f0d219e5db7f9dde693cbc5f413c952dffb3874b43d1"
Nov 01 03:26:15.414646 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:15.414594    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:26:18.146251 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:18.146214    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:26:19.122308 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:26:19.122262    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:26:20.206248 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:20.206209    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"]
Nov 01 03:26:20.206877 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:20.206857    4930 cpu_manager.go:414] "containerMap: removing container" podUID="eedbb6ba-9513-45df-804b-58a710c0f294" containerName="revoke-keys"
Nov 01 03:26:20.206877 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:20.206877    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eedbb6ba-9513-45df-804b-58a710c0f294" containerName="revoke-keys"
Nov 01 03:26:20.207019 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:20.207009    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="eedbb6ba-9513-45df-804b-58a710c0f294" containerName="revoke-keys"
Nov 01 03:26:20.210832 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:20.210809    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"
Nov 01 03:26:20.245945 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:20.245903    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2483a06-2f4b-4a1e-af4e-a63eb50d29bf-tls-certs\") pod \"distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2\" (UID: \"e2483a06-2f4b-4a1e-af4e-a63eb50d29bf\") " pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"
Nov 01 03:26:20.246093 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:20.245970    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2483a06-2f4b-4a1e-af4e-a63eb50d29bf-dshm\") pod \"distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2\" (UID: \"e2483a06-2f4b-4a1e-af4e-a63eb50d29bf\") " pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"
Nov 01 03:26:20.246093 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:20.246024    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qtlb\" (UniqueName: \"kubernetes.io/projected/e2483a06-2f4b-4a1e-af4e-a63eb50d29bf-kube-api-access-8qtlb\") pod \"distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2\" (UID: \"e2483a06-2f4b-4a1e-af4e-a63eb50d29bf\") " pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"
Nov 01 03:26:20.246217 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:20.246091    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2483a06-2f4b-4a1e-af4e-a63eb50d29bf-tmp-dir\") pod \"distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2\" (UID: \"e2483a06-2f4b-4a1e-af4e-a63eb50d29bf\") " pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"
Nov 01 03:26:20.246217 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:20.246120    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2483a06-2f4b-4a1e-af4e-a63eb50d29bf-model-cache\") pod \"distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2\" (UID: \"e2483a06-2f4b-4a1e-af4e-a63eb50d29bf\") " pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"
Nov 01 03:26:20.246326 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:20.246253    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2483a06-2f4b-4a1e-af4e-a63eb50d29bf-home\") pod \"distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2\" (UID: \"e2483a06-2f4b-4a1e-af4e-a63eb50d29bf\") " pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"
Nov 01 03:26:20.273284 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:20.273256    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"]
Nov 01 03:26:20.295345 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:20.295315    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:26:20.314542 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:20.314515    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-c22dd8\"/\"distinct-2-w5-c22dd8-kserve-self-signed-certs\""
Nov 01 03:26:20.347374 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:20.347347    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2483a06-2f4b-4a1e-af4e-a63eb50d29bf-tls-certs\") pod \"distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2\" (UID: \"e2483a06-2f4b-4a1e-af4e-a63eb50d29bf\") " pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"
Nov 01 03:26:20.347508 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:20.347395    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2483a06-2f4b-4a1e-af4e-a63eb50d29bf-dshm\") pod \"distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2\" (UID: \"e2483a06-2f4b-4a1e-af4e-a63eb50d29bf\") " pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"
Nov 01 03:26:20.347508 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:20.347428    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qtlb\" (UniqueName: \"kubernetes.io/projected/e2483a06-2f4b-4a1e-af4e-a63eb50d29bf-kube-api-access-8qtlb\") pod \"distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2\" (UID: \"e2483a06-2f4b-4a1e-af4e-a63eb50d29bf\") " pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"
Nov 01 03:26:20.347508 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:20.347473    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2483a06-2f4b-4a1e-af4e-a63eb50d29bf-tmp-dir\") pod \"distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2\" (UID: \"e2483a06-2f4b-4a1e-af4e-a63eb50d29bf\") " pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"
Nov 01 03:26:20.347508 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:20.347498    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2483a06-2f4b-4a1e-af4e-a63eb50d29bf-model-cache\") pod \"distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2\" (UID: \"e2483a06-2f4b-4a1e-af4e-a63eb50d29bf\") " pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"
Nov 01 03:26:20.347763 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:20.347645    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2483a06-2f4b-4a1e-af4e-a63eb50d29bf-home\") pod \"distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2\" (UID: \"e2483a06-2f4b-4a1e-af4e-a63eb50d29bf\") " pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"
Nov 01 03:26:20.347966 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:20.347941    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2483a06-2f4b-4a1e-af4e-a63eb50d29bf-model-cache\") pod \"distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2\" (UID: \"e2483a06-2f4b-4a1e-af4e-a63eb50d29bf\") " pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"
Nov 01 03:26:20.348056 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:20.348023    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2483a06-2f4b-4a1e-af4e-a63eb50d29bf-tmp-dir\") pod \"distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2\" (UID: \"e2483a06-2f4b-4a1e-af4e-a63eb50d29bf\") " pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"
Nov 01 03:26:20.348154 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:20.348137    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2483a06-2f4b-4a1e-af4e-a63eb50d29bf-home\") pod \"distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2\" (UID: \"e2483a06-2f4b-4a1e-af4e-a63eb50d29bf\") " pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"
Nov 01 03:26:20.349859 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:20.349838    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2483a06-2f4b-4a1e-af4e-a63eb50d29bf-dshm\") pod \"distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2\" (UID: \"e2483a06-2f4b-4a1e-af4e-a63eb50d29bf\") " pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"
Nov 01 03:26:20.350055 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:20.350040    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2483a06-2f4b-4a1e-af4e-a63eb50d29bf-tls-certs\") pod \"distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2\" (UID: \"e2483a06-2f4b-4a1e-af4e-a63eb50d29bf\") " pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"
Nov 01 03:26:20.408356 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:20.408323    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qtlb\" (UniqueName: \"kubernetes.io/projected/e2483a06-2f4b-4a1e-af4e-a63eb50d29bf-kube-api-access-8qtlb\") pod \"distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2\" (UID: \"e2483a06-2f4b-4a1e-af4e-a63eb50d29bf\") " pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"
Nov 01 03:26:20.523546 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:20.523465    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"
Nov 01 03:26:20.962618 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:26:20.962578    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2483a06_2f4b_4a1e_af4e_a63eb50d29bf.slice/crio-b6351225d45fceaa736af763b252889d2783faef265e49f2e9869686ddc2b791 WatchSource:0}: Error finding container b6351225d45fceaa736af763b252889d2783faef265e49f2e9869686ddc2b791: Status 404 returned error can't find the container with id b6351225d45fceaa736af763b252889d2783faef265e49f2e9869686ddc2b791
Nov 01 03:26:20.962755 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:20.962685    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"]
Nov 01 03:26:21.105214 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:21.105172    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2" event={"ID":"e2483a06-2f4b-4a1e-af4e-a63eb50d29bf","Type":"ContainerStarted","Data":"3dbdeb37dc708acde94d3845b87620f01fe594185a9c10e2eea1d6721e1c535c"}
Nov 01 03:26:21.105372 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:21.105219    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2" event={"ID":"e2483a06-2f4b-4a1e-af4e-a63eb50d29bf","Type":"ContainerStarted","Data":"b6351225d45fceaa736af763b252889d2783faef265e49f2e9869686ddc2b791"}
Nov 01 03:26:21.225671 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:21.225550    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2" podStartSLOduration=1.225531965 podStartE2EDuration="1.225531965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:26:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:26:21.221828853 +0000 UTC m=+1185.744835442" watchObservedRunningTime="2026-11-01 03:26:21.225531965 +0000 UTC m=+1185.748538553"
Nov 01 03:26:21.523951 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:21.523869    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"
Nov 01 03:26:22.949469 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:22.949430    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:26:23.971235 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:23.971193    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:26:24.994751 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:24.994703    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fffcf7a5-qjs29"]
Nov 01 03:26:25.006873 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:25.006840    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffcf7a5-qjs29"
Nov 01 03:26:25.122987 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:25.122951    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:26:25.123278 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:25.123260    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:26:25.123574 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:25.123539    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:26:25.123935 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:25.123908    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:26:25.201699 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:25.201640    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fffcf7a5-qjs29"]
Nov 01 03:26:25.203863 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:25.203833    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fffcf7a5-qjs29\" (UID: \"f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffcf7a5-qjs29"
Nov 01 03:26:25.204181 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:25.204157    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsv69\" (UniqueName: \"kubernetes.io/projected/f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8-kube-api-access-xsv69\") pod \"maas-api-revoke-sub-fffcf7a5-qjs29\" (UID: \"f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffcf7a5-qjs29"
Nov 01 03:26:25.218578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:25.204329    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fffcf7a5-qjs29\" (UID: \"f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffcf7a5-qjs29"
Nov 01 03:26:25.320665 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:25.320535    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsv69\" (UniqueName: \"kubernetes.io/projected/f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8-kube-api-access-xsv69\") pod \"maas-api-revoke-sub-fffcf7a5-qjs29\" (UID: \"f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffcf7a5-qjs29"
Nov 01 03:26:25.320996 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:25.320971    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fffcf7a5-qjs29\" (UID: \"f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffcf7a5-qjs29"
Nov 01 03:26:25.321372 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:25.321350    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fffcf7a5-qjs29\" (UID: \"f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffcf7a5-qjs29"
Nov 01 03:26:25.323408 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:25.323382    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fffcf7a5-qjs29\" (UID: \"f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffcf7a5-qjs29"
Nov 01 03:26:25.404582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:25.392754    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fffcf7a5-qjs29\" (UID: \"f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffcf7a5-qjs29"
Nov 01 03:26:25.417580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:25.415519    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsv69\" (UniqueName: \"kubernetes.io/projected/f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8-kube-api-access-xsv69\") pod \"maas-api-revoke-sub-fffcf7a5-qjs29\" (UID: \"f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffcf7a5-qjs29"
Nov 01 03:26:25.645938 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:25.645900    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffcf7a5-qjs29"
Nov 01 03:26:26.151027 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:26.139376    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n" podUID="d2511772-66f9-40a0-9166-87c4345485c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:26:26.307708 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:26.307657    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm" podUID="75d142b3-7172-4b57-bc07-a2f404c4656e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:26:26.459633 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:26.456436    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp" podUID="667240b0-64a7-45cd-84fe-9cd7e0d8a301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:26:26.854687 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:26:26.854653    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf14dfb49_9ee5_4d7d_a32b_c9f8b6d6b2e8.slice/crio-8bd1dfdabf5d427f2b37fb096dd3e1f23ef61cda9c8258fd5eff0cc9ac152e92 WatchSource:0}: Error finding container 8bd1dfdabf5d427f2b37fb096dd3e1f23ef61cda9c8258fd5eff0cc9ac152e92: Status 404 returned error can't find the container with id 8bd1dfdabf5d427f2b37fb096dd3e1f23ef61cda9c8258fd5eff0cc9ac152e92
Nov 01 03:26:26.854828 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:26.854703    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fffcf7a5-qjs29"]
Nov 01 03:26:27.194765 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:27.194736    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffcf7a5-qjs29" event={"ID":"f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8","Type":"ContainerStarted","Data":"3d286c6c758c527acdba11ced454de710d99b0b9d9a7b726b937875c5bd683b5"}
Nov 01 03:26:27.195104 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:27.194793    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffcf7a5-qjs29" event={"ID":"f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8","Type":"ContainerStarted","Data":"8bd1dfdabf5d427f2b37fb096dd3e1f23ef61cda9c8258fd5eff0cc9ac152e92"}
Nov 01 03:26:27.313494 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:27.313440    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffcf7a5-qjs29" podStartSLOduration=3.313424636 podStartE2EDuration="3.313424636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:26:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:26:27.312222177 +0000 UTC m=+1191.835228768" watchObservedRunningTime="2026-11-01 03:26:27.313424636 +0000 UTC m=+1191.836431224"
Nov 01 03:26:28.150416 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:28.150376    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:26:28.219808 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:28.219766    4930 generic.go:356] "Generic (PLEG): container finished" podID="f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8" containerID="3d286c6c758c527acdba11ced454de710d99b0b9d9a7b726b937875c5bd683b5" exitCode=0
Nov 01 03:26:28.220863 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:28.220812    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffcf7a5-qjs29" event={"ID":"f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8","Type":"ContainerDied","Data":"3d286c6c758c527acdba11ced454de710d99b0b9d9a7b726b937875c5bd683b5"}
Nov 01 03:26:29.395594 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:29.395550    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffcf7a5-qjs29"
Nov 01 03:26:29.413362 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:29.413335    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8-maas-api-xks-ca\") pod \"f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8\" (UID: \"f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8\") "
Nov 01 03:26:29.413453 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:29.413368    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8-kube-api-access-xsv69\" (UniqueName: \"kubernetes.io/projected/f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8-kube-api-access-xsv69\") pod \"f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8\" (UID: \"f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8\") "
Nov 01 03:26:29.413540 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:29.413525    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8-maas-api-openshift-ca\") pod \"f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8\" (UID: \"f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8\") "
Nov 01 03:26:29.413927 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:29.413894    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8-maas-api-openshift-ca" pod "f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8" (UID: "f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:26:29.414116 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:29.414095    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:26:29.415492 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:29.415468    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8-maas-api-xks-ca" pod "f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8" (UID: "f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:26:29.415702 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:29.415674    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8-kube-api-access-xsv69" pod "f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8" (UID: "f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8"). InnerVolumeSpecName "kube-api-access-xsv69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:26:29.518192 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:29.515221    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:26:29.518192 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:29.515257    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsv69\" (UniqueName: \"kubernetes.io/projected/f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8-kube-api-access-xsv69\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:26:30.240108 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:30.240078    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffcf7a5-qjs29"
Nov 01 03:26:30.240297 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:30.240124    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fffcf7a5-qjs29" event={"ID":"f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8","Type":"ContainerDied","Data":"8bd1dfdabf5d427f2b37fb096dd3e1f23ef61cda9c8258fd5eff0cc9ac152e92"}
Nov 01 03:26:30.240297 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:30.240185    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bd1dfdabf5d427f2b37fb096dd3e1f23ef61cda9c8258fd5eff0cc9ac152e92"
Nov 01 03:26:30.523673 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:30.523589    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"
Nov 01 03:26:30.536894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:30.536869    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"
Nov 01 03:26:31.052587 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:31.052532    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b908441b-vzd9t"]
Nov 01 03:26:31.056988 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:31.056959    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b908441b-vzd9t"]
Nov 01 03:26:31.258655 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:31.258621    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"
Nov 01 03:26:31.621380 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:31.621348    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21c3e8ae-f194-47f8-a35a-25e8382a41db" path="/var/lib/kubelet/pods/21c3e8ae-f194-47f8-a35a-25e8382a41db/volumes"
Nov 01 03:26:32.412221 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:32.412188    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:26:33.046168 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:33.046130    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:26:33.838598 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:33.838542    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:26:35.825053 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:35.825024    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jh955_f29ea61e-2a78-4c11-9566-f52497e135ff/console-operator/2.log"
Nov 01 03:26:35.827480 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:35.827456    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jh955_f29ea61e-2a78-4c11-9566-f52497e135ff/console-operator/2.log"
Nov 01 03:26:35.875479 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:35.875451    4930 scope.go:122] "RemoveContainer" containerID="d36c050b48c6294efbac6821cbff14e708a028d13888617f07533a3caa4ae7fa"
Nov 01 03:26:37.830786 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:37.830743    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:26:39.250318 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:39.250269    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:26:39.412011 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:39.411969    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:26:40.771642 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:40.771601    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q"]
Nov 01 03:26:40.772212 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:40.772195    4930 cpu_manager.go:414] "containerMap: removing container" podUID="f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8" containerName="revoke-keys"
Nov 01 03:26:40.772212 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:40.772211    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8" containerName="revoke-keys"
Nov 01 03:26:40.772340 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:40.772327    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8" containerName="revoke-keys"
Nov 01 03:26:40.777598 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:40.777576    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q"
Nov 01 03:26:40.891325 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:40.891284    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ff8v\" (UniqueName: \"kubernetes.io/projected/12d11ff3-9497-498d-8aeb-7f70f1874383-kube-api-access-2ff8v\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q\" (UID: \"12d11ff3-9497-498d-8aeb-7f70f1874383\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q"
Nov 01 03:26:40.891645 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:40.891617    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12d11ff3-9497-498d-8aeb-7f70f1874383-istio-envoy\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q\" (UID: \"12d11ff3-9497-498d-8aeb-7f70f1874383\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q"
Nov 01 03:26:40.891815 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:40.891797    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12d11ff3-9497-498d-8aeb-7f70f1874383-istio-podinfo\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q\" (UID: \"12d11ff3-9497-498d-8aeb-7f70f1874383\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q"
Nov 01 03:26:40.891942 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:40.891926    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d11ff3-9497-498d-8aeb-7f70f1874383-credential-socket\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q\" (UID: \"12d11ff3-9497-498d-8aeb-7f70f1874383\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q"
Nov 01 03:26:40.892094 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:40.892078    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12d11ff3-9497-498d-8aeb-7f70f1874383-istio-data\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q\" (UID: \"12d11ff3-9497-498d-8aeb-7f70f1874383\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q"
Nov 01 03:26:40.892328 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:40.892291    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d11ff3-9497-498d-8aeb-7f70f1874383-workload-certs\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q\" (UID: \"12d11ff3-9497-498d-8aeb-7f70f1874383\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q"
Nov 01 03:26:40.892462 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:40.892444    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12d11ff3-9497-498d-8aeb-7f70f1874383-istio-token\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q\" (UID: \"12d11ff3-9497-498d-8aeb-7f70f1874383\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q"
Nov 01 03:26:40.892613 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:40.892595    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d11ff3-9497-498d-8aeb-7f70f1874383-workload-socket\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q\" (UID: \"12d11ff3-9497-498d-8aeb-7f70f1874383\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q"
Nov 01 03:26:40.892812 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:40.892793    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12d11ff3-9497-498d-8aeb-7f70f1874383-istiod-ca-cert\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q\" (UID: \"12d11ff3-9497-498d-8aeb-7f70f1874383\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q"
Nov 01 03:26:40.897968 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:40.897940    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q"]
Nov 01 03:26:40.994140 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:40.994100    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ff8v\" (UniqueName: \"kubernetes.io/projected/12d11ff3-9497-498d-8aeb-7f70f1874383-kube-api-access-2ff8v\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q\" (UID: \"12d11ff3-9497-498d-8aeb-7f70f1874383\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q"
Nov 01 03:26:40.994140 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:40.994148    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12d11ff3-9497-498d-8aeb-7f70f1874383-istio-envoy\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q\" (UID: \"12d11ff3-9497-498d-8aeb-7f70f1874383\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q"
Nov 01 03:26:40.994455 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:40.994170    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12d11ff3-9497-498d-8aeb-7f70f1874383-istio-podinfo\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q\" (UID: \"12d11ff3-9497-498d-8aeb-7f70f1874383\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q"
Nov 01 03:26:40.994455 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:40.994207    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d11ff3-9497-498d-8aeb-7f70f1874383-credential-socket\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q\" (UID: \"12d11ff3-9497-498d-8aeb-7f70f1874383\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q"
Nov 01 03:26:40.994455 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:40.994258    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12d11ff3-9497-498d-8aeb-7f70f1874383-istio-data\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q\" (UID: \"12d11ff3-9497-498d-8aeb-7f70f1874383\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q"
Nov 01 03:26:40.994455 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:40.994363    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d11ff3-9497-498d-8aeb-7f70f1874383-workload-certs\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q\" (UID: \"12d11ff3-9497-498d-8aeb-7f70f1874383\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q"
Nov 01 03:26:40.994455 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:40.994394    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12d11ff3-9497-498d-8aeb-7f70f1874383-istio-token\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q\" (UID: \"12d11ff3-9497-498d-8aeb-7f70f1874383\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q"
Nov 01 03:26:40.994455 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:40.994424    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d11ff3-9497-498d-8aeb-7f70f1874383-workload-socket\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q\" (UID: \"12d11ff3-9497-498d-8aeb-7f70f1874383\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q"
Nov 01 03:26:40.994806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:40.994513    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12d11ff3-9497-498d-8aeb-7f70f1874383-istiod-ca-cert\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q\" (UID: \"12d11ff3-9497-498d-8aeb-7f70f1874383\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q"
Nov 01 03:26:40.994806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:40.994709    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d11ff3-9497-498d-8aeb-7f70f1874383-credential-socket\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q\" (UID: \"12d11ff3-9497-498d-8aeb-7f70f1874383\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q"
Nov 01 03:26:40.994912 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:40.994821    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d11ff3-9497-498d-8aeb-7f70f1874383-workload-certs\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q\" (UID: \"12d11ff3-9497-498d-8aeb-7f70f1874383\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q"
Nov 01 03:26:40.995075 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:40.995048    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d11ff3-9497-498d-8aeb-7f70f1874383-workload-socket\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q\" (UID: \"12d11ff3-9497-498d-8aeb-7f70f1874383\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q"
Nov 01 03:26:40.995484 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:40.995459    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12d11ff3-9497-498d-8aeb-7f70f1874383-istio-data\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q\" (UID: \"12d11ff3-9497-498d-8aeb-7f70f1874383\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q"
Nov 01 03:26:40.995818 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:40.995800    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12d11ff3-9497-498d-8aeb-7f70f1874383-istiod-ca-cert\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q\" (UID: \"12d11ff3-9497-498d-8aeb-7f70f1874383\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q"
Nov 01 03:26:40.997127 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:40.997095    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12d11ff3-9497-498d-8aeb-7f70f1874383-istio-envoy\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q\" (UID: \"12d11ff3-9497-498d-8aeb-7f70f1874383\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q"
Nov 01 03:26:40.997718 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:40.997693    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12d11ff3-9497-498d-8aeb-7f70f1874383-istio-podinfo\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q\" (UID: \"12d11ff3-9497-498d-8aeb-7f70f1874383\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q"
Nov 01 03:26:41.052647 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:41.052514    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0189dc42-rk8p8"]
Nov 01 03:26:41.056452 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:41.056424    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12d11ff3-9497-498d-8aeb-7f70f1874383-istio-token\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q\" (UID: \"12d11ff3-9497-498d-8aeb-7f70f1874383\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q"
Nov 01 03:26:41.056676 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:41.056656    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ff8v\" (UniqueName: \"kubernetes.io/projected/12d11ff3-9497-498d-8aeb-7f70f1874383-kube-api-access-2ff8v\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q\" (UID: \"12d11ff3-9497-498d-8aeb-7f70f1874383\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q"
Nov 01 03:26:41.057166 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:41.057147    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0189dc42-rk8p8"]
Nov 01 03:26:41.091810 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:41.091777    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q"
Nov 01 03:26:41.530988 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:26:41.530954    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12d11ff3_9497_498d_8aeb_7f70f1874383.slice/crio-f08bd3aea28a609c8cc94c8803410df69cb218a9089a1c7510b3f1282dc47cb7 WatchSource:0}: Error finding container f08bd3aea28a609c8cc94c8803410df69cb218a9089a1c7510b3f1282dc47cb7: Status 404 returned error can't find the container with id f08bd3aea28a609c8cc94c8803410df69cb218a9089a1c7510b3f1282dc47cb7
Nov 01 03:26:41.531154 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:41.531126    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q"]
Nov 01 03:26:41.533746 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:41.533717    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:26:41.533841 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:41.533783    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:26:41.533841 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:41.533815    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:26:41.624337 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:41.624311    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d" path="/var/lib/kubelet/pods/c2dc4dbf-ea70-4bf5-8693-9de7ea1ba33d/volumes"
Nov 01 03:26:42.391578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:42.390851    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q" event={"ID":"12d11ff3-9497-498d-8aeb-7f70f1874383","Type":"ContainerStarted","Data":"563b9115b7f7e666b6b16ccdff4721b02322eb406b1ed65d7a5fbc7ec0db8615"}
Nov 01 03:26:42.391578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:42.390920    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q" event={"ID":"12d11ff3-9497-498d-8aeb-7f70f1874383","Type":"ContainerStarted","Data":"f08bd3aea28a609c8cc94c8803410df69cb218a9089a1c7510b3f1282dc47cb7"}
Nov 01 03:26:42.526578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:42.517803    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:26:42.561582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:42.560911    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q" podStartSLOduration=2.560891624 podStartE2EDuration="2.560891624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:26:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:26:42.539040051 +0000 UTC m=+1207.062046668" watchObservedRunningTime="2026-11-01 03:26:42.560891624 +0000 UTC m=+1207.083898213"
Nov 01 03:26:42.764639 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:42.763604    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:26:43.094682 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:43.094633    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q"
Nov 01 03:26:43.098691 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:43.094830    4930 patch_prober.go:29] interesting pod/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 01 03:26:43.098946 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:43.098908    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q" podUID="12d11ff3-9497-498d-8aeb-7f70f1874383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 01 03:26:43.423747 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:43.423642    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp" podUID="667240b0-64a7-45cd-84fe-9cd7e0d8a301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:26:43.544581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:43.535663    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj" podUID="b0e32cb3-ac18-4c1f-819c-f1139a88bfea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:26:43.694640 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:43.691795    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm" podUID="b2637bc6-3828-4b75-9783-5415519403a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:26:43.837392 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:43.837330    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7" podUID="1ce834cc-d34d-46cc-a081-4d2cc261becb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:26:44.041219 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:44.041110    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t" podUID="13930c1e-2200-4412-b8d3-ae3e4126a851" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 01 03:26:44.097076 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:44.097036    4930 patch_prober.go:29] interesting pod/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 01 03:26:44.097356 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:44.097323    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q" podUID="12d11ff3-9497-498d-8aeb-7f70f1874383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 01 03:26:44.126685 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:44.126643    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw" podUID="b6d688b8-b911-4a6a-a842-2b0fb7a75e83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:26:44.143402 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:44.143361    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n" podUID="d2511772-66f9-40a0-9166-87c4345485c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:26:44.243664 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:44.243606    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8" podUID="88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:26:44.267760 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:44.267714    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps" podUID="d658772c-b072-4513-a639-d19a94c78525" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:26:44.268396 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:44.268022    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2" podUID="e2483a06-2f4b-4a1e-af4e-a63eb50d29bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": context deadline exceeded"
Nov 01 03:26:44.292762 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:44.292644    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm" podUID="75d142b3-7172-4b57-bc07-a2f404c4656e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:26:44.429587 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:44.428734    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp" podUID="667240b0-64a7-45cd-84fe-9cd7e0d8a301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:26:44.429587 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:44.429206    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp"
Nov 01 03:26:44.493967 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:44.493927    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" podUID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:26:44.540258 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:44.539634    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj" podUID="b0e32cb3-ac18-4c1f-819c-f1139a88bfea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:26:44.540734 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:44.540685    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:26:44.699998 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:44.699851    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm" podUID="b2637bc6-3828-4b75-9783-5415519403a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:26:44.863583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:44.860360    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7" podUID="1ce834cc-d34d-46cc-a081-4d2cc261becb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Nov 01 03:26:44.917677 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:44.917631    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"
Nov 01 03:26:44.918074 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:44.918055    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"
Nov 01 03:26:44.921054 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:44.921026    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf" podUID="40881383-61bb-4fc8-9c75-f8ffce4834c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:26:45.061658 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:45.050808    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf" podUID="40881383-61bb-4fc8-9c75-f8ffce4834c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:26:45.063484 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:45.063441    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t" podUID="13930c1e-2200-4412-b8d3-ae3e4126a851" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:26:45.065714 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:45.065691    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:26:45.147852 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:45.147814    4930 patch_prober.go:29] interesting pod/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 01 03:26:45.148129 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:45.148101    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q" podUID="12d11ff3-9497-498d-8aeb-7f70f1874383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 01 03:26:45.206579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:45.203016    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw" podUID="b6d688b8-b911-4a6a-a842-2b0fb7a75e83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded"
Nov 01 03:26:45.216083 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:45.216026    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw" podUID="b6d688b8-b911-4a6a-a842-2b0fb7a75e83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 01 03:26:45.239334 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:45.222692    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n" podUID="d2511772-66f9-40a0-9166-87c4345485c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 01 03:26:45.242568 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:45.242531    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:26:45.242869 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:45.242851    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"
Nov 01 03:26:45.255067 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:45.249034    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8" podUID="88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:26:45.255067 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:45.249442    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:26:45.297590 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:45.278016    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps" podUID="d658772c-b072-4513-a639-d19a94c78525" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 01 03:26:45.305588 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:45.298943    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:26:45.305588 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:45.300805    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm" podUID="75d142b3-7172-4b57-bc07-a2f404c4656e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:26:45.313671 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:45.307096    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:26:45.313671 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:45.312015    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"
Nov 01 03:26:45.368596 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:45.365792    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2" podUID="e2483a06-2f4b-4a1e-af4e-a63eb50d29bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:26:45.368596 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:45.366575    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"
Nov 01 03:26:45.434858 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:45.434809    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp" podUID="667240b0-64a7-45cd-84fe-9cd7e0d8a301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:26:45.501601 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:45.501530    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" podUID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:26:45.515989 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:45.515960    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:26:45.558128 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:45.558082    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj" podUID="b0e32cb3-ac18-4c1f-819c-f1139a88bfea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:26:45.591593 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:45.573683    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:26:45.591593 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:45.576913    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:26:45.715619 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:45.715467    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:26:45.715619 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:45.715531    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"
Nov 01 03:26:45.716701 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:45.715946    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"
Nov 01 03:26:45.716701 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:45.716382    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:26:45.726095 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:45.726051    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:26:45.733520 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:45.733497    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp"
Nov 01 03:26:45.733803 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:45.733772    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:26:45.733961 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:45.733947    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:26:45.745458 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:45.742785    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"
Nov 01 03:26:46.098585 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:46.097317    4930 patch_prober.go:29] interesting pod/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 01 03:26:46.098585 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:46.097370    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q" podUID="12d11ff3-9497-498d-8aeb-7f70f1874383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 01 03:26:47.092299 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:47.092208    4930 patch_prober.go:29] interesting pod/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 01 03:26:47.092299 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:47.092275    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q" podUID="12d11ff3-9497-498d-8aeb-7f70f1874383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 01 03:26:47.707599 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:47.703814    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:26:48.093117 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:48.093086    4930 patch_prober.go:29] interesting pod/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 01 03:26:48.093486 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:48.093151    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q" podUID="12d11ff3-9497-498d-8aeb-7f70f1874383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 01 03:26:48.390766 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:48.390683    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60f05eb9-x2qlg"]
Nov 01 03:26:48.395528 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:48.395506    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f05eb9-x2qlg"
Nov 01 03:26:48.446752 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:48.446724    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60f05eb9-x2qlg"]
Nov 01 03:26:48.447976 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:48.447950    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:26:48.448125 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:48.448048    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:26:48.448125 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:48.448051    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:26:48.448125 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:48.448051    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:26:48.469503 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:48.469463    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9979f531-593e-431a-a614-2b5dcad44446-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60f05eb9-x2qlg\" (UID: \"9979f531-593e-431a-a614-2b5dcad44446\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f05eb9-x2qlg"
Nov 01 03:26:48.469811 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:48.469789    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9979f531-593e-431a-a614-2b5dcad44446-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60f05eb9-x2qlg\" (UID: \"9979f531-593e-431a-a614-2b5dcad44446\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f05eb9-x2qlg"
Nov 01 03:26:48.470042 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:48.470025    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxx8f\" (UniqueName: \"kubernetes.io/projected/9979f531-593e-431a-a614-2b5dcad44446-kube-api-access-wxx8f\") pod \"maas-api-revoke-sub-60f05eb9-x2qlg\" (UID: \"9979f531-593e-431a-a614-2b5dcad44446\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f05eb9-x2qlg"
Nov 01 03:26:48.577580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:48.576326    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9979f531-593e-431a-a614-2b5dcad44446-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60f05eb9-x2qlg\" (UID: \"9979f531-593e-431a-a614-2b5dcad44446\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f05eb9-x2qlg"
Nov 01 03:26:48.577580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:48.576380    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9979f531-593e-431a-a614-2b5dcad44446-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60f05eb9-x2qlg\" (UID: \"9979f531-593e-431a-a614-2b5dcad44446\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f05eb9-x2qlg"
Nov 01 03:26:48.577580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:48.576466    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxx8f\" (UniqueName: \"kubernetes.io/projected/9979f531-593e-431a-a614-2b5dcad44446-kube-api-access-wxx8f\") pod \"maas-api-revoke-sub-60f05eb9-x2qlg\" (UID: \"9979f531-593e-431a-a614-2b5dcad44446\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f05eb9-x2qlg"
Nov 01 03:26:48.586189 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:48.586159    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9979f531-593e-431a-a614-2b5dcad44446-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60f05eb9-x2qlg\" (UID: \"9979f531-593e-431a-a614-2b5dcad44446\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f05eb9-x2qlg"
Nov 01 03:26:48.593411 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:48.593383    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9979f531-593e-431a-a614-2b5dcad44446-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60f05eb9-x2qlg\" (UID: \"9979f531-593e-431a-a614-2b5dcad44446\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f05eb9-x2qlg"
Nov 01 03:26:48.637631 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:48.637604    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxx8f\" (UniqueName: \"kubernetes.io/projected/9979f531-593e-431a-a614-2b5dcad44446-kube-api-access-wxx8f\") pod \"maas-api-revoke-sub-60f05eb9-x2qlg\" (UID: \"9979f531-593e-431a-a614-2b5dcad44446\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f05eb9-x2qlg"
Nov 01 03:26:48.710306 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:48.710226    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f05eb9-x2qlg"
Nov 01 03:26:49.092313 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:49.092280    4930 patch_prober.go:29] interesting pod/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Nov 01 03:26:49.092491 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:49.092334    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q" podUID="12d11ff3-9497-498d-8aeb-7f70f1874383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Nov 01 03:26:49.115385 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:49.115350    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60f05eb9-x2qlg"]
Nov 01 03:26:49.116135 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:26:49.115683    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9979f531_593e_431a_a614_2b5dcad44446.slice/crio-a4781cabcc6dea489fce827ba1391707c7cffdec0b628e2a2d4943df76011378 WatchSource:0}: Error finding container a4781cabcc6dea489fce827ba1391707c7cffdec0b628e2a2d4943df76011378: Status 404 returned error can't find the container with id a4781cabcc6dea489fce827ba1391707c7cffdec0b628e2a2d4943df76011378
Nov 01 03:26:49.136476 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:26:49.136441    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:26:49.648112 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:49.648073    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f05eb9-x2qlg" event={"ID":"9979f531-593e-431a-a614-2b5dcad44446","Type":"ContainerStarted","Data":"0f0c31c06f4d1873ee688f09fc18beba7597acbc871f93994877c34a26db583f"}
Nov 01 03:26:49.648309 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:49.648138    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f05eb9-x2qlg" event={"ID":"9979f531-593e-431a-a614-2b5dcad44446","Type":"ContainerStarted","Data":"a4781cabcc6dea489fce827ba1391707c7cffdec0b628e2a2d4943df76011378"}
Nov 01 03:26:49.769550 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:49.769446    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f05eb9-x2qlg" podStartSLOduration=1.769428387 podStartE2EDuration="1.769428387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:26:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:26:49.762846982 +0000 UTC m=+1214.285853587" watchObservedRunningTime="2026-11-01 03:26:49.769428387 +0000 UTC m=+1214.292434977"
Nov 01 03:26:50.158005 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:50.157974    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q"
Nov 01 03:26:50.158437 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:50.158412    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q"
Nov 01 03:26:50.159278 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:50.159258    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q"
Nov 01 03:26:50.659328 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:50.659294    4930 generic.go:356] "Generic (PLEG): container finished" podID="9979f531-593e-431a-a614-2b5dcad44446" containerID="0f0c31c06f4d1873ee688f09fc18beba7597acbc871f93994877c34a26db583f" exitCode=0
Nov 01 03:26:50.659500 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:50.659376    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f05eb9-x2qlg" event={"ID":"9979f531-593e-431a-a614-2b5dcad44446","Type":"ContainerDied","Data":"0f0c31c06f4d1873ee688f09fc18beba7597acbc871f93994877c34a26db583f"}
Nov 01 03:26:51.840858 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:51.840831    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f05eb9-x2qlg"
Nov 01 03:26:51.936831 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:51.936756    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9979f531-593e-431a-a614-2b5dcad44446-kube-api-access-wxx8f\" (UniqueName: \"kubernetes.io/projected/9979f531-593e-431a-a614-2b5dcad44446-kube-api-access-wxx8f\") pod \"9979f531-593e-431a-a614-2b5dcad44446\" (UID: \"9979f531-593e-431a-a614-2b5dcad44446\") "
Nov 01 03:26:51.936831 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:51.936811    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9979f531-593e-431a-a614-2b5dcad44446-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9979f531-593e-431a-a614-2b5dcad44446-maas-api-xks-ca\") pod \"9979f531-593e-431a-a614-2b5dcad44446\" (UID: \"9979f531-593e-431a-a614-2b5dcad44446\") "
Nov 01 03:26:51.937001 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:51.936882    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9979f531-593e-431a-a614-2b5dcad44446-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9979f531-593e-431a-a614-2b5dcad44446-maas-api-openshift-ca\") pod \"9979f531-593e-431a-a614-2b5dcad44446\" (UID: \"9979f531-593e-431a-a614-2b5dcad44446\") "
Nov 01 03:26:51.937218 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:51.937195    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9979f531-593e-431a-a614-2b5dcad44446-maas-api-openshift-ca" pod "9979f531-593e-431a-a614-2b5dcad44446" (UID: "9979f531-593e-431a-a614-2b5dcad44446"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:26:51.937452 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:51.937431    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9979f531-593e-431a-a614-2b5dcad44446-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:26:51.938848 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:51.938824    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9979f531-593e-431a-a614-2b5dcad44446-kube-api-access-wxx8f" pod "9979f531-593e-431a-a614-2b5dcad44446" (UID: "9979f531-593e-431a-a614-2b5dcad44446"). InnerVolumeSpecName "kube-api-access-wxx8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:26:51.938954 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:51.938932    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9979f531-593e-431a-a614-2b5dcad44446-maas-api-xks-ca" pod "9979f531-593e-431a-a614-2b5dcad44446" (UID: "9979f531-593e-431a-a614-2b5dcad44446"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:26:52.039184 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:52.039146    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxx8f\" (UniqueName: \"kubernetes.io/projected/9979f531-593e-431a-a614-2b5dcad44446-kube-api-access-wxx8f\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:26:52.039184 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:52.039179    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9979f531-593e-431a-a614-2b5dcad44446-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:26:52.528842 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:52.528805    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:26:52.677257 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:52.677224    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f05eb9-x2qlg"
Nov 01 03:26:52.677403 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:52.677216    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f05eb9-x2qlg" event={"ID":"9979f531-593e-431a-a614-2b5dcad44446","Type":"ContainerDied","Data":"a4781cabcc6dea489fce827ba1391707c7cffdec0b628e2a2d4943df76011378"}
Nov 01 03:26:52.677403 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:52.677346    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4781cabcc6dea489fce827ba1391707c7cffdec0b628e2a2d4943df76011378"
Nov 01 03:26:57.354476 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:57.354439    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:26:57.514403 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:57.514369    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp"]
Nov 01 03:26:57.515011 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:57.514990    4930 cpu_manager.go:414] "containerMap: removing container" podUID="9979f531-593e-431a-a614-2b5dcad44446" containerName="revoke-keys"
Nov 01 03:26:57.515011 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:57.515008    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9979f531-593e-431a-a614-2b5dcad44446" containerName="revoke-keys"
Nov 01 03:26:57.515167 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:57.515149    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="9979f531-593e-431a-a614-2b5dcad44446" containerName="revoke-keys"
Nov 01 03:26:57.518752 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:57.518735    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp"
Nov 01 03:26:57.625648 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:57.625539    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-istio-data\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp\" (UID: \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp"
Nov 01 03:26:57.625821 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:57.625648    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-istio-podinfo\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp\" (UID: \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp"
Nov 01 03:26:57.625821 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:57.625686    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-workload-certs\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp\" (UID: \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp"
Nov 01 03:26:57.625821 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:57.625762    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvtsj\" (UniqueName: \"kubernetes.io/projected/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-kube-api-access-kvtsj\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp\" (UID: \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp"
Nov 01 03:26:57.625821 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:57.625798    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-istiod-ca-cert\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp\" (UID: \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp"
Nov 01 03:26:57.625821 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:57.625821    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-istio-token\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp\" (UID: \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp"
Nov 01 03:26:57.626090 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:57.625853    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-workload-socket\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp\" (UID: \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp"
Nov 01 03:26:57.626090 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:57.625898    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-credential-socket\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp\" (UID: \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp"
Nov 01 03:26:57.626090 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:57.625923    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-istio-envoy\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp\" (UID: \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp"
Nov 01 03:26:57.628476 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:57.628444    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp"]
Nov 01 03:26:57.727439 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:57.727400    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-credential-socket\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp\" (UID: \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp"
Nov 01 03:26:57.727644 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:57.727463    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-istio-envoy\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp\" (UID: \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp"
Nov 01 03:26:57.727704 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:57.727692    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-istio-data\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp\" (UID: \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp"
Nov 01 03:26:57.727766 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:57.727728    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-istio-podinfo\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp\" (UID: \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp"
Nov 01 03:26:57.727826 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:57.727763    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-workload-certs\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp\" (UID: \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp"
Nov 01 03:26:57.727826 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:57.727808    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvtsj\" (UniqueName: \"kubernetes.io/projected/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-kube-api-access-kvtsj\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp\" (UID: \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp"
Nov 01 03:26:57.727927 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:57.727861    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-istiod-ca-cert\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp\" (UID: \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp"
Nov 01 03:26:57.727927 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:57.727886    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-credential-socket\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp\" (UID: \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp"
Nov 01 03:26:57.727927 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:57.727897    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-istio-token\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp\" (UID: \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp"
Nov 01 03:26:57.728079 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:57.727975    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-workload-socket\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp\" (UID: \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp"
Nov 01 03:26:57.728079 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:57.728002    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-istio-data\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp\" (UID: \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp"
Nov 01 03:26:57.728358 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:57.728336    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-workload-certs\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp\" (UID: \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp"
Nov 01 03:26:57.728466 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:57.728445    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-workload-socket\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp\" (UID: \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp"
Nov 01 03:26:57.728833 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:57.728810    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-istiod-ca-cert\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp\" (UID: \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp"
Nov 01 03:26:57.730293 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:57.730266    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-istio-envoy\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp\" (UID: \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp"
Nov 01 03:26:57.730860 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:57.730837    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-istio-podinfo\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp\" (UID: \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp"
Nov 01 03:26:57.788827 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:57.788796    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-istio-token\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp\" (UID: \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp"
Nov 01 03:26:57.788967 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:57.788936    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvtsj\" (UniqueName: \"kubernetes.io/projected/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-kube-api-access-kvtsj\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp\" (UID: \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp"
Nov 01 03:26:57.834767 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:57.834736    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp"
Nov 01 03:26:58.272027 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:58.271997    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp"]
Nov 01 03:26:58.272296 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:26:58.272277    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c321a5a_ae6b_49aa_bbc0_e02778d1b539.slice/crio-332eeb49ad462293c02802b5fc0a04aaeb2bd676eac8ddfe63e8af0f4775e3f7 WatchSource:0}: Error finding container 332eeb49ad462293c02802b5fc0a04aaeb2bd676eac8ddfe63e8af0f4775e3f7: Status 404 returned error can't find the container with id 332eeb49ad462293c02802b5fc0a04aaeb2bd676eac8ddfe63e8af0f4775e3f7
Nov 01 03:26:58.275501 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:58.275458    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:26:58.275616 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:58.275551    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:26:58.275658 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:58.275641    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:26:58.737718 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:58.737676    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp" event={"ID":"3c321a5a-ae6b-49aa-bbc0-e02778d1b539","Type":"ContainerStarted","Data":"2f2e341e5a8e6c32912b60d0bc335b4b5c6f2620c873665171a4981e96b77973"}
Nov 01 03:26:58.738102 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:58.737748    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp" event={"ID":"3c321a5a-ae6b-49aa-bbc0-e02778d1b539","Type":"ContainerStarted","Data":"332eeb49ad462293c02802b5fc0a04aaeb2bd676eac8ddfe63e8af0f4775e3f7"}
Nov 01 03:26:58.835351 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:58.835321    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp"
Nov 01 03:26:58.859617 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:58.859523    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp" podStartSLOduration=1.8595056730000001 podStartE2EDuration="1.859505673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:26:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:26:58.857741029 +0000 UTC m=+1223.380747617" watchObservedRunningTime="2026-11-01 03:26:58.859505673 +0000 UTC m=+1223.382512263"
Nov 01 03:26:59.837210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:59.837173    4930 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 01 03:26:59.837649 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:26:59.837228    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp" podUID="3c321a5a-ae6b-49aa-bbc0-e02778d1b539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 01 03:27:00.041718 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:00.041675    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:27:00.835186 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:00.835145    4930 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 01 03:27:00.835363 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:00.835216    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp" podUID="3c321a5a-ae6b-49aa-bbc0-e02778d1b539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 01 03:27:01.838992 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:01.838953    4930 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 01 03:27:01.839602 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:01.839548    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp" podUID="3c321a5a-ae6b-49aa-bbc0-e02778d1b539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 01 03:27:02.429651 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:02.429613    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:27:02.838633 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:02.836356    4930 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 01 03:27:02.838633 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:02.836412    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp" podUID="3c321a5a-ae6b-49aa-bbc0-e02778d1b539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 01 03:27:03.675664 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:03.675620    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2" podUID="e2483a06-2f4b-4a1e-af4e-a63eb50d29bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": context deadline exceeded"
Nov 01 03:27:03.840459 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:03.835151    4930 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 01 03:27:03.840459 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:03.835222    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp" podUID="3c321a5a-ae6b-49aa-bbc0-e02778d1b539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 01 03:27:03.914579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:03.914137    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf" podUID="40881383-61bb-4fc8-9c75-f8ffce4834c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:04.013657 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:04.012659    4930 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 03:27:04.013657 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:04.012729    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw" podUID="9cd5db87-0172-46ce-b856-cfc6366d7cb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:04.356701 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:04.330987    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n" podUID="d2511772-66f9-40a0-9166-87c4345485c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:04.568582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:04.543078    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp" podUID="667240b0-64a7-45cd-84fe-9cd7e0d8a301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 01 03:27:04.690640 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:04.599299    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw" podUID="b6d688b8-b911-4a6a-a842-2b0fb7a75e83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:04.690640 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:04.599383    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm" podUID="75d142b3-7172-4b57-bc07-a2f404c4656e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": context deadline exceeded"
Nov 01 03:27:04.690640 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:04.664663    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8" podUID="88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:04.716582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:04.703096    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2" podUID="e2483a06-2f4b-4a1e-af4e-a63eb50d29bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": context deadline exceeded"
Nov 01 03:27:04.716582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:04.704141    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"
Nov 01 03:27:04.716582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:04.705330    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm" podUID="b2637bc6-3828-4b75-9783-5415519403a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:04.750579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:04.747609    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps" podUID="d658772c-b072-4513-a639-d19a94c78525" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:04.750579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:04.749640    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj" podUID="b0e32cb3-ac18-4c1f-819c-f1139a88bfea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:04.750923 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:04.750892    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t" podUID="13930c1e-2200-4412-b8d3-ae3e4126a851" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:04.759641 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:04.751070    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" podUID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded"
Nov 01 03:27:04.786611 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:04.751109    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7" podUID="1ce834cc-d34d-46cc-a081-4d2cc261becb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:04.808104 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:04.807970    4930 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 03:27:04.808104 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:04.808029    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr" podUID="f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:04.845205 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:04.843362    4930 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 01 03:27:04.845205 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:04.843463    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp" podUID="3c321a5a-ae6b-49aa-bbc0-e02778d1b539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 01 03:27:04.921228 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:04.921185    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf" podUID="40881383-61bb-4fc8-9c75-f8ffce4834c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Nov 01 03:27:04.922815 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:04.922789    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:27:04.989404 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:04.989303    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf" podUID="40881383-61bb-4fc8-9c75-f8ffce4834c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:05.129580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:05.107255    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw" podUID="b6d688b8-b911-4a6a-a842-2b0fb7a75e83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:05.178821 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:05.167235    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce6047d8-wnpkp"]
Nov 01 03:27:05.231972 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:05.231936    4930 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 03:27:05.232254 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:05.232221    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll" podUID="0405e5bf-ff6e-467d-9f4c-9bd36005c3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:05.555282 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:05.555218    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n" podUID="d2511772-66f9-40a0-9166-87c4345485c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 01 03:27:05.618588 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:05.581702    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce6047d8-wnpkp"]
Nov 01 03:27:05.618588 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:05.581867    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"
Nov 01 03:27:05.618588 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:05.612850    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm" podUID="b2637bc6-3828-4b75-9783-5415519403a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": context deadline exceeded"
Nov 01 03:27:05.682688 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:05.682178    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm" podUID="75d142b3-7172-4b57-bc07-a2f404c4656e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:05.716493 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:05.716450    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw" podUID="b6d688b8-b911-4a6a-a842-2b0fb7a75e83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 01 03:27:05.732591 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:05.723761    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2" podUID="e2483a06-2f4b-4a1e-af4e-a63eb50d29bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": context deadline exceeded"
Nov 01 03:27:05.751428 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:05.744500    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp" podUID="667240b0-64a7-45cd-84fe-9cd7e0d8a301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:05.751428 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:05.749943    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d58c6470-aafb-44c0-ae6a-6b61ddacfd2d" path="/var/lib/kubelet/pods/d58c6470-aafb-44c0-ae6a-6b61ddacfd2d/volumes"
Nov 01 03:27:05.751428 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:05.750444    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:27:05.751428 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:05.750496    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:27:05.751428 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:05.750528    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp"
Nov 01 03:27:05.751831 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:05.751476    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj" podUID="b0e32cb3-ac18-4c1f-819c-f1139a88bfea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:05.754821 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:05.754799    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:27:05.755342 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:05.755320    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t" podUID="13930c1e-2200-4412-b8d3-ae3e4126a851" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:05.755709 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:05.755695    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:27:05.756049 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:05.755779    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm" podUID="b2637bc6-3828-4b75-9783-5415519403a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:05.756321 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:05.756308    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"
Nov 01 03:27:05.760091 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:05.758480    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps" podUID="d658772c-b072-4513-a639-d19a94c78525" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:05.760091 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:05.758849    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:27:05.768282 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:05.766993    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:27:05.774458 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:05.774437    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:27:05.804581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:05.788551    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" podUID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded"
Nov 01 03:27:05.804581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:05.788934    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:27:05.811036 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:05.810732    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:27:05.817673 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:05.817539    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"
Nov 01 03:27:05.835629 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:05.835609    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:27:05.835796 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:05.835783    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"
Nov 01 03:27:05.839206 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:05.839181    4930 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 01 03:27:05.839373 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:05.839351    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp" podUID="3c321a5a-ae6b-49aa-bbc0-e02778d1b539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 01 03:27:05.857079 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:05.854725    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"
Nov 01 03:27:05.896339 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:05.896120    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:27:05.900054 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:05.897265    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp"
Nov 01 03:27:05.900606 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:05.900584    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:27:05.901067 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:05.901047    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:27:06.788777 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:06.788740    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fdc5677-fs7dr"]
Nov 01 03:27:06.794422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:06.794397    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc5677-fs7dr"
Nov 01 03:27:06.847708 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:06.847675    4930 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 01 03:27:06.847922 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:06.847894    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:27:06.848050 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:06.848021    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp" podUID="3c321a5a-ae6b-49aa-bbc0-e02778d1b539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 01 03:27:06.848160 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:06.848118    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:27:06.848279 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:06.848178    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:27:06.848366 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:06.848281    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:27:06.862938 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:06.862336    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k7s2\" (UniqueName: \"kubernetes.io/projected/604d1816-9db7-471b-8ad7-7f11c819937d-kube-api-access-8k7s2\") pod \"maas-api-revoke-sub-4fdc5677-fs7dr\" (UID: \"604d1816-9db7-471b-8ad7-7f11c819937d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc5677-fs7dr"
Nov 01 03:27:06.862938 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:06.862430    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/604d1816-9db7-471b-8ad7-7f11c819937d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fdc5677-fs7dr\" (UID: \"604d1816-9db7-471b-8ad7-7f11c819937d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc5677-fs7dr"
Nov 01 03:27:06.862938 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:06.862624    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/604d1816-9db7-471b-8ad7-7f11c819937d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fdc5677-fs7dr\" (UID: \"604d1816-9db7-471b-8ad7-7f11c819937d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc5677-fs7dr"
Nov 01 03:27:06.863876 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:06.863842    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fdc5677-fs7dr"]
Nov 01 03:27:06.964748 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:06.964703    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k7s2\" (UniqueName: \"kubernetes.io/projected/604d1816-9db7-471b-8ad7-7f11c819937d-kube-api-access-8k7s2\") pod \"maas-api-revoke-sub-4fdc5677-fs7dr\" (UID: \"604d1816-9db7-471b-8ad7-7f11c819937d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc5677-fs7dr"
Nov 01 03:27:06.966533 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:06.966489    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/604d1816-9db7-471b-8ad7-7f11c819937d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fdc5677-fs7dr\" (UID: \"604d1816-9db7-471b-8ad7-7f11c819937d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc5677-fs7dr"
Nov 01 03:27:06.966962 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:06.966883    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/604d1816-9db7-471b-8ad7-7f11c819937d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fdc5677-fs7dr\" (UID: \"604d1816-9db7-471b-8ad7-7f11c819937d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc5677-fs7dr"
Nov 01 03:27:06.969326 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:06.969293    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/604d1816-9db7-471b-8ad7-7f11c819937d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fdc5677-fs7dr\" (UID: \"604d1816-9db7-471b-8ad7-7f11c819937d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc5677-fs7dr"
Nov 01 03:27:06.972256 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:06.972225    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/604d1816-9db7-471b-8ad7-7f11c819937d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fdc5677-fs7dr\" (UID: \"604d1816-9db7-471b-8ad7-7f11c819937d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc5677-fs7dr"
Nov 01 03:27:07.038101 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:07.038073    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k7s2\" (UniqueName: \"kubernetes.io/projected/604d1816-9db7-471b-8ad7-7f11c819937d-kube-api-access-8k7s2\") pod \"maas-api-revoke-sub-4fdc5677-fs7dr\" (UID: \"604d1816-9db7-471b-8ad7-7f11c819937d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc5677-fs7dr"
Nov 01 03:27:07.111155 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:07.111071    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc5677-fs7dr"
Nov 01 03:27:07.555156 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:07.555039    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:27:07.641789 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:07.641758    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fdc5677-fs7dr"]
Nov 01 03:27:07.643218 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:27:07.643187    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod604d1816_9db7_471b_8ad7_7f11c819937d.slice/crio-9140a167c246b91a721db80e1f9996afe15d3bf926055c5fc771efa32fe264bf WatchSource:0}: Error finding container 9140a167c246b91a721db80e1f9996afe15d3bf926055c5fc771efa32fe264bf: Status 404 returned error can't find the container with id 9140a167c246b91a721db80e1f9996afe15d3bf926055c5fc771efa32fe264bf
Nov 01 03:27:07.840258 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:07.840222    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp"
Nov 01 03:27:07.840957 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:07.840935    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp"
Nov 01 03:27:07.841805 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:07.841777    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp"
Nov 01 03:27:08.028666 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:08.028627    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc5677-fs7dr" event={"ID":"604d1816-9db7-471b-8ad7-7f11c819937d","Type":"ContainerStarted","Data":"9140a167c246b91a721db80e1f9996afe15d3bf926055c5fc771efa32fe264bf"}
Nov 01 03:27:09.039545 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:09.039455    4930 generic.go:356] "Generic (PLEG): container finished" podID="604d1816-9db7-471b-8ad7-7f11c819937d" containerID="c5f1d6e5d57fe27cb0a8482c509f75f8ca456ca0151a7cb7c7bae76ceae32410" exitCode=0
Nov 01 03:27:09.039974 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:09.039538    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc5677-fs7dr" event={"ID":"604d1816-9db7-471b-8ad7-7f11c819937d","Type":"ContainerDied","Data":"c5f1d6e5d57fe27cb0a8482c509f75f8ca456ca0151a7cb7c7bae76ceae32410"}
Nov 01 03:27:10.203842 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:10.203816    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc5677-fs7dr"
Nov 01 03:27:10.315052 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:10.314973    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/604d1816-9db7-471b-8ad7-7f11c819937d-kube-api-access-8k7s2\" (UniqueName: \"kubernetes.io/projected/604d1816-9db7-471b-8ad7-7f11c819937d-kube-api-access-8k7s2\") pod \"604d1816-9db7-471b-8ad7-7f11c819937d\" (UID: \"604d1816-9db7-471b-8ad7-7f11c819937d\") "
Nov 01 03:27:10.315052 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:10.315017    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/604d1816-9db7-471b-8ad7-7f11c819937d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/604d1816-9db7-471b-8ad7-7f11c819937d-maas-api-xks-ca\") pod \"604d1816-9db7-471b-8ad7-7f11c819937d\" (UID: \"604d1816-9db7-471b-8ad7-7f11c819937d\") "
Nov 01 03:27:10.315237 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:10.315135    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/604d1816-9db7-471b-8ad7-7f11c819937d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/604d1816-9db7-471b-8ad7-7f11c819937d-maas-api-openshift-ca\") pod \"604d1816-9db7-471b-8ad7-7f11c819937d\" (UID: \"604d1816-9db7-471b-8ad7-7f11c819937d\") "
Nov 01 03:27:10.315521 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:10.315493    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/604d1816-9db7-471b-8ad7-7f11c819937d-maas-api-openshift-ca" pod "604d1816-9db7-471b-8ad7-7f11c819937d" (UID: "604d1816-9db7-471b-8ad7-7f11c819937d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:27:10.315828 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:10.315791    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/604d1816-9db7-471b-8ad7-7f11c819937d-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:27:10.317230 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:10.317204    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/604d1816-9db7-471b-8ad7-7f11c819937d-maas-api-xks-ca" pod "604d1816-9db7-471b-8ad7-7f11c819937d" (UID: "604d1816-9db7-471b-8ad7-7f11c819937d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:27:10.317382 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:10.317354    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/604d1816-9db7-471b-8ad7-7f11c819937d-kube-api-access-8k7s2" pod "604d1816-9db7-471b-8ad7-7f11c819937d" (UID: "604d1816-9db7-471b-8ad7-7f11c819937d"). InnerVolumeSpecName "kube-api-access-8k7s2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:27:10.416951 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:10.416919    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k7s2\" (UniqueName: \"kubernetes.io/projected/604d1816-9db7-471b-8ad7-7f11c819937d-kube-api-access-8k7s2\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:27:10.416951 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:10.416947    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/604d1816-9db7-471b-8ad7-7f11c819937d-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:27:11.057048 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:11.057012    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc5677-fs7dr"
Nov 01 03:27:11.057048 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:11.057042    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fdc5677-fs7dr" event={"ID":"604d1816-9db7-471b-8ad7-7f11c819937d","Type":"ContainerDied","Data":"9140a167c246b91a721db80e1f9996afe15d3bf926055c5fc771efa32fe264bf"}
Nov 01 03:27:11.057260 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:11.057091    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9140a167c246b91a721db80e1f9996afe15d3bf926055c5fc771efa32fe264bf"
Nov 01 03:27:12.369270 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:12.369235    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:27:13.050634 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:13.050593    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73cabe60-pd29j"]
Nov 01 03:27:13.099721 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:13.099692    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73cabe60-pd29j"]
Nov 01 03:27:13.621457 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:13.621422    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1" path="/var/lib/kubelet/pods/2158baa8-ec88-4f7c-bb5b-42d7eba3e9f1/volumes"
Nov 01 03:27:17.264275 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:17.264236    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:27:19.051817 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:19.051785    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a673175-gpzwp"]
Nov 01 03:27:19.103953 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:19.103918    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a673175-gpzwp"]
Nov 01 03:27:19.122943 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:27:19.122915    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:27:19.130131 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:19.130106    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8358d178-lj8hq"]
Nov 01 03:27:19.130820 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:19.130798    4930 cpu_manager.go:414] "containerMap: removing container" podUID="604d1816-9db7-471b-8ad7-7f11c819937d" containerName="revoke-keys"
Nov 01 03:27:19.130820 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:19.130819    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="604d1816-9db7-471b-8ad7-7f11c819937d" containerName="revoke-keys"
Nov 01 03:27:19.131002 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:19.130990    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="604d1816-9db7-471b-8ad7-7f11c819937d" containerName="revoke-keys"
Nov 01 03:27:19.136028 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:19.136008    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8358d178-lj8hq"
Nov 01 03:27:19.188159 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:19.188129    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:27:19.188771 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:19.188746    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:27:19.195642 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:19.195615    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:27:19.195766 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:19.195750    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:27:19.195883 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:19.195864    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8358d178-lj8hq"]
Nov 01 03:27:19.238563 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:19.238531    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b1a43c-397a-43d7-8c23-4a0638f1ac18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8358d178-lj8hq\" (UID: \"c8b1a43c-397a-43d7-8c23-4a0638f1ac18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8358d178-lj8hq"
Nov 01 03:27:19.238710 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:19.238691    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b1a43c-397a-43d7-8c23-4a0638f1ac18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8358d178-lj8hq\" (UID: \"c8b1a43c-397a-43d7-8c23-4a0638f1ac18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8358d178-lj8hq"
Nov 01 03:27:19.238758 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:19.238748    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w59hz\" (UniqueName: \"kubernetes.io/projected/c8b1a43c-397a-43d7-8c23-4a0638f1ac18-kube-api-access-w59hz\") pod \"maas-api-revoke-sub-8358d178-lj8hq\" (UID: \"c8b1a43c-397a-43d7-8c23-4a0638f1ac18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8358d178-lj8hq"
Nov 01 03:27:19.339923 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:19.339894    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b1a43c-397a-43d7-8c23-4a0638f1ac18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8358d178-lj8hq\" (UID: \"c8b1a43c-397a-43d7-8c23-4a0638f1ac18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8358d178-lj8hq"
Nov 01 03:27:19.340057 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:19.339958    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b1a43c-397a-43d7-8c23-4a0638f1ac18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8358d178-lj8hq\" (UID: \"c8b1a43c-397a-43d7-8c23-4a0638f1ac18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8358d178-lj8hq"
Nov 01 03:27:19.340057 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:19.339986    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w59hz\" (UniqueName: \"kubernetes.io/projected/c8b1a43c-397a-43d7-8c23-4a0638f1ac18-kube-api-access-w59hz\") pod \"maas-api-revoke-sub-8358d178-lj8hq\" (UID: \"c8b1a43c-397a-43d7-8c23-4a0638f1ac18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8358d178-lj8hq"
Nov 01 03:27:19.340814 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:19.340792    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b1a43c-397a-43d7-8c23-4a0638f1ac18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8358d178-lj8hq\" (UID: \"c8b1a43c-397a-43d7-8c23-4a0638f1ac18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8358d178-lj8hq"
Nov 01 03:27:19.342403 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:19.342378    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b1a43c-397a-43d7-8c23-4a0638f1ac18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8358d178-lj8hq\" (UID: \"c8b1a43c-397a-43d7-8c23-4a0638f1ac18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8358d178-lj8hq"
Nov 01 03:27:19.401218 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:19.401193    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w59hz\" (UniqueName: \"kubernetes.io/projected/c8b1a43c-397a-43d7-8c23-4a0638f1ac18-kube-api-access-w59hz\") pod \"maas-api-revoke-sub-8358d178-lj8hq\" (UID: \"c8b1a43c-397a-43d7-8c23-4a0638f1ac18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8358d178-lj8hq"
Nov 01 03:27:19.450314 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:19.450285    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8358d178-lj8hq"
Nov 01 03:27:19.624880 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:19.624804    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa78a11f-6052-4a54-aabf-dc93c44310d5" path="/var/lib/kubelet/pods/aa78a11f-6052-4a54-aabf-dc93c44310d5/volumes"
Nov 01 03:27:19.834847 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:19.834681    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8358d178-lj8hq"]
Nov 01 03:27:19.838485 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:27:19.838441    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b1a43c_397a_43d7_8c23_4a0638f1ac18.slice/crio-3cb4543bf8a486313b88a0e1c07ea3cf16ecd683980a32a6ef2b0f6b73b11d8e WatchSource:0}: Error finding container 3cb4543bf8a486313b88a0e1c07ea3cf16ecd683980a32a6ef2b0f6b73b11d8e: Status 404 returned error can't find the container with id 3cb4543bf8a486313b88a0e1c07ea3cf16ecd683980a32a6ef2b0f6b73b11d8e
Nov 01 03:27:20.144981 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:20.144900    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8358d178-lj8hq" event={"ID":"c8b1a43c-397a-43d7-8c23-4a0638f1ac18","Type":"ContainerStarted","Data":"3cb4543bf8a486313b88a0e1c07ea3cf16ecd683980a32a6ef2b0f6b73b11d8e"}
Nov 01 03:27:21.157176 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:21.157076    4930 generic.go:356] "Generic (PLEG): container finished" podID="c8b1a43c-397a-43d7-8c23-4a0638f1ac18" containerID="cb8d2b46ccff7c7c1f632d505b787f705a90af3e39fd92caa73385a99ab16ff3" exitCode=0
Nov 01 03:27:21.157710 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:21.157266    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8358d178-lj8hq" event={"ID":"c8b1a43c-397a-43d7-8c23-4a0638f1ac18","Type":"ContainerDied","Data":"cb8d2b46ccff7c7c1f632d505b787f705a90af3e39fd92caa73385a99ab16ff3"}
Nov 01 03:27:22.104232 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:22.104193    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69e6dcb6-x7gdw"]
Nov 01 03:27:22.179171 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:22.179136    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69e6dcb6-x7gdw"]
Nov 01 03:27:22.666591 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:22.664647    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8" podUID="88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:22.750748 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:22.743238    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2" podUID="e2483a06-2f4b-4a1e-af4e-a63eb50d29bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": context deadline exceeded"
Nov 01 03:27:22.781820 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:22.777662    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm" podUID="75d142b3-7172-4b57-bc07-a2f404c4656e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:22.781820 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:22.781629    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf" podUID="40881383-61bb-4fc8-9c75-f8ffce4834c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:22.819685 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:22.819638    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t" podUID="13930c1e-2200-4412-b8d3-ae3e4126a851" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:22.820246 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:22.820215    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n" podUID="d2511772-66f9-40a0-9166-87c4345485c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 01 03:27:22.831189 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:22.831163    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:27:22.839659 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:22.839626    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm" podUID="b2637bc6-3828-4b75-9783-5415519403a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:22.840089 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:22.840064    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" podUID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded"
Nov 01 03:27:22.904578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:22.900657    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw" podUID="b6d688b8-b911-4a6a-a842-2b0fb7a75e83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:22.904578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:22.901048    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp" podUID="667240b0-64a7-45cd-84fe-9cd7e0d8a301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 01 03:27:22.905441 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:22.905402    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj" podUID="b0e32cb3-ac18-4c1f-819c-f1139a88bfea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:22.910834 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:22.910688    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps" podUID="d658772c-b072-4513-a639-d19a94c78525" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:23.150937 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:23.150687    4930 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 03:27:23.150937 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:23.150771    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr" podUID="267415f5-02bf-4571-914a-45aa36ae41c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:23.451916 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:23.451815    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" podUID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:23.471652 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:23.471617    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8358d178-lj8hq" event={"ID":"c8b1a43c-397a-43d7-8c23-4a0638f1ac18","Type":"ContainerDied","Data":"3cb4543bf8a486313b88a0e1c07ea3cf16ecd683980a32a6ef2b0f6b73b11d8e"}
Nov 01 03:27:23.471870 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:23.471854    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cb4543bf8a486313b88a0e1c07ea3cf16ecd683980a32a6ef2b0f6b73b11d8e"
Nov 01 03:27:23.476011 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:23.474001    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8358d178-lj8hq"
Nov 01 03:27:23.616443 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:23.613406    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8b1a43c-397a-43d7-8c23-4a0638f1ac18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b1a43c-397a-43d7-8c23-4a0638f1ac18-maas-api-openshift-ca\") pod \"c8b1a43c-397a-43d7-8c23-4a0638f1ac18\" (UID: \"c8b1a43c-397a-43d7-8c23-4a0638f1ac18\") "
Nov 01 03:27:23.616443 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:23.613539    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8b1a43c-397a-43d7-8c23-4a0638f1ac18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b1a43c-397a-43d7-8c23-4a0638f1ac18-maas-api-xks-ca\") pod \"c8b1a43c-397a-43d7-8c23-4a0638f1ac18\" (UID: \"c8b1a43c-397a-43d7-8c23-4a0638f1ac18\") "
Nov 01 03:27:23.638618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:23.632711    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8b1a43c-397a-43d7-8c23-4a0638f1ac18-kube-api-access-w59hz\" (UniqueName: \"kubernetes.io/projected/c8b1a43c-397a-43d7-8c23-4a0638f1ac18-kube-api-access-w59hz\") pod \"c8b1a43c-397a-43d7-8c23-4a0638f1ac18\" (UID: \"c8b1a43c-397a-43d7-8c23-4a0638f1ac18\") "
Nov 01 03:27:23.694162 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:23.684016    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8" podUID="88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Nov 01 03:27:23.694162 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:23.624048    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8b1a43c-397a-43d7-8c23-4a0638f1ac18-maas-api-openshift-ca" pod "c8b1a43c-397a-43d7-8c23-4a0638f1ac18" (UID: "c8b1a43c-397a-43d7-8c23-4a0638f1ac18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:27:23.741313 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:23.741217    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b1a43c-397a-43d7-8c23-4a0638f1ac18-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:27:23.758579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:23.743866    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2" podUID="e2483a06-2f4b-4a1e-af4e-a63eb50d29bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:23.760883 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:23.760739    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7" podUID="1ce834cc-d34d-46cc-a081-4d2cc261becb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:23.779620 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:23.766935    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94819519-6491-4744-9a04-c323de973006" path="/var/lib/kubelet/pods/94819519-6491-4744-9a04-c323de973006/volumes"
Nov 01 03:27:23.783542 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:23.783506    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf" podUID="40881383-61bb-4fc8-9c75-f8ffce4834c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:23.799606 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:23.784247    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:27:23.799606 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:23.784305    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"
Nov 01 03:27:23.799606 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:23.784339    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:27:23.825766 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:23.821723    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t" podUID="13930c1e-2200-4412-b8d3-ae3e4126a851" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:23.825766 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:23.824663    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:27:23.864580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:23.840587    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" podUID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:23.887611 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:23.871034    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:27:23.893580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:23.842798    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm" podUID="b2637bc6-3828-4b75-9783-5415519403a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 01 03:27:23.912060 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:23.912013    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp" podUID="667240b0-64a7-45cd-84fe-9cd7e0d8a301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:23.915843 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:23.915787    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw" podUID="b6d688b8-b911-4a6a-a842-2b0fb7a75e83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 01 03:27:23.984364 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:23.932437    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"
Nov 01 03:27:23.988702 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:23.988677    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp"
Nov 01 03:27:23.988924 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:23.988907    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:27:23.994682 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:23.933689    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps" podUID="d658772c-b072-4513-a639-d19a94c78525" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:24.012795 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:24.003495    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:27:24.027489 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:24.027457    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8b1a43c-397a-43d7-8c23-4a0638f1ac18-kube-api-access-w59hz" pod "c8b1a43c-397a-43d7-8c23-4a0638f1ac18" (UID: "c8b1a43c-397a-43d7-8c23-4a0638f1ac18"). InnerVolumeSpecName "kube-api-access-w59hz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:27:24.110036 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:24.109997    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w59hz\" (UniqueName: \"kubernetes.io/projected/c8b1a43c-397a-43d7-8c23-4a0638f1ac18-kube-api-access-w59hz\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:27:24.326645 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:24.304652    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj" podUID="b0e32cb3-ac18-4c1f-819c-f1139a88bfea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 01 03:27:24.533746 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:24.362261    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8b1a43c-397a-43d7-8c23-4a0638f1ac18-maas-api-xks-ca" pod "c8b1a43c-397a-43d7-8c23-4a0638f1ac18" (UID: "c8b1a43c-397a-43d7-8c23-4a0638f1ac18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:27:24.534526 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:24.534289    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b1a43c-397a-43d7-8c23-4a0638f1ac18-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:27:24.539584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:24.539539    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:27:24.602572 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:24.599768    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:27:24.604758 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:24.604732    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:27:24.613794 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:24.613768    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:27:24.617498 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:24.617478    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:27:24.634689 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:24.634668    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:27:24.634950 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:24.634922    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"
Nov 01 03:27:24.635466 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:24.635451    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:27:24.639872 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:24.639856    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"
Nov 01 03:27:24.643479 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:24.643254    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8358d178-lj8hq"
Nov 01 03:27:24.645123 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:24.645104    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp"
Nov 01 03:27:24.658360 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:24.658341    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40ab1571-h4jqz"]
Nov 01 03:27:24.659376 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:24.659356    4930 cpu_manager.go:414] "containerMap: removing container" podUID="c8b1a43c-397a-43d7-8c23-4a0638f1ac18" containerName="revoke-keys"
Nov 01 03:27:24.660864 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:24.660847    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8b1a43c-397a-43d7-8c23-4a0638f1ac18" containerName="revoke-keys"
Nov 01 03:27:24.661220 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:24.661208    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8b1a43c-397a-43d7-8c23-4a0638f1ac18" containerName="revoke-keys"
Nov 01 03:27:24.668846 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:24.668825    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:27:24.668945 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:24.668857    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40ab1571-h4jqz"]
Nov 01 03:27:24.669111 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:24.669092    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ab1571-h4jqz"
Nov 01 03:27:24.723212 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:24.723174    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:27:24.725434 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:24.725414    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:27:24.725575 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:24.725548    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:27:24.725631 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:24.725590    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:27:24.749197 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:24.749137    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7b6d3eb-e0cf-40b9-bfa2-e543626883fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40ab1571-h4jqz\" (UID: \"e7b6d3eb-e0cf-40b9-bfa2-e543626883fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ab1571-h4jqz"
Nov 01 03:27:24.749336 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:24.749267    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d54xq\" (UniqueName: \"kubernetes.io/projected/e7b6d3eb-e0cf-40b9-bfa2-e543626883fa-kube-api-access-d54xq\") pod \"maas-api-revoke-sub-40ab1571-h4jqz\" (UID: \"e7b6d3eb-e0cf-40b9-bfa2-e543626883fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ab1571-h4jqz"
Nov 01 03:27:24.749660 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:24.749628    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7b6d3eb-e0cf-40b9-bfa2-e543626883fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40ab1571-h4jqz\" (UID: \"e7b6d3eb-e0cf-40b9-bfa2-e543626883fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ab1571-h4jqz"
Nov 01 03:27:24.851460 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:24.851427    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7b6d3eb-e0cf-40b9-bfa2-e543626883fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40ab1571-h4jqz\" (UID: \"e7b6d3eb-e0cf-40b9-bfa2-e543626883fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ab1571-h4jqz"
Nov 01 03:27:24.851678 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:24.851615    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7b6d3eb-e0cf-40b9-bfa2-e543626883fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40ab1571-h4jqz\" (UID: \"e7b6d3eb-e0cf-40b9-bfa2-e543626883fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ab1571-h4jqz"
Nov 01 03:27:24.851678 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:24.851660    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d54xq\" (UniqueName: \"kubernetes.io/projected/e7b6d3eb-e0cf-40b9-bfa2-e543626883fa-kube-api-access-d54xq\") pod \"maas-api-revoke-sub-40ab1571-h4jqz\" (UID: \"e7b6d3eb-e0cf-40b9-bfa2-e543626883fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ab1571-h4jqz"
Nov 01 03:27:24.852111 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:24.852086    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7b6d3eb-e0cf-40b9-bfa2-e543626883fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40ab1571-h4jqz\" (UID: \"e7b6d3eb-e0cf-40b9-bfa2-e543626883fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ab1571-h4jqz"
Nov 01 03:27:24.857086 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:24.857003    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7b6d3eb-e0cf-40b9-bfa2-e543626883fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40ab1571-h4jqz\" (UID: \"e7b6d3eb-e0cf-40b9-bfa2-e543626883fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ab1571-h4jqz"
Nov 01 03:27:24.912932 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:24.912901    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d54xq\" (UniqueName: \"kubernetes.io/projected/e7b6d3eb-e0cf-40b9-bfa2-e543626883fa-kube-api-access-d54xq\") pod \"maas-api-revoke-sub-40ab1571-h4jqz\" (UID: \"e7b6d3eb-e0cf-40b9-bfa2-e543626883fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ab1571-h4jqz"
Nov 01 03:27:24.998264 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:24.998229    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ab1571-h4jqz"
Nov 01 03:27:25.119548 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:25.119470    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d87394e-pgwwl"]
Nov 01 03:27:25.124690 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:25.124663    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d87394e-pgwwl"
Nov 01 03:27:25.151841 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:25.151810    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d87394e-pgwwl"]
Nov 01 03:27:25.160209 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:25.160182    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5211cbab-e13b-4e4e-899d-3195a8c8ce23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d87394e-pgwwl\" (UID: \"5211cbab-e13b-4e4e-899d-3195a8c8ce23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d87394e-pgwwl"
Nov 01 03:27:25.160364 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:25.160342    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfp7l\" (UniqueName: \"kubernetes.io/projected/5211cbab-e13b-4e4e-899d-3195a8c8ce23-kube-api-access-bfp7l\") pod \"maas-api-revoke-sub-5d87394e-pgwwl\" (UID: \"5211cbab-e13b-4e4e-899d-3195a8c8ce23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d87394e-pgwwl"
Nov 01 03:27:25.160432 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:25.160416    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5211cbab-e13b-4e4e-899d-3195a8c8ce23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d87394e-pgwwl\" (UID: \"5211cbab-e13b-4e4e-899d-3195a8c8ce23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d87394e-pgwwl"
Nov 01 03:27:25.261832 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:25.261803    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5211cbab-e13b-4e4e-899d-3195a8c8ce23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d87394e-pgwwl\" (UID: \"5211cbab-e13b-4e4e-899d-3195a8c8ce23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d87394e-pgwwl"
Nov 01 03:27:25.261999 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:25.261945    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfp7l\" (UniqueName: \"kubernetes.io/projected/5211cbab-e13b-4e4e-899d-3195a8c8ce23-kube-api-access-bfp7l\") pod \"maas-api-revoke-sub-5d87394e-pgwwl\" (UID: \"5211cbab-e13b-4e4e-899d-3195a8c8ce23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d87394e-pgwwl"
Nov 01 03:27:25.261999 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:25.261985    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5211cbab-e13b-4e4e-899d-3195a8c8ce23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d87394e-pgwwl\" (UID: \"5211cbab-e13b-4e4e-899d-3195a8c8ce23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d87394e-pgwwl"
Nov 01 03:27:25.262504 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:25.262475    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5211cbab-e13b-4e4e-899d-3195a8c8ce23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d87394e-pgwwl\" (UID: \"5211cbab-e13b-4e4e-899d-3195a8c8ce23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d87394e-pgwwl"
Nov 01 03:27:25.264400 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:25.264372    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5211cbab-e13b-4e4e-899d-3195a8c8ce23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d87394e-pgwwl\" (UID: \"5211cbab-e13b-4e4e-899d-3195a8c8ce23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d87394e-pgwwl"
Nov 01 03:27:25.323277 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:25.323242    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfp7l\" (UniqueName: \"kubernetes.io/projected/5211cbab-e13b-4e4e-899d-3195a8c8ce23-kube-api-access-bfp7l\") pod \"maas-api-revoke-sub-5d87394e-pgwwl\" (UID: \"5211cbab-e13b-4e4e-899d-3195a8c8ce23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d87394e-pgwwl"
Nov 01 03:27:25.406980 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:27:25.406916    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7b6d3eb_e0cf_40b9_bfa2_e543626883fa.slice/crio-716ed6760eba909589d9d8267d81ecad1402436b4e81f0a1c44ee90366ef0508 WatchSource:0}: Error finding container 716ed6760eba909589d9d8267d81ecad1402436b4e81f0a1c44ee90366ef0508: Status 404 returned error can't find the container with id 716ed6760eba909589d9d8267d81ecad1402436b4e81f0a1c44ee90366ef0508
Nov 01 03:27:25.408699 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:25.408590    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40ab1571-h4jqz"]
Nov 01 03:27:25.440438 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:25.440410    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d87394e-pgwwl"
Nov 01 03:27:25.653545 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:25.653467    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ab1571-h4jqz" event={"ID":"e7b6d3eb-e0cf-40b9-bfa2-e543626883fa","Type":"ContainerStarted","Data":"716ed6760eba909589d9d8267d81ecad1402436b4e81f0a1c44ee90366ef0508"}
Nov 01 03:27:25.870745 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:25.870706    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n"]
Nov 01 03:27:25.888991 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:25.888937    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n"
Nov 01 03:27:25.895818 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:25.895793    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d87394e-pgwwl"]
Nov 01 03:27:25.937104 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:25.936940    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n"]
Nov 01 03:27:25.990719 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:25.990679    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-istio-token\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n\" (UID: \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n"
Nov 01 03:27:25.990923 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:25.990733    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-workload-certs\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n\" (UID: \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n"
Nov 01 03:27:25.990923 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:25.990852    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kd2w\" (UniqueName: \"kubernetes.io/projected/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-kube-api-access-5kd2w\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n\" (UID: \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n"
Nov 01 03:27:25.990923 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:25.990902    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-workload-socket\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n\" (UID: \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n"
Nov 01 03:27:25.991103 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:25.991035    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-istio-envoy\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n\" (UID: \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n"
Nov 01 03:27:25.991206 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:25.991186    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-istiod-ca-cert\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n\" (UID: \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n"
Nov 01 03:27:25.991352 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:25.991336    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-istio-data\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n\" (UID: \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n"
Nov 01 03:27:25.991408 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:25.991385    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-credential-socket\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n\" (UID: \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n"
Nov 01 03:27:25.991469 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:25.991455    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-istio-podinfo\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n\" (UID: \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n"
Nov 01 03:27:26.093031 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:26.092993    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-istio-data\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n\" (UID: \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n"
Nov 01 03:27:26.093286 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:26.093141    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-credential-socket\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n\" (UID: \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n"
Nov 01 03:27:26.093286 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:26.093236    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-istio-podinfo\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n\" (UID: \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n"
Nov 01 03:27:26.093436 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:26.093290    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-istio-token\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n\" (UID: \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n"
Nov 01 03:27:26.093436 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:26.093328    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-workload-certs\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n\" (UID: \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n"
Nov 01 03:27:26.093436 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:26.093404    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kd2w\" (UniqueName: \"kubernetes.io/projected/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-kube-api-access-5kd2w\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n\" (UID: \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n"
Nov 01 03:27:26.093640 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:26.093440    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-istio-data\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n\" (UID: \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n"
Nov 01 03:27:26.093640 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:26.093523    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-workload-socket\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n\" (UID: \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n"
Nov 01 03:27:26.093640 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:26.093615    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-istio-envoy\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n\" (UID: \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n"
Nov 01 03:27:26.093804 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:26.093745    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-istiod-ca-cert\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n\" (UID: \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n"
Nov 01 03:27:26.094734 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:26.094704    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-credential-socket\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n\" (UID: \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n"
Nov 01 03:27:26.095116 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:26.095095    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-workload-socket\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n\" (UID: \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n"
Nov 01 03:27:26.095263 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:26.095184    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-workload-certs\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n\" (UID: \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n"
Nov 01 03:27:26.097664 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:26.097619    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-istio-envoy\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n\" (UID: \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n"
Nov 01 03:27:26.097859 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:26.097833    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-istio-podinfo\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n\" (UID: \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n"
Nov 01 03:27:26.103078 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:26.103052    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-istiod-ca-cert\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n\" (UID: \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n"
Nov 01 03:27:26.153810 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:26.153784    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-istio-token\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n\" (UID: \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n"
Nov 01 03:27:26.153969 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:26.153929    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kd2w\" (UniqueName: \"kubernetes.io/projected/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-kube-api-access-5kd2w\") pod \"e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n\" (UID: \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\") " pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n"
Nov 01 03:27:26.212355 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:26.212316    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n"
Nov 01 03:27:26.626947 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:26.626848    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n"]
Nov 01 03:27:26.629624 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:27:26.629536    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72c2d7cf_fac4_4d80_a5f1_82ae300a255a.slice/crio-5448b858764ca320402cd209fc052704bff5b28df25aac1434dde5dbd46d84fa WatchSource:0}: Error finding container 5448b858764ca320402cd209fc052704bff5b28df25aac1434dde5dbd46d84fa: Status 404 returned error can't find the container with id 5448b858764ca320402cd209fc052704bff5b28df25aac1434dde5dbd46d84fa
Nov 01 03:27:26.633071 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:26.633023    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:27:26.633264 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:26.633245    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:27:26.633377 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:26.633363    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:27:26.671180 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:26.671150    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d87394e-pgwwl" event={"ID":"5211cbab-e13b-4e4e-899d-3195a8c8ce23","Type":"ContainerStarted","Data":"4bf3ff6d81c4497ca3407757663dc755565167c97960239edbddc30e2117e273"}
Nov 01 03:27:26.671600 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:26.671542    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d87394e-pgwwl" event={"ID":"5211cbab-e13b-4e4e-899d-3195a8c8ce23","Type":"ContainerStarted","Data":"5df996f3af8c1394fb2eccf7ff1f4e1b461d0cd5d8d843cb405b6b0e035876e5"}
Nov 01 03:27:26.673635 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:26.673613    4930 generic.go:356] "Generic (PLEG): container finished" podID="e7b6d3eb-e0cf-40b9-bfa2-e543626883fa" containerID="68db2c2c473cc995b6d6dee1d2994a64b642a01311a6daf14e0679a803c9baa4" exitCode=0
Nov 01 03:27:26.673752 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:26.673707    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ab1571-h4jqz" event={"ID":"e7b6d3eb-e0cf-40b9-bfa2-e543626883fa","Type":"ContainerDied","Data":"68db2c2c473cc995b6d6dee1d2994a64b642a01311a6daf14e0679a803c9baa4"}
Nov 01 03:27:26.675316 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:26.675295    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n" event={"ID":"72c2d7cf-fac4-4d80-a5f1-82ae300a255a","Type":"ContainerStarted","Data":"5448b858764ca320402cd209fc052704bff5b28df25aac1434dde5dbd46d84fa"}
Nov 01 03:27:27.149740 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:27.149681    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d87394e-pgwwl" podStartSLOduration=2.149666559 podStartE2EDuration="2.149666559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:27:27.148218241 +0000 UTC m=+1251.671224832" watchObservedRunningTime="2026-11-01 03:27:27.149666559 +0000 UTC m=+1251.672673146"
Nov 01 03:27:27.699876 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:27.699841    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n" event={"ID":"72c2d7cf-fac4-4d80-a5f1-82ae300a255a","Type":"ContainerStarted","Data":"dbf783d89d9705308d560551d328a4077052c6f3944362a3803f829548325208"}
Nov 01 03:27:27.702933 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:27.702900    4930 generic.go:356] "Generic (PLEG): container finished" podID="5211cbab-e13b-4e4e-899d-3195a8c8ce23" containerID="4bf3ff6d81c4497ca3407757663dc755565167c97960239edbddc30e2117e273" exitCode=0
Nov 01 03:27:27.703047 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:27.702991    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d87394e-pgwwl" event={"ID":"5211cbab-e13b-4e4e-899d-3195a8c8ce23","Type":"ContainerDied","Data":"4bf3ff6d81c4497ca3407757663dc755565167c97960239edbddc30e2117e273"}
Nov 01 03:27:27.824626 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:27.821772    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n" podStartSLOduration=2.821753683 podStartE2EDuration="2.821753683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:27:27.821234135 +0000 UTC m=+1252.344240737" watchObservedRunningTime="2026-11-01 03:27:27.821753683 +0000 UTC m=+1252.344760271"
Nov 01 03:27:27.934656 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:27.934631    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ab1571-h4jqz"
Nov 01 03:27:28.025489 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:28.025401    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7b6d3eb-e0cf-40b9-bfa2-e543626883fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7b6d3eb-e0cf-40b9-bfa2-e543626883fa-maas-api-xks-ca\") pod \"e7b6d3eb-e0cf-40b9-bfa2-e543626883fa\" (UID: \"e7b6d3eb-e0cf-40b9-bfa2-e543626883fa\") "
Nov 01 03:27:28.025659 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:28.025481    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7b6d3eb-e0cf-40b9-bfa2-e543626883fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7b6d3eb-e0cf-40b9-bfa2-e543626883fa-maas-api-openshift-ca\") pod \"e7b6d3eb-e0cf-40b9-bfa2-e543626883fa\" (UID: \"e7b6d3eb-e0cf-40b9-bfa2-e543626883fa\") "
Nov 01 03:27:28.025795 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:28.025768    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7b6d3eb-e0cf-40b9-bfa2-e543626883fa-kube-api-access-d54xq\" (UniqueName: \"kubernetes.io/projected/e7b6d3eb-e0cf-40b9-bfa2-e543626883fa-kube-api-access-d54xq\") pod \"e7b6d3eb-e0cf-40b9-bfa2-e543626883fa\" (UID: \"e7b6d3eb-e0cf-40b9-bfa2-e543626883fa\") "
Nov 01 03:27:28.025895 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:28.025875    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7b6d3eb-e0cf-40b9-bfa2-e543626883fa-maas-api-openshift-ca" pod "e7b6d3eb-e0cf-40b9-bfa2-e543626883fa" (UID: "e7b6d3eb-e0cf-40b9-bfa2-e543626883fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:27:28.026354 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:28.026331    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7b6d3eb-e0cf-40b9-bfa2-e543626883fa-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:27:28.027863 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:28.027835    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7b6d3eb-e0cf-40b9-bfa2-e543626883fa-kube-api-access-d54xq" pod "e7b6d3eb-e0cf-40b9-bfa2-e543626883fa" (UID: "e7b6d3eb-e0cf-40b9-bfa2-e543626883fa"). InnerVolumeSpecName "kube-api-access-d54xq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:27:28.028317 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:28.028297    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7b6d3eb-e0cf-40b9-bfa2-e543626883fa-maas-api-xks-ca" pod "e7b6d3eb-e0cf-40b9-bfa2-e543626883fa" (UID: "e7b6d3eb-e0cf-40b9-bfa2-e543626883fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:27:28.128124 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:28.128080    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7b6d3eb-e0cf-40b9-bfa2-e543626883fa-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:27:28.128124 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:28.128116    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d54xq\" (UniqueName: \"kubernetes.io/projected/e7b6d3eb-e0cf-40b9-bfa2-e543626883fa-kube-api-access-d54xq\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:27:28.145322 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:28.145291    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:27:28.223855 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:28.216065    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n"
Nov 01 03:27:28.223855 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:28.223384    4930 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Nov 01 03:27:28.223855 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:28.223462    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n" podUID="72c2d7cf-fac4-4d80-a5f1-82ae300a255a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Nov 01 03:27:28.713841 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:28.713750    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ab1571-h4jqz"
Nov 01 03:27:28.713841 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:28.713756    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40ab1571-h4jqz" event={"ID":"e7b6d3eb-e0cf-40b9-bfa2-e543626883fa","Type":"ContainerDied","Data":"716ed6760eba909589d9d8267d81ecad1402436b4e81f0a1c44ee90366ef0508"}
Nov 01 03:27:28.714373 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:28.713888    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="716ed6760eba909589d9d8267d81ecad1402436b4e81f0a1c44ee90366ef0508"
Nov 01 03:27:28.936856 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:28.936832    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d87394e-pgwwl"
Nov 01 03:27:29.040958 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:29.040879    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5211cbab-e13b-4e4e-899d-3195a8c8ce23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5211cbab-e13b-4e4e-899d-3195a8c8ce23-maas-api-openshift-ca\") pod \"5211cbab-e13b-4e4e-899d-3195a8c8ce23\" (UID: \"5211cbab-e13b-4e4e-899d-3195a8c8ce23\") "
Nov 01 03:27:29.041091 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:29.040968    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5211cbab-e13b-4e4e-899d-3195a8c8ce23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5211cbab-e13b-4e4e-899d-3195a8c8ce23-maas-api-xks-ca\") pod \"5211cbab-e13b-4e4e-899d-3195a8c8ce23\" (UID: \"5211cbab-e13b-4e4e-899d-3195a8c8ce23\") "
Nov 01 03:27:29.041091 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:29.041006    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5211cbab-e13b-4e4e-899d-3195a8c8ce23-kube-api-access-bfp7l\" (UniqueName: \"kubernetes.io/projected/5211cbab-e13b-4e4e-899d-3195a8c8ce23-kube-api-access-bfp7l\") pod \"5211cbab-e13b-4e4e-899d-3195a8c8ce23\" (UID: \"5211cbab-e13b-4e4e-899d-3195a8c8ce23\") "
Nov 01 03:27:29.041281 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:29.041252    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5211cbab-e13b-4e4e-899d-3195a8c8ce23-maas-api-openshift-ca" pod "5211cbab-e13b-4e4e-899d-3195a8c8ce23" (UID: "5211cbab-e13b-4e4e-899d-3195a8c8ce23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:27:29.041650 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:29.041616    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5211cbab-e13b-4e4e-899d-3195a8c8ce23-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:27:29.043115 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:29.043092    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5211cbab-e13b-4e4e-899d-3195a8c8ce23-maas-api-xks-ca" pod "5211cbab-e13b-4e4e-899d-3195a8c8ce23" (UID: "5211cbab-e13b-4e4e-899d-3195a8c8ce23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:27:29.043214 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:29.043142    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5211cbab-e13b-4e4e-899d-3195a8c8ce23-kube-api-access-bfp7l" pod "5211cbab-e13b-4e4e-899d-3195a8c8ce23" (UID: "5211cbab-e13b-4e4e-899d-3195a8c8ce23"). InnerVolumeSpecName "kube-api-access-bfp7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:27:29.143123 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:29.143086    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfp7l\" (UniqueName: \"kubernetes.io/projected/5211cbab-e13b-4e4e-899d-3195a8c8ce23-kube-api-access-bfp7l\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:27:29.143123 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:29.143120    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5211cbab-e13b-4e4e-899d-3195a8c8ce23-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:27:29.213294 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:29.213255    4930 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Nov 01 03:27:29.213446 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:29.213321    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n" podUID="72c2d7cf-fac4-4d80-a5f1-82ae300a255a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Nov 01 03:27:29.722849 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:29.722818    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d87394e-pgwwl"
Nov 01 03:27:29.723224 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:29.722842    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d87394e-pgwwl" event={"ID":"5211cbab-e13b-4e4e-899d-3195a8c8ce23","Type":"ContainerDied","Data":"5df996f3af8c1394fb2eccf7ff1f4e1b461d0cd5d8d843cb405b6b0e035876e5"}
Nov 01 03:27:29.723224 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:29.722894    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5df996f3af8c1394fb2eccf7ff1f4e1b461d0cd5d8d843cb405b6b0e035876e5"
Nov 01 03:27:30.051056 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:30.050959    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fba185c6-kdrjn"]
Nov 01 03:27:30.103998 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:30.103960    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fba185c6-kdrjn"]
Nov 01 03:27:30.212739 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:30.212705    4930 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Nov 01 03:27:30.212894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:30.212756    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n" podUID="72c2d7cf-fac4-4d80-a5f1-82ae300a255a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Nov 01 03:27:30.703975 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:30.703938    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:27:31.213317 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:31.213281    4930 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Nov 01 03:27:31.213792 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:31.213372    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n" podUID="72c2d7cf-fac4-4d80-a5f1-82ae300a255a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Nov 01 03:27:31.625698 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:31.625648    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ea59bb2-5f2c-439a-8407-55c99376a22b" path="/var/lib/kubelet/pods/8ea59bb2-5f2c-439a-8407-55c99376a22b/volumes"
Nov 01 03:27:32.212735 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:32.212693    4930 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Nov 01 03:27:32.212893 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:32.212749    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n" podUID="72c2d7cf-fac4-4d80-a5f1-82ae300a255a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Nov 01 03:27:32.752784 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:32.751334    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:27:33.072392 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:33.072285    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:27:33.213095 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:33.213048    4930 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Nov 01 03:27:33.213302 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:33.213158    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n" podUID="72c2d7cf-fac4-4d80-a5f1-82ae300a255a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Nov 01 03:27:34.217324 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:34.217295    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n"
Nov 01 03:27:34.217771 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:34.217733    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n"
Nov 01 03:27:34.218541 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:34.218521    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n"
Nov 01 03:27:35.733062 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:35.733029    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9134a935-tb9nz"]
Nov 01 03:27:35.733648 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:35.733631    4930 cpu_manager.go:414] "containerMap: removing container" podUID="5211cbab-e13b-4e4e-899d-3195a8c8ce23" containerName="revoke-keys"
Nov 01 03:27:35.733701 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:35.733650    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5211cbab-e13b-4e4e-899d-3195a8c8ce23" containerName="revoke-keys"
Nov 01 03:27:35.733701 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:35.733681    4930 cpu_manager.go:414] "containerMap: removing container" podUID="e7b6d3eb-e0cf-40b9-bfa2-e543626883fa" containerName="revoke-keys"
Nov 01 03:27:35.733701 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:35.733686    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7b6d3eb-e0cf-40b9-bfa2-e543626883fa" containerName="revoke-keys"
Nov 01 03:27:35.733802 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:35.733791    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="5211cbab-e13b-4e4e-899d-3195a8c8ce23" containerName="revoke-keys"
Nov 01 03:27:35.733836 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:35.733807    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7b6d3eb-e0cf-40b9-bfa2-e543626883fa" containerName="revoke-keys"
Nov 01 03:27:35.738691 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:35.738672    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9134a935-tb9nz"
Nov 01 03:27:35.793823 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:35.793798    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:27:35.793963 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:35.793905    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:27:35.794012 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:35.793971    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:27:35.796011 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:35.795992    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9134a935-tb9nz"]
Nov 01 03:27:35.839688 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:35.839661    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:27:35.848400 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:35.848377    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bd986dd-fk4kj"]
Nov 01 03:27:35.852517 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:35.852500    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bd986dd-fk4kj"
Nov 01 03:27:35.865492 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:35.865471    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82e946ea-cee9-4205-bace-d5132fc6b754-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9134a935-tb9nz\" (UID: \"82e946ea-cee9-4205-bace-d5132fc6b754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9134a935-tb9nz"
Nov 01 03:27:35.865796 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:35.865771    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82e946ea-cee9-4205-bace-d5132fc6b754-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9134a935-tb9nz\" (UID: \"82e946ea-cee9-4205-bace-d5132fc6b754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9134a935-tb9nz"
Nov 01 03:27:35.865998 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:35.865969    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks4bf\" (UniqueName: \"kubernetes.io/projected/82e946ea-cee9-4205-bace-d5132fc6b754-kube-api-access-ks4bf\") pod \"maas-api-revoke-sub-9134a935-tb9nz\" (UID: \"82e946ea-cee9-4205-bace-d5132fc6b754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9134a935-tb9nz"
Nov 01 03:27:35.882906 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:35.882885    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bd986dd-fk4kj"]
Nov 01 03:27:35.908433 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:35.908403    4930 scope.go:122] "RemoveContainer" containerID="f5228339a7b58360f4a9cfbb872e38f56983eaf8d17cc00a2f340df45d9d4e2b"
Nov 01 03:27:35.919172 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:35.919151    4930 scope.go:122] "RemoveContainer" containerID="c3c91b4814cbf9a3a9c51b4d93c694ba0f59679e109119cbbcda8cb9ead4de0e"
Nov 01 03:27:35.928193 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:35.928173    4930 scope.go:122] "RemoveContainer" containerID="10560f21e02865ea1d48acaa51904b8ad35f57151d6014554bf51fc3911d6144"
Nov 01 03:27:35.937376 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:35.937358    4930 scope.go:122] "RemoveContainer" containerID="4e771283049020b6524d0bbd77c6ea1b3a3e2ebf352c375dee17b055f325727a"
Nov 01 03:27:35.967138 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:35.967116    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dc62e14-76de-4a5a-bb5c-0d79b591be39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bd986dd-fk4kj\" (UID: \"2dc62e14-76de-4a5a-bb5c-0d79b591be39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bd986dd-fk4kj"
Nov 01 03:27:35.967271 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:35.967250    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82e946ea-cee9-4205-bace-d5132fc6b754-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9134a935-tb9nz\" (UID: \"82e946ea-cee9-4205-bace-d5132fc6b754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9134a935-tb9nz"
Nov 01 03:27:35.967325 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:35.967310    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dc62e14-76de-4a5a-bb5c-0d79b591be39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bd986dd-fk4kj\" (UID: \"2dc62e14-76de-4a5a-bb5c-0d79b591be39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bd986dd-fk4kj"
Nov 01 03:27:35.967372 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:35.967342    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljljf\" (UniqueName: \"kubernetes.io/projected/2dc62e14-76de-4a5a-bb5c-0d79b591be39-kube-api-access-ljljf\") pod \"maas-api-revoke-sub-7bd986dd-fk4kj\" (UID: \"2dc62e14-76de-4a5a-bb5c-0d79b591be39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bd986dd-fk4kj"
Nov 01 03:27:35.967439 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:35.967419    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks4bf\" (UniqueName: \"kubernetes.io/projected/82e946ea-cee9-4205-bace-d5132fc6b754-kube-api-access-ks4bf\") pod \"maas-api-revoke-sub-9134a935-tb9nz\" (UID: \"82e946ea-cee9-4205-bace-d5132fc6b754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9134a935-tb9nz"
Nov 01 03:27:35.967515 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:35.967496    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82e946ea-cee9-4205-bace-d5132fc6b754-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9134a935-tb9nz\" (UID: \"82e946ea-cee9-4205-bace-d5132fc6b754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9134a935-tb9nz"
Nov 01 03:27:35.968212 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:35.968190    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82e946ea-cee9-4205-bace-d5132fc6b754-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9134a935-tb9nz\" (UID: \"82e946ea-cee9-4205-bace-d5132fc6b754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9134a935-tb9nz"
Nov 01 03:27:35.969896 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:35.969880    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82e946ea-cee9-4205-bace-d5132fc6b754-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9134a935-tb9nz\" (UID: \"82e946ea-cee9-4205-bace-d5132fc6b754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9134a935-tb9nz"
Nov 01 03:27:36.028919 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:36.028862    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks4bf\" (UniqueName: \"kubernetes.io/projected/82e946ea-cee9-4205-bace-d5132fc6b754-kube-api-access-ks4bf\") pod \"maas-api-revoke-sub-9134a935-tb9nz\" (UID: \"82e946ea-cee9-4205-bace-d5132fc6b754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9134a935-tb9nz"
Nov 01 03:27:36.051669 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:36.051640    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c3bbbf0-flzbv"]
Nov 01 03:27:36.052968 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:36.052943    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9134a935-tb9nz"
Nov 01 03:27:36.056471 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:36.056447    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c3bbbf0-flzbv"]
Nov 01 03:27:36.069493 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:36.069467    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dc62e14-76de-4a5a-bb5c-0d79b591be39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bd986dd-fk4kj\" (UID: \"2dc62e14-76de-4a5a-bb5c-0d79b591be39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bd986dd-fk4kj"
Nov 01 03:27:36.069652 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:36.069635    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dc62e14-76de-4a5a-bb5c-0d79b591be39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bd986dd-fk4kj\" (UID: \"2dc62e14-76de-4a5a-bb5c-0d79b591be39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bd986dd-fk4kj"
Nov 01 03:27:36.069716 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:36.069672    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljljf\" (UniqueName: \"kubernetes.io/projected/2dc62e14-76de-4a5a-bb5c-0d79b591be39-kube-api-access-ljljf\") pod \"maas-api-revoke-sub-7bd986dd-fk4kj\" (UID: \"2dc62e14-76de-4a5a-bb5c-0d79b591be39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bd986dd-fk4kj"
Nov 01 03:27:36.070194 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:36.070173    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dc62e14-76de-4a5a-bb5c-0d79b591be39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bd986dd-fk4kj\" (UID: \"2dc62e14-76de-4a5a-bb5c-0d79b591be39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bd986dd-fk4kj"
Nov 01 03:27:36.072095 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:36.072071    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dc62e14-76de-4a5a-bb5c-0d79b591be39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bd986dd-fk4kj\" (UID: \"2dc62e14-76de-4a5a-bb5c-0d79b591be39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bd986dd-fk4kj"
Nov 01 03:27:36.142732 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:36.137682    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljljf\" (UniqueName: \"kubernetes.io/projected/2dc62e14-76de-4a5a-bb5c-0d79b591be39-kube-api-access-ljljf\") pod \"maas-api-revoke-sub-7bd986dd-fk4kj\" (UID: \"2dc62e14-76de-4a5a-bb5c-0d79b591be39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bd986dd-fk4kj"
Nov 01 03:27:36.167815 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:36.167787    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bd986dd-fk4kj"
Nov 01 03:27:36.436134 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:27:36.436105    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82e946ea_cee9_4205_bace_d5132fc6b754.slice/crio-6686483214d8d956bf6a4387dc94756ccd6e00655aa2b4e71a57e6d2f4cd6706 WatchSource:0}: Error finding container 6686483214d8d956bf6a4387dc94756ccd6e00655aa2b4e71a57e6d2f4cd6706: Status 404 returned error can't find the container with id 6686483214d8d956bf6a4387dc94756ccd6e00655aa2b4e71a57e6d2f4cd6706
Nov 01 03:27:36.438027 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:36.438001    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9134a935-tb9nz"]
Nov 01 03:27:36.551825 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:27:36.551800    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dc62e14_76de_4a5a_bb5c_0d79b591be39.slice/crio-e4827268b7c953ca0d4187ded03aafd931e843de679c3d7c471a5402ab63328b WatchSource:0}: Error finding container e4827268b7c953ca0d4187ded03aafd931e843de679c3d7c471a5402ab63328b: Status 404 returned error can't find the container with id e4827268b7c953ca0d4187ded03aafd931e843de679c3d7c471a5402ab63328b
Nov 01 03:27:36.554921 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:36.554897    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bd986dd-fk4kj"]
Nov 01 03:27:36.789374 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:36.789344    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9134a935-tb9nz" event={"ID":"82e946ea-cee9-4205-bace-d5132fc6b754","Type":"ContainerStarted","Data":"6686483214d8d956bf6a4387dc94756ccd6e00655aa2b4e71a57e6d2f4cd6706"}
Nov 01 03:27:36.790866 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:36.790841    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bd986dd-fk4kj" event={"ID":"2dc62e14-76de-4a5a-bb5c-0d79b591be39","Type":"ContainerStarted","Data":"e4827268b7c953ca0d4187ded03aafd931e843de679c3d7c471a5402ab63328b"}
Nov 01 03:27:37.632188 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:37.632153    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da0448da-122c-47c5-aa81-b3c5cc5193c3" path="/var/lib/kubelet/pods/da0448da-122c-47c5-aa81-b3c5cc5193c3/volumes"
Nov 01 03:27:37.801042 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:37.800960    4930 generic.go:356] "Generic (PLEG): container finished" podID="2dc62e14-76de-4a5a-bb5c-0d79b591be39" containerID="587ff8897f2cea2f537c74d13a8538eac6f033e9c70ee5f1428ffcd5577b0afd" exitCode=0
Nov 01 03:27:37.801467 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:37.801046    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bd986dd-fk4kj" event={"ID":"2dc62e14-76de-4a5a-bb5c-0d79b591be39","Type":"ContainerDied","Data":"587ff8897f2cea2f537c74d13a8538eac6f033e9c70ee5f1428ffcd5577b0afd"}
Nov 01 03:27:37.802937 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:37.802911    4930 generic.go:356] "Generic (PLEG): container finished" podID="82e946ea-cee9-4205-bace-d5132fc6b754" containerID="53aee18bd15c501c67142113f3a365221036327971ceb60cdb3d4a3df51a1574" exitCode=0
Nov 01 03:27:37.803070 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:37.802965    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9134a935-tb9nz" event={"ID":"82e946ea-cee9-4205-bace-d5132fc6b754","Type":"ContainerDied","Data":"53aee18bd15c501c67142113f3a365221036327971ceb60cdb3d4a3df51a1574"}
Nov 01 03:27:38.145794 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:38.145758    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:27:39.007988 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:39.007963    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bd986dd-fk4kj"
Nov 01 03:27:39.010669 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:39.010648    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9134a935-tb9nz"
Nov 01 03:27:39.027089 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:39.027060    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82e946ea-cee9-4205-bace-d5132fc6b754-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82e946ea-cee9-4205-bace-d5132fc6b754-maas-api-openshift-ca\") pod \"82e946ea-cee9-4205-bace-d5132fc6b754\" (UID: \"82e946ea-cee9-4205-bace-d5132fc6b754\") "
Nov 01 03:27:39.027217 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:39.027131    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dc62e14-76de-4a5a-bb5c-0d79b591be39-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dc62e14-76de-4a5a-bb5c-0d79b591be39-maas-api-openshift-ca\") pod \"2dc62e14-76de-4a5a-bb5c-0d79b591be39\" (UID: \"2dc62e14-76de-4a5a-bb5c-0d79b591be39\") "
Nov 01 03:27:39.027217 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:39.027196    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82e946ea-cee9-4205-bace-d5132fc6b754-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82e946ea-cee9-4205-bace-d5132fc6b754-maas-api-xks-ca\") pod \"82e946ea-cee9-4205-bace-d5132fc6b754\" (UID: \"82e946ea-cee9-4205-bace-d5132fc6b754\") "
Nov 01 03:27:39.027404 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:39.027232    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dc62e14-76de-4a5a-bb5c-0d79b591be39-kube-api-access-ljljf\" (UniqueName: \"kubernetes.io/projected/2dc62e14-76de-4a5a-bb5c-0d79b591be39-kube-api-access-ljljf\") pod \"2dc62e14-76de-4a5a-bb5c-0d79b591be39\" (UID: \"2dc62e14-76de-4a5a-bb5c-0d79b591be39\") "
Nov 01 03:27:39.027404 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:39.027316    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dc62e14-76de-4a5a-bb5c-0d79b591be39-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dc62e14-76de-4a5a-bb5c-0d79b591be39-maas-api-xks-ca\") pod \"2dc62e14-76de-4a5a-bb5c-0d79b591be39\" (UID: \"2dc62e14-76de-4a5a-bb5c-0d79b591be39\") "
Nov 01 03:27:39.027518 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:39.027427    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82e946ea-cee9-4205-bace-d5132fc6b754-maas-api-openshift-ca" pod "82e946ea-cee9-4205-bace-d5132fc6b754" (UID: "82e946ea-cee9-4205-bace-d5132fc6b754"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:27:39.027518 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:39.027449    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dc62e14-76de-4a5a-bb5c-0d79b591be39-maas-api-openshift-ca" pod "2dc62e14-76de-4a5a-bb5c-0d79b591be39" (UID: "2dc62e14-76de-4a5a-bb5c-0d79b591be39"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:27:39.027518 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:39.027491    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82e946ea-cee9-4205-bace-d5132fc6b754-kube-api-access-ks4bf\" (UniqueName: \"kubernetes.io/projected/82e946ea-cee9-4205-bace-d5132fc6b754-kube-api-access-ks4bf\") pod \"82e946ea-cee9-4205-bace-d5132fc6b754\" (UID: \"82e946ea-cee9-4205-bace-d5132fc6b754\") "
Nov 01 03:27:39.028124 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:39.028062    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82e946ea-cee9-4205-bace-d5132fc6b754-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:27:39.028124 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:39.028087    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dc62e14-76de-4a5a-bb5c-0d79b591be39-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:27:39.030804 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:39.030700    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82e946ea-cee9-4205-bace-d5132fc6b754-maas-api-xks-ca" pod "82e946ea-cee9-4205-bace-d5132fc6b754" (UID: "82e946ea-cee9-4205-bace-d5132fc6b754"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:27:39.030804 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:39.030713    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dc62e14-76de-4a5a-bb5c-0d79b591be39-maas-api-xks-ca" pod "2dc62e14-76de-4a5a-bb5c-0d79b591be39" (UID: "2dc62e14-76de-4a5a-bb5c-0d79b591be39"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:27:39.030804 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:39.030733    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dc62e14-76de-4a5a-bb5c-0d79b591be39-kube-api-access-ljljf" pod "2dc62e14-76de-4a5a-bb5c-0d79b591be39" (UID: "2dc62e14-76de-4a5a-bb5c-0d79b591be39"). InnerVolumeSpecName "kube-api-access-ljljf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:27:39.030804 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:39.030762    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82e946ea-cee9-4205-bace-d5132fc6b754-kube-api-access-ks4bf" pod "82e946ea-cee9-4205-bace-d5132fc6b754" (UID: "82e946ea-cee9-4205-bace-d5132fc6b754"). InnerVolumeSpecName "kube-api-access-ks4bf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:27:39.129410 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:39.129381    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dc62e14-76de-4a5a-bb5c-0d79b591be39-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:27:39.129410 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:39.129409    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks4bf\" (UniqueName: \"kubernetes.io/projected/82e946ea-cee9-4205-bace-d5132fc6b754-kube-api-access-ks4bf\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:27:39.129636 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:39.129422    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82e946ea-cee9-4205-bace-d5132fc6b754-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:27:39.129636 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:39.129431    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljljf\" (UniqueName: \"kubernetes.io/projected/2dc62e14-76de-4a5a-bb5c-0d79b591be39-kube-api-access-ljljf\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:27:39.821512 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:39.821478    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9134a935-tb9nz"
Nov 01 03:27:39.821713 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:39.821483    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9134a935-tb9nz" event={"ID":"82e946ea-cee9-4205-bace-d5132fc6b754","Type":"ContainerDied","Data":"6686483214d8d956bf6a4387dc94756ccd6e00655aa2b4e71a57e6d2f4cd6706"}
Nov 01 03:27:39.821713 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:39.821673    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6686483214d8d956bf6a4387dc94756ccd6e00655aa2b4e71a57e6d2f4cd6706"
Nov 01 03:27:39.824120 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:39.824093    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bd986dd-fk4kj" event={"ID":"2dc62e14-76de-4a5a-bb5c-0d79b591be39","Type":"ContainerDied","Data":"e4827268b7c953ca0d4187ded03aafd931e843de679c3d7c471a5402ab63328b"}
Nov 01 03:27:39.824271 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:39.824137    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4827268b7c953ca0d4187ded03aafd931e843de679c3d7c471a5402ab63328b"
Nov 01 03:27:39.824271 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:39.824204    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bd986dd-fk4kj"
Nov 01 03:27:39.936165 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:39.936085    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faa90817-9lvwc"]
Nov 01 03:27:39.937039 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:39.937018    4930 cpu_manager.go:414] "containerMap: removing container" podUID="82e946ea-cee9-4205-bace-d5132fc6b754" containerName="revoke-keys"
Nov 01 03:27:39.937105 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:39.937042    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82e946ea-cee9-4205-bace-d5132fc6b754" containerName="revoke-keys"
Nov 01 03:27:39.937105 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:39.937074    4930 cpu_manager.go:414] "containerMap: removing container" podUID="2dc62e14-76de-4a5a-bb5c-0d79b591be39" containerName="revoke-keys"
Nov 01 03:27:39.937105 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:39.937083    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dc62e14-76de-4a5a-bb5c-0d79b591be39" containerName="revoke-keys"
Nov 01 03:27:39.937270 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:39.937258    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="82e946ea-cee9-4205-bace-d5132fc6b754" containerName="revoke-keys"
Nov 01 03:27:39.937306 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:39.937278    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dc62e14-76de-4a5a-bb5c-0d79b591be39" containerName="revoke-keys"
Nov 01 03:27:39.941651 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:39.941623    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faa90817-9lvwc"
Nov 01 03:27:39.995172 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:39.995145    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:27:39.995316 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:39.995252    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:27:39.995376 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:39.995341    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:27:39.995817 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:39.995404    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:27:40.002109 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:40.002081    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faa90817-9lvwc"]
Nov 01 03:27:40.043320 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:40.043283    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19139bc8-a17f-4093-b596-4ac8225e260c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-faa90817-9lvwc\" (UID: \"19139bc8-a17f-4093-b596-4ac8225e260c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faa90817-9lvwc"
Nov 01 03:27:40.043710 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:40.043464    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19139bc8-a17f-4093-b596-4ac8225e260c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-faa90817-9lvwc\" (UID: \"19139bc8-a17f-4093-b596-4ac8225e260c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faa90817-9lvwc"
Nov 01 03:27:40.043710 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:40.043506    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9slpm\" (UniqueName: \"kubernetes.io/projected/19139bc8-a17f-4093-b596-4ac8225e260c-kube-api-access-9slpm\") pod \"maas-api-revoke-sub-faa90817-9lvwc\" (UID: \"19139bc8-a17f-4093-b596-4ac8225e260c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faa90817-9lvwc"
Nov 01 03:27:40.055714 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:40.055691    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9305848-8fqmz"]
Nov 01 03:27:40.060354 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:40.060329    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9305848-8fqmz"]
Nov 01 03:27:40.144528 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:40.144497    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19139bc8-a17f-4093-b596-4ac8225e260c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-faa90817-9lvwc\" (UID: \"19139bc8-a17f-4093-b596-4ac8225e260c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faa90817-9lvwc"
Nov 01 03:27:40.144673 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:40.144654    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19139bc8-a17f-4093-b596-4ac8225e260c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-faa90817-9lvwc\" (UID: \"19139bc8-a17f-4093-b596-4ac8225e260c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faa90817-9lvwc"
Nov 01 03:27:40.144738 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:40.144704    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9slpm\" (UniqueName: \"kubernetes.io/projected/19139bc8-a17f-4093-b596-4ac8225e260c-kube-api-access-9slpm\") pod \"maas-api-revoke-sub-faa90817-9lvwc\" (UID: \"19139bc8-a17f-4093-b596-4ac8225e260c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faa90817-9lvwc"
Nov 01 03:27:40.145414 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:40.145389    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19139bc8-a17f-4093-b596-4ac8225e260c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-faa90817-9lvwc\" (UID: \"19139bc8-a17f-4093-b596-4ac8225e260c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faa90817-9lvwc"
Nov 01 03:27:40.146817 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:40.146800    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19139bc8-a17f-4093-b596-4ac8225e260c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-faa90817-9lvwc\" (UID: \"19139bc8-a17f-4093-b596-4ac8225e260c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faa90817-9lvwc"
Nov 01 03:27:40.207402 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:40.207347    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9slpm\" (UniqueName: \"kubernetes.io/projected/19139bc8-a17f-4093-b596-4ac8225e260c-kube-api-access-9slpm\") pod \"maas-api-revoke-sub-faa90817-9lvwc\" (UID: \"19139bc8-a17f-4093-b596-4ac8225e260c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faa90817-9lvwc"
Nov 01 03:27:40.257362 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:40.257340    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faa90817-9lvwc"
Nov 01 03:27:40.648287 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:40.648259    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faa90817-9lvwc"]
Nov 01 03:27:40.650548 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:27:40.650521    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19139bc8_a17f_4093_b596_4ac8225e260c.slice/crio-ea6f7efbd43d105e59526170a9b4d27a26016e9d87bd9f1816d12c8199d892db WatchSource:0}: Error finding container ea6f7efbd43d105e59526170a9b4d27a26016e9d87bd9f1816d12c8199d892db: Status 404 returned error can't find the container with id ea6f7efbd43d105e59526170a9b4d27a26016e9d87bd9f1816d12c8199d892db
Nov 01 03:27:40.835055 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:40.835023    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faa90817-9lvwc" event={"ID":"19139bc8-a17f-4093-b596-4ac8225e260c","Type":"ContainerStarted","Data":"ea6f7efbd43d105e59526170a9b4d27a26016e9d87bd9f1816d12c8199d892db"}
Nov 01 03:27:41.622650 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:41.622616    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9008621-ca43-429a-9d79-7bcf46ebbd55" path="/var/lib/kubelet/pods/b9008621-ca43-429a-9d79-7bcf46ebbd55/volumes"
Nov 01 03:27:41.844504 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:41.844469    4930 generic.go:356] "Generic (PLEG): container finished" podID="19139bc8-a17f-4093-b596-4ac8225e260c" containerID="0c607acf23828e0b0a5f054c8b275a806b69a186aed3972a492d090900f8a4e5" exitCode=0
Nov 01 03:27:41.844698 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:41.844550    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faa90817-9lvwc" event={"ID":"19139bc8-a17f-4093-b596-4ac8225e260c","Type":"ContainerDied","Data":"0c607acf23828e0b0a5f054c8b275a806b69a186aed3972a492d090900f8a4e5"}
Nov 01 03:27:41.959733 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:41.959706    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl"]
Nov 01 03:27:41.964088 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:41.964065    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl"
Nov 01 03:27:42.059630 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.059593    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl"]
Nov 01 03:27:42.069963 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.069928    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c02ff41-a095-4c49-a78c-c78f920a42a4-istio-podinfo\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl\" (UID: \"8c02ff41-a095-4c49-a78c-c78f920a42a4\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl"
Nov 01 03:27:42.070103 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.070055    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c02ff41-a095-4c49-a78c-c78f920a42a4-istiod-ca-cert\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl\" (UID: \"8c02ff41-a095-4c49-a78c-c78f920a42a4\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl"
Nov 01 03:27:42.070146 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.070116    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c02ff41-a095-4c49-a78c-c78f920a42a4-workload-socket\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl\" (UID: \"8c02ff41-a095-4c49-a78c-c78f920a42a4\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl"
Nov 01 03:27:42.070184 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.070157    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c02ff41-a095-4c49-a78c-c78f920a42a4-istio-envoy\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl\" (UID: \"8c02ff41-a095-4c49-a78c-c78f920a42a4\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl"
Nov 01 03:27:42.070294 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.070273    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c02ff41-a095-4c49-a78c-c78f920a42a4-istio-token\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl\" (UID: \"8c02ff41-a095-4c49-a78c-c78f920a42a4\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl"
Nov 01 03:27:42.070390 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.070372    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c02ff41-a095-4c49-a78c-c78f920a42a4-workload-certs\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl\" (UID: \"8c02ff41-a095-4c49-a78c-c78f920a42a4\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl"
Nov 01 03:27:42.070435 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.070408    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcqmj\" (UniqueName: \"kubernetes.io/projected/8c02ff41-a095-4c49-a78c-c78f920a42a4-kube-api-access-vcqmj\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl\" (UID: \"8c02ff41-a095-4c49-a78c-c78f920a42a4\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl"
Nov 01 03:27:42.070513 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.070497    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c02ff41-a095-4c49-a78c-c78f920a42a4-istio-data\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl\" (UID: \"8c02ff41-a095-4c49-a78c-c78f920a42a4\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl"
Nov 01 03:27:42.071734 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.071702    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c02ff41-a095-4c49-a78c-c78f920a42a4-credential-socket\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl\" (UID: \"8c02ff41-a095-4c49-a78c-c78f920a42a4\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl"
Nov 01 03:27:42.172955 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.172867    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcqmj\" (UniqueName: \"kubernetes.io/projected/8c02ff41-a095-4c49-a78c-c78f920a42a4-kube-api-access-vcqmj\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl\" (UID: \"8c02ff41-a095-4c49-a78c-c78f920a42a4\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl"
Nov 01 03:27:42.173525 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.173497    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c02ff41-a095-4c49-a78c-c78f920a42a4-istio-data\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl\" (UID: \"8c02ff41-a095-4c49-a78c-c78f920a42a4\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl"
Nov 01 03:27:42.173651 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.173615    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c02ff41-a095-4c49-a78c-c78f920a42a4-credential-socket\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl\" (UID: \"8c02ff41-a095-4c49-a78c-c78f920a42a4\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl"
Nov 01 03:27:42.173740 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.173720    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c02ff41-a095-4c49-a78c-c78f920a42a4-istio-podinfo\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl\" (UID: \"8c02ff41-a095-4c49-a78c-c78f920a42a4\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl"
Nov 01 03:27:42.173840 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.173826    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c02ff41-a095-4c49-a78c-c78f920a42a4-istiod-ca-cert\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl\" (UID: \"8c02ff41-a095-4c49-a78c-c78f920a42a4\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl"
Nov 01 03:27:42.173907 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.173892    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c02ff41-a095-4c49-a78c-c78f920a42a4-workload-socket\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl\" (UID: \"8c02ff41-a095-4c49-a78c-c78f920a42a4\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl"
Nov 01 03:27:42.173960 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.173947    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c02ff41-a095-4c49-a78c-c78f920a42a4-istio-envoy\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl\" (UID: \"8c02ff41-a095-4c49-a78c-c78f920a42a4\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl"
Nov 01 03:27:42.174010 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.173984    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c02ff41-a095-4c49-a78c-c78f920a42a4-istio-token\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl\" (UID: \"8c02ff41-a095-4c49-a78c-c78f920a42a4\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl"
Nov 01 03:27:42.174069 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.174034    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c02ff41-a095-4c49-a78c-c78f920a42a4-workload-certs\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl\" (UID: \"8c02ff41-a095-4c49-a78c-c78f920a42a4\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl"
Nov 01 03:27:42.174669 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.174488    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c02ff41-a095-4c49-a78c-c78f920a42a4-workload-certs\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl\" (UID: \"8c02ff41-a095-4c49-a78c-c78f920a42a4\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl"
Nov 01 03:27:42.174831 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.174805    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c02ff41-a095-4c49-a78c-c78f920a42a4-istio-data\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl\" (UID: \"8c02ff41-a095-4c49-a78c-c78f920a42a4\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl"
Nov 01 03:27:42.175092 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.175067    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c02ff41-a095-4c49-a78c-c78f920a42a4-credential-socket\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl\" (UID: \"8c02ff41-a095-4c49-a78c-c78f920a42a4\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl"
Nov 01 03:27:42.176488 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.176457    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c02ff41-a095-4c49-a78c-c78f920a42a4-istiod-ca-cert\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl\" (UID: \"8c02ff41-a095-4c49-a78c-c78f920a42a4\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl"
Nov 01 03:27:42.177920 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.177894    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c02ff41-a095-4c49-a78c-c78f920a42a4-workload-socket\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl\" (UID: \"8c02ff41-a095-4c49-a78c-c78f920a42a4\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl"
Nov 01 03:27:42.180467 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.180444    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c02ff41-a095-4c49-a78c-c78f920a42a4-istio-podinfo\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl\" (UID: \"8c02ff41-a095-4c49-a78c-c78f920a42a4\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl"
Nov 01 03:27:42.193604 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.189081    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c02ff41-a095-4c49-a78c-c78f920a42a4-istio-envoy\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl\" (UID: \"8c02ff41-a095-4c49-a78c-c78f920a42a4\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl"
Nov 01 03:27:42.234133 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.234109    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcqmj\" (UniqueName: \"kubernetes.io/projected/8c02ff41-a095-4c49-a78c-c78f920a42a4-kube-api-access-vcqmj\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl\" (UID: \"8c02ff41-a095-4c49-a78c-c78f920a42a4\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl"
Nov 01 03:27:42.237208 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.237188    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c02ff41-a095-4c49-a78c-c78f920a42a4-istio-token\") pod \"e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl\" (UID: \"8c02ff41-a095-4c49-a78c-c78f920a42a4\") " pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl"
Nov 01 03:27:42.278471 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.278441    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl"
Nov 01 03:27:42.693940 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.693903    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7"]
Nov 01 03:27:42.700608 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.700582    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7"
Nov 01 03:27:42.723017 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:27:42.722995    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c02ff41_a095_4c49_a78c_c78f920a42a4.slice/crio-e4a9fbbb148d55524a5f04f6135d26c387aba96447dd87b0c6215a39b3a33cd2 WatchSource:0}: Error finding container e4a9fbbb148d55524a5f04f6135d26c387aba96447dd87b0c6215a39b3a33cd2: Status 404 returned error can't find the container with id e4a9fbbb148d55524a5f04f6135d26c387aba96447dd87b0c6215a39b3a33cd2
Nov 01 03:27:42.724645 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.724627    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl"]
Nov 01 03:27:42.725970 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.725942    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:27:42.726047 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.726002    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:27:42.726047 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.726031    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:27:42.729530 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.729509    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk"]
Nov 01 03:27:42.734966 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.734946    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk"
Nov 01 03:27:42.782760 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.782043    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7"]
Nov 01 03:27:42.782760 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.782653    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b528dcb-1501-4764-bf26-d676a6e18474-istio-envoy\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7\" (UID: \"4b528dcb-1501-4764-bf26-d676a6e18474\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7"
Nov 01 03:27:42.782760 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.782694    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b528dcb-1501-4764-bf26-d676a6e18474-istio-data\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7\" (UID: \"4b528dcb-1501-4764-bf26-d676a6e18474\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7"
Nov 01 03:27:42.783019 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.782816    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9wln\" (UniqueName: \"kubernetes.io/projected/4b528dcb-1501-4764-bf26-d676a6e18474-kube-api-access-j9wln\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7\" (UID: \"4b528dcb-1501-4764-bf26-d676a6e18474\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7"
Nov 01 03:27:42.783019 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.782868    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd824f02-6c5a-4428-a995-80cbf1b20571-istio-envoy\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk\" (UID: \"fd824f02-6c5a-4428-a995-80cbf1b20571\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk"
Nov 01 03:27:42.783019 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.782894    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd824f02-6c5a-4428-a995-80cbf1b20571-istio-data\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk\" (UID: \"fd824f02-6c5a-4428-a995-80cbf1b20571\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk"
Nov 01 03:27:42.783019 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.782935    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b528dcb-1501-4764-bf26-d676a6e18474-istio-token\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7\" (UID: \"4b528dcb-1501-4764-bf26-d676a6e18474\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7"
Nov 01 03:27:42.783019 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.782958    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd824f02-6c5a-4428-a995-80cbf1b20571-credential-socket\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk\" (UID: \"fd824f02-6c5a-4428-a995-80cbf1b20571\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk"
Nov 01 03:27:42.783019 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.782987    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b528dcb-1501-4764-bf26-d676a6e18474-istiod-ca-cert\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7\" (UID: \"4b528dcb-1501-4764-bf26-d676a6e18474\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7"
Nov 01 03:27:42.783309 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.783055    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd824f02-6c5a-4428-a995-80cbf1b20571-istio-token\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk\" (UID: \"fd824f02-6c5a-4428-a995-80cbf1b20571\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk"
Nov 01 03:27:42.783309 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.783116    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd824f02-6c5a-4428-a995-80cbf1b20571-workload-socket\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk\" (UID: \"fd824f02-6c5a-4428-a995-80cbf1b20571\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk"
Nov 01 03:27:42.783309 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.783163    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b528dcb-1501-4764-bf26-d676a6e18474-workload-socket\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7\" (UID: \"4b528dcb-1501-4764-bf26-d676a6e18474\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7"
Nov 01 03:27:42.783309 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.783267    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b528dcb-1501-4764-bf26-d676a6e18474-credential-socket\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7\" (UID: \"4b528dcb-1501-4764-bf26-d676a6e18474\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7"
Nov 01 03:27:42.783508 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.783349    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b528dcb-1501-4764-bf26-d676a6e18474-workload-certs\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7\" (UID: \"4b528dcb-1501-4764-bf26-d676a6e18474\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7"
Nov 01 03:27:42.783508 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.783373    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd824f02-6c5a-4428-a995-80cbf1b20571-workload-certs\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk\" (UID: \"fd824f02-6c5a-4428-a995-80cbf1b20571\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk"
Nov 01 03:27:42.783508 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.783412    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd824f02-6c5a-4428-a995-80cbf1b20571-istiod-ca-cert\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk\" (UID: \"fd824f02-6c5a-4428-a995-80cbf1b20571\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk"
Nov 01 03:27:42.783508 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.783435    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd824f02-6c5a-4428-a995-80cbf1b20571-istio-podinfo\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk\" (UID: \"fd824f02-6c5a-4428-a995-80cbf1b20571\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk"
Nov 01 03:27:42.783508 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.783465    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b528dcb-1501-4764-bf26-d676a6e18474-istio-podinfo\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7\" (UID: \"4b528dcb-1501-4764-bf26-d676a6e18474\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7"
Nov 01 03:27:42.783778 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.783549    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsh5t\" (UniqueName: \"kubernetes.io/projected/fd824f02-6c5a-4428-a995-80cbf1b20571-kube-api-access-bsh5t\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk\" (UID: \"fd824f02-6c5a-4428-a995-80cbf1b20571\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk"
Nov 01 03:27:42.810487 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.810439    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk"]
Nov 01 03:27:42.838800 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.838767    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:27:42.862568 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.862521    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl" event={"ID":"8c02ff41-a095-4c49-a78c-c78f920a42a4","Type":"ContainerStarted","Data":"7aea0a0f098ea4c0bfcbbfa96e06ef16db65fcfcff65ad262369f16211547e62"}
Nov 01 03:27:42.862700 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.862604    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl" event={"ID":"8c02ff41-a095-4c49-a78c-c78f920a42a4","Type":"ContainerStarted","Data":"e4a9fbbb148d55524a5f04f6135d26c387aba96447dd87b0c6215a39b3a33cd2"}
Nov 01 03:27:42.886406 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.884492    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9wln\" (UniqueName: \"kubernetes.io/projected/4b528dcb-1501-4764-bf26-d676a6e18474-kube-api-access-j9wln\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7\" (UID: \"4b528dcb-1501-4764-bf26-d676a6e18474\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7"
Nov 01 03:27:42.886406 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.884592    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd824f02-6c5a-4428-a995-80cbf1b20571-istio-envoy\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk\" (UID: \"fd824f02-6c5a-4428-a995-80cbf1b20571\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk"
Nov 01 03:27:42.886406 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.884623    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd824f02-6c5a-4428-a995-80cbf1b20571-istio-data\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk\" (UID: \"fd824f02-6c5a-4428-a995-80cbf1b20571\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk"
Nov 01 03:27:42.886406 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.884666    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b528dcb-1501-4764-bf26-d676a6e18474-istio-token\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7\" (UID: \"4b528dcb-1501-4764-bf26-d676a6e18474\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7"
Nov 01 03:27:42.886406 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.884720    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd824f02-6c5a-4428-a995-80cbf1b20571-credential-socket\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk\" (UID: \"fd824f02-6c5a-4428-a995-80cbf1b20571\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk"
Nov 01 03:27:42.886406 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.884750    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b528dcb-1501-4764-bf26-d676a6e18474-istiod-ca-cert\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7\" (UID: \"4b528dcb-1501-4764-bf26-d676a6e18474\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7"
Nov 01 03:27:42.886406 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.884838    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd824f02-6c5a-4428-a995-80cbf1b20571-istio-token\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk\" (UID: \"fd824f02-6c5a-4428-a995-80cbf1b20571\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk"
Nov 01 03:27:42.886406 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.884917    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd824f02-6c5a-4428-a995-80cbf1b20571-workload-socket\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk\" (UID: \"fd824f02-6c5a-4428-a995-80cbf1b20571\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk"
Nov 01 03:27:42.886406 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.884967    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b528dcb-1501-4764-bf26-d676a6e18474-workload-socket\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7\" (UID: \"4b528dcb-1501-4764-bf26-d676a6e18474\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7"
Nov 01 03:27:42.886406 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.885074    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b528dcb-1501-4764-bf26-d676a6e18474-credential-socket\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7\" (UID: \"4b528dcb-1501-4764-bf26-d676a6e18474\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7"
Nov 01 03:27:42.886406 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.885156    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b528dcb-1501-4764-bf26-d676a6e18474-workload-certs\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7\" (UID: \"4b528dcb-1501-4764-bf26-d676a6e18474\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7"
Nov 01 03:27:42.886406 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.885182    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd824f02-6c5a-4428-a995-80cbf1b20571-workload-certs\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk\" (UID: \"fd824f02-6c5a-4428-a995-80cbf1b20571\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk"
Nov 01 03:27:42.886406 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.885236    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd824f02-6c5a-4428-a995-80cbf1b20571-istiod-ca-cert\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk\" (UID: \"fd824f02-6c5a-4428-a995-80cbf1b20571\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk"
Nov 01 03:27:42.886406 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.885260    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd824f02-6c5a-4428-a995-80cbf1b20571-istio-podinfo\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk\" (UID: \"fd824f02-6c5a-4428-a995-80cbf1b20571\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk"
Nov 01 03:27:42.886406 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.885286    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b528dcb-1501-4764-bf26-d676a6e18474-istio-podinfo\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7\" (UID: \"4b528dcb-1501-4764-bf26-d676a6e18474\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7"
Nov 01 03:27:42.886406 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.885373    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsh5t\" (UniqueName: \"kubernetes.io/projected/fd824f02-6c5a-4428-a995-80cbf1b20571-kube-api-access-bsh5t\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk\" (UID: \"fd824f02-6c5a-4428-a995-80cbf1b20571\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk"
Nov 01 03:27:42.886406 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.885411    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b528dcb-1501-4764-bf26-d676a6e18474-istio-envoy\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7\" (UID: \"4b528dcb-1501-4764-bf26-d676a6e18474\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7"
Nov 01 03:27:42.886406 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.885438    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b528dcb-1501-4764-bf26-d676a6e18474-istio-data\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7\" (UID: \"4b528dcb-1501-4764-bf26-d676a6e18474\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7"
Nov 01 03:27:42.889125 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.888820    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd824f02-6c5a-4428-a995-80cbf1b20571-istio-envoy\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk\" (UID: \"fd824f02-6c5a-4428-a995-80cbf1b20571\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk"
Nov 01 03:27:42.891732 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.890512    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b528dcb-1501-4764-bf26-d676a6e18474-istio-podinfo\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7\" (UID: \"4b528dcb-1501-4764-bf26-d676a6e18474\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7"
Nov 01 03:27:42.891732 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.890815    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b528dcb-1501-4764-bf26-d676a6e18474-istio-data\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7\" (UID: \"4b528dcb-1501-4764-bf26-d676a6e18474\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7"
Nov 01 03:27:42.891732 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.890915    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b528dcb-1501-4764-bf26-d676a6e18474-workload-certs\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7\" (UID: \"4b528dcb-1501-4764-bf26-d676a6e18474\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7"
Nov 01 03:27:42.891732 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.891278    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd824f02-6c5a-4428-a995-80cbf1b20571-istio-podinfo\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk\" (UID: \"fd824f02-6c5a-4428-a995-80cbf1b20571\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk"
Nov 01 03:27:42.891732 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.891579    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b528dcb-1501-4764-bf26-d676a6e18474-credential-socket\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7\" (UID: \"4b528dcb-1501-4764-bf26-d676a6e18474\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7"
Nov 01 03:27:42.891732 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.891587    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd824f02-6c5a-4428-a995-80cbf1b20571-istiod-ca-cert\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk\" (UID: \"fd824f02-6c5a-4428-a995-80cbf1b20571\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk"
Nov 01 03:27:42.891732 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.891683    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd824f02-6c5a-4428-a995-80cbf1b20571-workload-certs\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk\" (UID: \"fd824f02-6c5a-4428-a995-80cbf1b20571\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk"
Nov 01 03:27:42.892137 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.891888    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd824f02-6c5a-4428-a995-80cbf1b20571-istio-data\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk\" (UID: \"fd824f02-6c5a-4428-a995-80cbf1b20571\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk"
Nov 01 03:27:42.892190 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.892156    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd824f02-6c5a-4428-a995-80cbf1b20571-credential-socket\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk\" (UID: \"fd824f02-6c5a-4428-a995-80cbf1b20571\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk"
Nov 01 03:27:42.892882 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.892841    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd824f02-6c5a-4428-a995-80cbf1b20571-workload-socket\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk\" (UID: \"fd824f02-6c5a-4428-a995-80cbf1b20571\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk"
Nov 01 03:27:42.892882 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.892857    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b528dcb-1501-4764-bf26-d676a6e18474-istiod-ca-cert\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7\" (UID: \"4b528dcb-1501-4764-bf26-d676a6e18474\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7"
Nov 01 03:27:42.893029 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.892928    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b528dcb-1501-4764-bf26-d676a6e18474-workload-socket\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7\" (UID: \"4b528dcb-1501-4764-bf26-d676a6e18474\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7"
Nov 01 03:27:42.896267 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.896205    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b528dcb-1501-4764-bf26-d676a6e18474-istio-envoy\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7\" (UID: \"4b528dcb-1501-4764-bf26-d676a6e18474\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7"
Nov 01 03:27:42.953200 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.953125    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9wln\" (UniqueName: \"kubernetes.io/projected/4b528dcb-1501-4764-bf26-d676a6e18474-kube-api-access-j9wln\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7\" (UID: \"4b528dcb-1501-4764-bf26-d676a6e18474\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7"
Nov 01 03:27:42.962613 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.962379    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd824f02-6c5a-4428-a995-80cbf1b20571-istio-token\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk\" (UID: \"fd824f02-6c5a-4428-a995-80cbf1b20571\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk"
Nov 01 03:27:42.964008 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.963947    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b528dcb-1501-4764-bf26-d676a6e18474-istio-token\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7\" (UID: \"4b528dcb-1501-4764-bf26-d676a6e18474\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7"
Nov 01 03:27:42.967710 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:42.967682    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsh5t\" (UniqueName: \"kubernetes.io/projected/fd824f02-6c5a-4428-a995-80cbf1b20571-kube-api-access-bsh5t\") pod \"e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk\" (UID: \"fd824f02-6c5a-4428-a995-80cbf1b20571\") " pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk"
Nov 01 03:27:43.013344 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:43.013317    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faa90817-9lvwc"
Nov 01 03:27:43.019775 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:43.019752    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7"
Nov 01 03:27:43.020320 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:43.020292    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:27:43.055315 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:43.055282    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk"
Nov 01 03:27:43.099075 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:43.091175    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19139bc8-a17f-4093-b596-4ac8225e260c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19139bc8-a17f-4093-b596-4ac8225e260c-maas-api-openshift-ca\") pod \"19139bc8-a17f-4093-b596-4ac8225e260c\" (UID: \"19139bc8-a17f-4093-b596-4ac8225e260c\") "
Nov 01 03:27:43.099075 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:43.091379    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19139bc8-a17f-4093-b596-4ac8225e260c-kube-api-access-9slpm\" (UniqueName: \"kubernetes.io/projected/19139bc8-a17f-4093-b596-4ac8225e260c-kube-api-access-9slpm\") pod \"19139bc8-a17f-4093-b596-4ac8225e260c\" (UID: \"19139bc8-a17f-4093-b596-4ac8225e260c\") "
Nov 01 03:27:43.100551 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:43.100493    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19139bc8-a17f-4093-b596-4ac8225e260c-maas-api-openshift-ca" pod "19139bc8-a17f-4093-b596-4ac8225e260c" (UID: "19139bc8-a17f-4093-b596-4ac8225e260c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:27:43.105191 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:43.105143    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19139bc8-a17f-4093-b596-4ac8225e260c-kube-api-access-9slpm" pod "19139bc8-a17f-4093-b596-4ac8225e260c" (UID: "19139bc8-a17f-4093-b596-4ac8225e260c"). InnerVolumeSpecName "kube-api-access-9slpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:27:43.106806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:43.106770    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19139bc8-a17f-4093-b596-4ac8225e260c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19139bc8-a17f-4093-b596-4ac8225e260c-maas-api-xks-ca\") pod \"19139bc8-a17f-4093-b596-4ac8225e260c\" (UID: \"19139bc8-a17f-4093-b596-4ac8225e260c\") "
Nov 01 03:27:43.107589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:43.107532    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19139bc8-a17f-4093-b596-4ac8225e260c-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:27:43.107589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:43.107578    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9slpm\" (UniqueName: \"kubernetes.io/projected/19139bc8-a17f-4093-b596-4ac8225e260c-kube-api-access-9slpm\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:27:43.110396 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:43.110312    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl" podStartSLOduration=2.110293214 podStartE2EDuration="2.110293214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:27:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:27:43.100005439 +0000 UTC m=+1267.623012028" watchObservedRunningTime="2026-11-01 03:27:43.110293214 +0000 UTC m=+1267.633299803"
Nov 01 03:27:43.114522 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:43.114493    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19139bc8-a17f-4093-b596-4ac8225e260c-maas-api-xks-ca" pod "19139bc8-a17f-4093-b596-4ac8225e260c" (UID: "19139bc8-a17f-4093-b596-4ac8225e260c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:27:43.210632 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:43.209798    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19139bc8-a17f-4093-b596-4ac8225e260c-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:27:43.280579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:43.280512    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl"
Nov 01 03:27:43.281140 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:43.280811    4930 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 01 03:27:43.281140 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:43.280895    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl" podUID="8c02ff41-a095-4c49-a78c-c78f920a42a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 01 03:27:43.448822 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:43.448788    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7"]
Nov 01 03:27:43.450539 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:43.450508    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:27:43.450654 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:43.450590    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:27:43.450654 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:43.450636    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:27:43.605781 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:43.604334    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk"]
Nov 01 03:27:43.605781 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:43.605000    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:27:43.605781 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:43.605079    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:27:43.605781 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:43.605122    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:27:43.907599 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:43.907524    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk" event={"ID":"fd824f02-6c5a-4428-a995-80cbf1b20571","Type":"ContainerStarted","Data":"5778c35aad39918487bc447f89f9ff1a99fc3fcf8d7df7b4060f4a6425121b77"}
Nov 01 03:27:43.908107 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:43.907626    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk" event={"ID":"fd824f02-6c5a-4428-a995-80cbf1b20571","Type":"ContainerStarted","Data":"77904612833e66680b4f4011e7e039e7c264c8ebac051cb65a0c3f6a121504c1"}
Nov 01 03:27:43.909840 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:43.909812    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7" event={"ID":"4b528dcb-1501-4764-bf26-d676a6e18474","Type":"ContainerStarted","Data":"8e3a98e5f62c2e50232c9ac38e464abe3bee0d658efdb9265bd52afb76278b1a"}
Nov 01 03:27:43.909984 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:43.909858    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7" event={"ID":"4b528dcb-1501-4764-bf26-d676a6e18474","Type":"ContainerStarted","Data":"fe158cb237022690dc27d46e631057bced9c33dbd953125b2474f45fbbfc3003"}
Nov 01 03:27:43.912166 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:43.912139    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faa90817-9lvwc"
Nov 01 03:27:43.912166 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:43.912140    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faa90817-9lvwc" event={"ID":"19139bc8-a17f-4093-b596-4ac8225e260c","Type":"ContainerDied","Data":"ea6f7efbd43d105e59526170a9b4d27a26016e9d87bd9f1816d12c8199d892db"}
Nov 01 03:27:43.912352 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:43.912197    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea6f7efbd43d105e59526170a9b4d27a26016e9d87bd9f1816d12c8199d892db"
Nov 01 03:27:44.020977 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:44.020879    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7"
Nov 01 03:27:44.022284 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:44.022256    4930 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 01 03:27:44.022411 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:44.022312    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7" podUID="4b528dcb-1501-4764-bf26-d676a6e18474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 01 03:27:44.040212 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:44.040153    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk" podStartSLOduration=2.040135214 podStartE2EDuration="2.040135214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:27:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:27:44.03572721 +0000 UTC m=+1268.558733797" watchObservedRunningTime="2026-11-01 03:27:44.040135214 +0000 UTC m=+1268.563141802"
Nov 01 03:27:44.057167 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:44.057137    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk"
Nov 01 03:27:44.057298 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:44.057248    4930 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 01 03:27:44.057298 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:44.057288    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk" podUID="fd824f02-6c5a-4428-a995-80cbf1b20571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 01 03:27:44.158552 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:44.158499    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7" podStartSLOduration=2.158485296 podStartE2EDuration="2.158485296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:27:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:27:44.156367611 +0000 UTC m=+1268.679374200" watchObservedRunningTime="2026-11-01 03:27:44.158485296 +0000 UTC m=+1268.681491884"
Nov 01 03:27:44.293107 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:44.293011    4930 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 01 03:27:44.294413 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:44.294379    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl" podUID="8c02ff41-a095-4c49-a78c-c78f920a42a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 01 03:27:45.027577 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:45.021983    4930 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 01 03:27:45.027577 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:45.022048    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7" podUID="4b528dcb-1501-4764-bf26-d676a6e18474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 01 03:27:45.054447 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:45.054408    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02454908-l2l78"]
Nov 01 03:27:45.056252 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:45.056227    4930 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 01 03:27:45.056358 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:45.056273    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk" podUID="fd824f02-6c5a-4428-a995-80cbf1b20571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 01 03:27:45.059584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:45.059329    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02454908-l2l78"]
Nov 01 03:27:45.279139 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:45.279045    4930 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 01 03:27:45.279394 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:45.279352    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl" podUID="8c02ff41-a095-4c49-a78c-c78f920a42a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 01 03:27:45.712579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:45.709338    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa58e66b-8dbc-430c-a8c0-479fbd7999c1" path="/var/lib/kubelet/pods/fa58e66b-8dbc-430c-a8c0-479fbd7999c1/volumes"
Nov 01 03:27:46.021464 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:46.020942    4930 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 01 03:27:46.021464 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:46.021007    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7" podUID="4b528dcb-1501-4764-bf26-d676a6e18474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 01 03:27:46.080581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:46.059995    4930 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 01 03:27:46.080581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:46.060066    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk" podUID="fd824f02-6c5a-4428-a995-80cbf1b20571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 01 03:27:46.279054 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:46.278963    4930 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 01 03:27:46.279324 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:46.279291    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl" podUID="8c02ff41-a095-4c49-a78c-c78f920a42a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 01 03:27:46.635579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:46.624024    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c979941-n9zgq"]
Nov 01 03:27:46.635579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:46.625031    4930 cpu_manager.go:414] "containerMap: removing container" podUID="19139bc8-a17f-4093-b596-4ac8225e260c" containerName="revoke-keys"
Nov 01 03:27:46.635579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:46.625052    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19139bc8-a17f-4093-b596-4ac8225e260c" containerName="revoke-keys"
Nov 01 03:27:46.635579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:46.625229    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="19139bc8-a17f-4093-b596-4ac8225e260c" containerName="revoke-keys"
Nov 01 03:27:46.781581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:46.749827    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c979941-n9zgq"
Nov 01 03:27:46.847579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:46.839000    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c979941-n9zgq"]
Nov 01 03:27:46.896968 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:46.894912    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:27:46.896968 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:46.895023    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:27:46.896968 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:46.895125    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:27:46.896968 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:46.895154    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:27:46.962583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:46.955690    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwdbn\" (UniqueName: \"kubernetes.io/projected/0f046a7c-23d0-4995-88a1-2046b50666e6-kube-api-access-nwdbn\") pod \"maas-api-revoke-sub-6c979941-n9zgq\" (UID: \"0f046a7c-23d0-4995-88a1-2046b50666e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c979941-n9zgq"
Nov 01 03:27:46.962583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:46.955770    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f046a7c-23d0-4995-88a1-2046b50666e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c979941-n9zgq\" (UID: \"0f046a7c-23d0-4995-88a1-2046b50666e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c979941-n9zgq"
Nov 01 03:27:46.962583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:46.955840    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f046a7c-23d0-4995-88a1-2046b50666e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c979941-n9zgq\" (UID: \"0f046a7c-23d0-4995-88a1-2046b50666e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c979941-n9zgq"
Nov 01 03:27:47.042578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:47.042291    4930 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 01 03:27:47.042578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:47.042350    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7" podUID="4b528dcb-1501-4764-bf26-d676a6e18474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 01 03:27:47.076578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:47.075949    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwdbn\" (UniqueName: \"kubernetes.io/projected/0f046a7c-23d0-4995-88a1-2046b50666e6-kube-api-access-nwdbn\") pod \"maas-api-revoke-sub-6c979941-n9zgq\" (UID: \"0f046a7c-23d0-4995-88a1-2046b50666e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c979941-n9zgq"
Nov 01 03:27:47.076578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:47.076003    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f046a7c-23d0-4995-88a1-2046b50666e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c979941-n9zgq\" (UID: \"0f046a7c-23d0-4995-88a1-2046b50666e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c979941-n9zgq"
Nov 01 03:27:47.076578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:47.076057    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f046a7c-23d0-4995-88a1-2046b50666e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c979941-n9zgq\" (UID: \"0f046a7c-23d0-4995-88a1-2046b50666e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c979941-n9zgq"
Nov 01 03:27:47.081607 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:47.078670    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f046a7c-23d0-4995-88a1-2046b50666e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c979941-n9zgq\" (UID: \"0f046a7c-23d0-4995-88a1-2046b50666e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c979941-n9zgq"
Nov 01 03:27:47.121972 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:47.082922    4930 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 01 03:27:47.121972 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:47.082975    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk" podUID="fd824f02-6c5a-4428-a995-80cbf1b20571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 01 03:27:47.142633 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:47.140602    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f046a7c-23d0-4995-88a1-2046b50666e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c979941-n9zgq\" (UID: \"0f046a7c-23d0-4995-88a1-2046b50666e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c979941-n9zgq"
Nov 01 03:27:47.206633 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:47.187042    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwdbn\" (UniqueName: \"kubernetes.io/projected/0f046a7c-23d0-4995-88a1-2046b50666e6-kube-api-access-nwdbn\") pod \"maas-api-revoke-sub-6c979941-n9zgq\" (UID: \"0f046a7c-23d0-4995-88a1-2046b50666e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c979941-n9zgq"
Nov 01 03:27:47.242580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:47.241610    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46ed40ca-qhc7q"]
Nov 01 03:27:47.258592 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:47.250805    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c979941-n9zgq"
Nov 01 03:27:47.284583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:47.264142    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46ed40ca-qhc7q"]
Nov 01 03:27:47.304684 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:47.293988    4930 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 01 03:27:47.304942 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:47.304913    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl" podUID="8c02ff41-a095-4c49-a78c-c78f920a42a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 01 03:27:47.649758 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:47.648042    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf" podUID="40881383-61bb-4fc8-9c75-f8ffce4834c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:47.649758 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:47.648292    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t" podUID="13930c1e-2200-4412-b8d3-ae3e4126a851" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 01 03:27:47.649758 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:47.648466    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm" podUID="b2637bc6-3828-4b75-9783-5415519403a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 01 03:27:47.659744 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:47.659627    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp" podUID="667240b0-64a7-45cd-84fe-9cd7e0d8a301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:47.671700 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:47.671657    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" podUID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:47.718129 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:47.677136    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw" podUID="b6d688b8-b911-4a6a-a842-2b0fb7a75e83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:47.766224 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:47.747634    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7" podUID="1ce834cc-d34d-46cc-a081-4d2cc261becb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:47.799582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:47.783900    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm" podUID="75d142b3-7172-4b57-bc07-a2f404c4656e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": context deadline exceeded"
Nov 01 03:27:47.881586 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:47.859327    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n" podUID="d2511772-66f9-40a0-9166-87c4345485c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 01 03:27:47.889586 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:47.889441    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cbb22dc-a592-4452-b87f-fb1eaba7fd47" path="/var/lib/kubelet/pods/6cbb22dc-a592-4452-b87f-fb1eaba7fd47/volumes"
Nov 01 03:27:48.023582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:48.023460    4930 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 01 03:27:48.023582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:48.023518    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7" podUID="4b528dcb-1501-4764-bf26-d676a6e18474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 01 03:27:48.101624 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:48.101585    4930 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 01 03:27:48.102245 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:48.102212    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk" podUID="fd824f02-6c5a-4428-a995-80cbf1b20571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 01 03:27:48.331500 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:48.331414    4930 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 01 03:27:48.334995 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:48.334940    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl" podUID="8c02ff41-a095-4c49-a78c-c78f920a42a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 01 03:27:48.580782 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:48.571338    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps" podUID="d658772c-b072-4513-a639-d19a94c78525" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:48.609579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:48.594650    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj" podUID="b0e32cb3-ac18-4c1f-819c-f1139a88bfea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:48.631718 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:48.631673    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps" podUID="d658772c-b072-4513-a639-d19a94c78525" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 01 03:27:48.745320 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:48.688041    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t" podUID="13930c1e-2200-4412-b8d3-ae3e4126a851" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 01 03:27:48.791792 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:48.752645    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp" podUID="667240b0-64a7-45cd-84fe-9cd7e0d8a301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": context deadline exceeded"
Nov 01 03:27:48.792243 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:48.792212    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:27:48.843883 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:48.843843    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp" podUID="667240b0-64a7-45cd-84fe-9cd7e0d8a301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 01 03:27:48.845089 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:48.844470    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:27:48.921851 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:48.846169    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" podUID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded"
Nov 01 03:27:48.921851 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:48.847126    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp"
Nov 01 03:27:48.921851 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:48.847193    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:27:48.921851 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:48.864880    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n" podUID="d2511772-66f9-40a0-9166-87c4345485c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:48.921851 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:48.872112    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"
Nov 01 03:27:48.921851 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:48.891244    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw" podUID="b6d688b8-b911-4a6a-a842-2b0fb7a75e83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 01 03:27:48.921851 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:48.891756    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:27:48.921851 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:48.895020    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7" podUID="1ce834cc-d34d-46cc-a081-4d2cc261becb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:48.921851 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:48.895867    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm" podUID="75d142b3-7172-4b57-bc07-a2f404c4656e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:48.921851 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:48.919123    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:27:48.949375 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:48.949324    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7" podUID="1ce834cc-d34d-46cc-a081-4d2cc261becb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:48.965580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:48.961696    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"
Nov 01 03:27:49.325353 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:49.320419    4930 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 01 03:27:49.325353 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:49.320474    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl" podUID="8c02ff41-a095-4c49-a78c-c78f920a42a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 01 03:27:49.581893 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:49.442911    4930 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 01 03:27:49.581893 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:49.442968    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk" podUID="fd824f02-6c5a-4428-a995-80cbf1b20571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 01 03:27:49.581893 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:49.443045    4930 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 01 03:27:49.581893 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:49.443070    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7" podUID="4b528dcb-1501-4764-bf26-d676a6e18474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 01 03:27:49.581893 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:49.446056    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"
Nov 01 03:27:49.581893 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:49.447262    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c979941-n9zgq"]
Nov 01 03:27:49.657026 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:49.656981    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps" podUID="d658772c-b072-4513-a639-d19a94c78525" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:49.744130 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:49.684633    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8" podUID="88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:49.928059 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:49.918473    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n" podUID="d2511772-66f9-40a0-9166-87c4345485c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 01 03:27:50.048589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:50.045623    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" podUID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded"
Nov 01 03:27:50.048589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:50.045991    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm" podUID="75d142b3-7172-4b57-bc07-a2f404c4656e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": context deadline exceeded"
Nov 01 03:27:50.073613 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:50.056292    4930 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 01 03:27:50.073613 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:50.056341    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7" podUID="4b528dcb-1501-4764-bf26-d676a6e18474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 01 03:27:50.085996 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:50.074044    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj" podUID="b0e32cb3-ac18-4c1f-819c-f1139a88bfea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:50.156608 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:50.143173    4930 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 01 03:27:50.156608 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:50.143228    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk" podUID="fd824f02-6c5a-4428-a995-80cbf1b20571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 01 03:27:50.302411 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:27:50.302365    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:27:50.353784 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:50.353629    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t" podUID="13930c1e-2200-4412-b8d3-ae3e4126a851" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:50.356990 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:50.356707    4930 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 01 03:27:50.356990 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:50.356767    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl" podUID="8c02ff41-a095-4c49-a78c-c78f920a42a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 01 03:27:50.381073 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:50.381028    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:27:50.403922 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:50.403891    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp"
Nov 01 03:27:50.427028 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:50.426988    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:27:50.427487 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:50.427469    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:27:50.427749 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:50.427681    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:27:50.436537 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:50.436045    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c979941-n9zgq" event={"ID":"0f046a7c-23d0-4995-88a1-2046b50666e6","Type":"ContainerStarted","Data":"1e8b701983510c5e77a5871e2d5a385ca957f1cd70285ce077939a9865150a3c"}
Nov 01 03:27:50.441634 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:50.441612    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"
Nov 01 03:27:50.503785 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:50.501635    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:27:50.525493 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:50.521827    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:27:51.031613 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:51.030749    4930 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 01 03:27:51.031613 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:51.030829    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7" podUID="4b528dcb-1501-4764-bf26-d676a6e18474" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 01 03:27:51.076089 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:51.076045    4930 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 01 03:27:51.076258 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:51.076115    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk" podUID="fd824f02-6c5a-4428-a995-80cbf1b20571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 01 03:27:51.360767 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:51.360730    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl"
Nov 01 03:27:51.361467 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:51.361329    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl"
Nov 01 03:27:51.362829 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:51.362807    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl"
Nov 01 03:27:51.472860 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:51.472829    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c979941-n9zgq" event={"ID":"0f046a7c-23d0-4995-88a1-2046b50666e6","Type":"ContainerStarted","Data":"3d912421efa5cee727d9206633b34f73d81622c0e1a0f8f52946313ab0b3b83a"}
Nov 01 03:27:51.846847 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:51.846786    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c979941-n9zgq" podStartSLOduration=5.846767904 podStartE2EDuration="5.846767904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:27:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:27:51.838993543 +0000 UTC m=+1276.362000133" watchObservedRunningTime="2026-11-01 03:27:51.846767904 +0000 UTC m=+1276.369774493"
Nov 01 03:27:52.261343 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:52.261255    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7"
Nov 01 03:27:52.262051 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:52.261990    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7"
Nov 01 03:27:52.264482 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:52.263841    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7"
Nov 01 03:27:52.496089 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:52.496038    4930 generic.go:356] "Generic (PLEG): container finished" podID="0f046a7c-23d0-4995-88a1-2046b50666e6" containerID="3d912421efa5cee727d9206633b34f73d81622c0e1a0f8f52946313ab0b3b83a" exitCode=0
Nov 01 03:27:52.496611 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:52.496307    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c979941-n9zgq" event={"ID":"0f046a7c-23d0-4995-88a1-2046b50666e6","Type":"ContainerDied","Data":"3d912421efa5cee727d9206633b34f73d81622c0e1a0f8f52946313ab0b3b83a"}
Nov 01 03:27:52.562544 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:52.562479    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk"
Nov 01 03:27:52.563124 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:52.563073    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk"
Nov 01 03:27:52.564047 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:52.564023    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk"
Nov 01 03:27:52.732717 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:52.732672    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa25001-m46sl"]
Nov 01 03:27:52.737540 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:52.737516    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa25001-m46sl"
Nov 01 03:27:52.800712 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:52.800678    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f68d831-6076-4bf4-806d-742446d2dfbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afa25001-m46sl\" (UID: \"4f68d831-6076-4bf4-806d-742446d2dfbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa25001-m46sl"
Nov 01 03:27:52.800972 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:52.800943    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f68d831-6076-4bf4-806d-742446d2dfbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afa25001-m46sl\" (UID: \"4f68d831-6076-4bf4-806d-742446d2dfbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa25001-m46sl"
Nov 01 03:27:52.801475 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:52.801227    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7vfm\" (UniqueName: \"kubernetes.io/projected/4f68d831-6076-4bf4-806d-742446d2dfbf-kube-api-access-f7vfm\") pod \"maas-api-revoke-sub-afa25001-m46sl\" (UID: \"4f68d831-6076-4bf4-806d-742446d2dfbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa25001-m46sl"
Nov 01 03:27:52.805353 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:52.805327    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa25001-m46sl"]
Nov 01 03:27:52.902626 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:52.902575    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7vfm\" (UniqueName: \"kubernetes.io/projected/4f68d831-6076-4bf4-806d-742446d2dfbf-kube-api-access-f7vfm\") pod \"maas-api-revoke-sub-afa25001-m46sl\" (UID: \"4f68d831-6076-4bf4-806d-742446d2dfbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa25001-m46sl"
Nov 01 03:27:52.902820 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:52.902706    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f68d831-6076-4bf4-806d-742446d2dfbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afa25001-m46sl\" (UID: \"4f68d831-6076-4bf4-806d-742446d2dfbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa25001-m46sl"
Nov 01 03:27:52.902820 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:52.902751    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f68d831-6076-4bf4-806d-742446d2dfbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afa25001-m46sl\" (UID: \"4f68d831-6076-4bf4-806d-742446d2dfbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa25001-m46sl"
Nov 01 03:27:52.903383 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:52.903365    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f68d831-6076-4bf4-806d-742446d2dfbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afa25001-m46sl\" (UID: \"4f68d831-6076-4bf4-806d-742446d2dfbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa25001-m46sl"
Nov 01 03:27:52.904949 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:52.904923    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f68d831-6076-4bf4-806d-742446d2dfbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afa25001-m46sl\" (UID: \"4f68d831-6076-4bf4-806d-742446d2dfbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa25001-m46sl"
Nov 01 03:27:52.963479 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:52.963451    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7vfm\" (UniqueName: \"kubernetes.io/projected/4f68d831-6076-4bf4-806d-742446d2dfbf-kube-api-access-f7vfm\") pod \"maas-api-revoke-sub-afa25001-m46sl\" (UID: \"4f68d831-6076-4bf4-806d-742446d2dfbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa25001-m46sl"
Nov 01 03:27:53.059280 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:53.059241    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa25001-m46sl"
Nov 01 03:27:53.604159 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:53.604127    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa25001-m46sl"]
Nov 01 03:27:53.636579 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:27:53.636514    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f68d831_6076_4bf4_806d_742446d2dfbf.slice/crio-4238ff42079a2ee08a25198831d2258d823d2f231c0654ae4f9d1e5468880482 WatchSource:0}: Error finding container 4238ff42079a2ee08a25198831d2258d823d2f231c0654ae4f9d1e5468880482: Status 404 returned error can't find the container with id 4238ff42079a2ee08a25198831d2258d823d2f231c0654ae4f9d1e5468880482
Nov 01 03:27:53.755607 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:53.755410    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:27:53.761550 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:53.761531    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c979941-n9zgq"
Nov 01 03:27:53.870534 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:53.866491    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f046a7c-23d0-4995-88a1-2046b50666e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f046a7c-23d0-4995-88a1-2046b50666e6-maas-api-openshift-ca\") pod \"0f046a7c-23d0-4995-88a1-2046b50666e6\" (UID: \"0f046a7c-23d0-4995-88a1-2046b50666e6\") "
Nov 01 03:27:53.870534 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:53.867020    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f046a7c-23d0-4995-88a1-2046b50666e6-maas-api-openshift-ca" pod "0f046a7c-23d0-4995-88a1-2046b50666e6" (UID: "0f046a7c-23d0-4995-88a1-2046b50666e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:27:53.870534 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:53.867095    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f046a7c-23d0-4995-88a1-2046b50666e6-kube-api-access-nwdbn\" (UniqueName: \"kubernetes.io/projected/0f046a7c-23d0-4995-88a1-2046b50666e6-kube-api-access-nwdbn\") pod \"0f046a7c-23d0-4995-88a1-2046b50666e6\" (UID: \"0f046a7c-23d0-4995-88a1-2046b50666e6\") "
Nov 01 03:27:53.870534 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:53.867280    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f046a7c-23d0-4995-88a1-2046b50666e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f046a7c-23d0-4995-88a1-2046b50666e6-maas-api-xks-ca\") pod \"0f046a7c-23d0-4995-88a1-2046b50666e6\" (UID: \"0f046a7c-23d0-4995-88a1-2046b50666e6\") "
Nov 01 03:27:53.870534 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:53.867994    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f046a7c-23d0-4995-88a1-2046b50666e6-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:27:53.891550 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:53.882753    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f046a7c-23d0-4995-88a1-2046b50666e6-kube-api-access-nwdbn" pod "0f046a7c-23d0-4995-88a1-2046b50666e6" (UID: "0f046a7c-23d0-4995-88a1-2046b50666e6"). InnerVolumeSpecName "kube-api-access-nwdbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:27:53.891550 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:53.883186    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f046a7c-23d0-4995-88a1-2046b50666e6-maas-api-xks-ca" pod "0f046a7c-23d0-4995-88a1-2046b50666e6" (UID: "0f046a7c-23d0-4995-88a1-2046b50666e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:27:54.046640 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:54.019684    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f046a7c-23d0-4995-88a1-2046b50666e6-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:27:54.046640 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:54.019724    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwdbn\" (UniqueName: \"kubernetes.io/projected/0f046a7c-23d0-4995-88a1-2046b50666e6-kube-api-access-nwdbn\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:27:54.046640 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:27:54.029665    4930 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f68d831_6076_4bf4_806d_742446d2dfbf.slice/crio-05502d3f861de4736ff8f75334354ea2734f2fca74bbf0547fd571a08b14c021.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 03:27:54.656585 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:54.651881    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c979941-n9zgq" event={"ID":"0f046a7c-23d0-4995-88a1-2046b50666e6","Type":"ContainerDied","Data":"1e8b701983510c5e77a5871e2d5a385ca957f1cd70285ce077939a9865150a3c"}
Nov 01 03:27:54.656585 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:54.651945    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e8b701983510c5e77a5871e2d5a385ca957f1cd70285ce077939a9865150a3c"
Nov 01 03:27:54.656585 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:54.652016    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c979941-n9zgq"
Nov 01 03:27:54.684324 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:54.679605    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa25001-m46sl" event={"ID":"4f68d831-6076-4bf4-806d-742446d2dfbf","Type":"ContainerStarted","Data":"05502d3f861de4736ff8f75334354ea2734f2fca74bbf0547fd571a08b14c021"}
Nov 01 03:27:54.684324 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:54.679699    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa25001-m46sl" event={"ID":"4f68d831-6076-4bf4-806d-742446d2dfbf","Type":"ContainerStarted","Data":"4238ff42079a2ee08a25198831d2258d823d2f231c0654ae4f9d1e5468880482"}
Nov 01 03:27:54.829582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:54.827817    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa25001-m46sl" podStartSLOduration=2.82779777 podStartE2EDuration="2.82779777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:27:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:27:54.821276267 +0000 UTC m=+1279.344282856" watchObservedRunningTime="2026-11-01 03:27:54.82779777 +0000 UTC m=+1279.350804361"
Nov 01 03:27:55.025639 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:55.024671    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf" podUID="40881383-61bb-4fc8-9c75-f8ffce4834c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:55.139581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:55.077652    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm" podUID="75d142b3-7172-4b57-bc07-a2f404c4656e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:55.139581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:55.092165    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp" podUID="667240b0-64a7-45cd-84fe-9cd7e0d8a301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:55.139581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:55.092507    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw" podUID="b6d688b8-b911-4a6a-a842-2b0fb7a75e83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:55.364659 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:55.356661    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t" podUID="13930c1e-2200-4412-b8d3-ae3e4126a851" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:55.486610 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:55.479267    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7" podUID="1ce834cc-d34d-46cc-a081-4d2cc261becb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:55.491682 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:55.491643    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw" podUID="b6d688b8-b911-4a6a-a842-2b0fb7a75e83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:55.493475 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:55.491883    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm" podUID="b2637bc6-3828-4b75-9783-5415519403a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:55.511731 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:55.511656    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" podUID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:55.611492 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:55.611446    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj" podUID="b0e32cb3-ac18-4c1f-819c-f1139a88bfea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:55.633959 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:55.633858    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm" podUID="b2637bc6-3828-4b75-9783-5415519403a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:55.634571 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:55.634523    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8" podUID="88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Nov 01 03:27:55.637879 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:55.636863    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps" podUID="d658772c-b072-4513-a639-d19a94c78525" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 01 03:27:55.641887 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:55.636891    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf" podUID="40881383-61bb-4fc8-9c75-f8ffce4834c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Nov 01 03:27:55.642296 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:55.642268    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2" podUID="e2483a06-2f4b-4a1e-af4e-a63eb50d29bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:55.798581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:55.798464    4930 generic.go:356] "Generic (PLEG): container finished" podID="4f68d831-6076-4bf4-806d-742446d2dfbf" containerID="05502d3f861de4736ff8f75334354ea2734f2fca74bbf0547fd571a08b14c021" exitCode=0
Nov 01 03:27:55.799710 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:55.799212    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa25001-m46sl" event={"ID":"4f68d831-6076-4bf4-806d-742446d2dfbf","Type":"ContainerDied","Data":"05502d3f861de4736ff8f75334354ea2734f2fca74bbf0547fd571a08b14c021"}
Nov 01 03:27:56.003803 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:56.003689    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t" podUID="13930c1e-2200-4412-b8d3-ae3e4126a851" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": context deadline exceeded"
Nov 01 03:27:56.085946 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:56.082957    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm" podUID="75d142b3-7172-4b57-bc07-a2f404c4656e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:56.085946 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:56.083464    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:27:56.117579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:56.114760    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp" podUID="667240b0-64a7-45cd-84fe-9cd7e0d8a301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:56.117579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:56.115174    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp"
Nov 01 03:27:56.336642 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:56.334101    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm" podUID="75d142b3-7172-4b57-bc07-a2f404c4656e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:56.448582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:56.412908    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t" podUID="13930c1e-2200-4412-b8d3-ae3e4126a851" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 01 03:27:56.494583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:56.467680    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:27:56.494583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:56.468787    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n" podUID="d2511772-66f9-40a0-9166-87c4345485c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 01 03:27:56.528582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:56.497003    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw" podUID="b6d688b8-b911-4a6a-a842-2b0fb7a75e83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 01 03:27:56.540955 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:56.497895    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7" podUID="1ce834cc-d34d-46cc-a081-4d2cc261becb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:56.566581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:56.554379    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:27:56.566956 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:56.566924    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"
Nov 01 03:27:56.592776 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:56.529502    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" podUID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:56.616614 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:56.612484    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj" podUID="b0e32cb3-ac18-4c1f-819c-f1139a88bfea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:56.616614 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:56.613698    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:27:56.616614 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:56.613755    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:27:56.639904 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:56.639806    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8" podUID="88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:56.641917 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:56.640720    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm" podUID="b2637bc6-3828-4b75-9783-5415519403a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 01 03:27:56.641917 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:56.640949    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:27:56.642121 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:56.641996    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps" podUID="d658772c-b072-4513-a639-d19a94c78525" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:56.668141 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:56.643018    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"
Nov 01 03:27:56.668141 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:56.643091    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:27:56.668141 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:56.655844    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2" podUID="e2483a06-2f4b-4a1e-af4e-a63eb50d29bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:56.668141 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:56.656100    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf" podUID="40881383-61bb-4fc8-9c75-f8ffce4834c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Nov 01 03:27:56.668141 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:56.656323    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"
Nov 01 03:27:56.668141 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:56.656378    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:27:57.083563 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:57.083505    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm" podUID="75d142b3-7172-4b57-bc07-a2f404c4656e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:57.150965 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:57.132983    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp" podUID="667240b0-64a7-45cd-84fe-9cd7e0d8a301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 01 03:27:57.510578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:57.472700    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n" podUID="d2511772-66f9-40a0-9166-87c4345485c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:57.510578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:57.473049    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t" podUID="13930c1e-2200-4412-b8d3-ae3e4126a851" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:57.510578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:57.473305    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"
Nov 01 03:27:57.597569 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:57.597507    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" podUID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:57.598057 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:57.598028    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw" podUID="b6d688b8-b911-4a6a-a842-2b0fb7a75e83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 01 03:27:57.614753 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:57.614720    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj" podUID="b0e32cb3-ac18-4c1f-819c-f1139a88bfea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:57.654606 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:57.644775    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8" podUID="88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Nov 01 03:27:57.655178 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:57.655149    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm" podUID="b2637bc6-3828-4b75-9783-5415519403a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 01 03:27:57.655591 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:57.655539    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps" podUID="d658772c-b072-4513-a639-d19a94c78525" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 01 03:27:57.663577 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:57.657350    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7" podUID="1ce834cc-d34d-46cc-a081-4d2cc261becb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:57.663801 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:57.659613    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf" podUID="40881383-61bb-4fc8-9c75-f8ffce4834c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Nov 01 03:27:57.664113 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:57.659641    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2" podUID="e2483a06-2f4b-4a1e-af4e-a63eb50d29bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": context deadline exceeded"
Nov 01 03:27:58.102820 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:58.102772    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8" podUID="88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": context deadline exceeded"
Nov 01 03:27:58.143674 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:58.143631    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp" podUID="667240b0-64a7-45cd-84fe-9cd7e0d8a301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:27:58.167990 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:58.152686    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:27:58.168286 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:58.168268    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"
Nov 01 03:27:58.178589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:58.177384    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:27:58.204088 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:58.204054    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:27:58.218843 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:58.218722    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp"
Nov 01 03:27:58.236202 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:58.219478    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:27:58.236202 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:58.221085    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"
Nov 01 03:27:58.236202 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:58.221923    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"
Nov 01 03:27:58.236202 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:58.221986    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:27:58.236202 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:58.230276    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:27:58.236202 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:58.234334    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:27:58.240151 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:58.240131    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:27:58.254860 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:58.254840    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"
Nov 01 03:27:58.374713 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:58.374646    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa25001-m46sl"
Nov 01 03:27:58.448093 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:58.448055    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f68d831-6076-4bf4-806d-742446d2dfbf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f68d831-6076-4bf4-806d-742446d2dfbf-maas-api-xks-ca\") pod \"4f68d831-6076-4bf4-806d-742446d2dfbf\" (UID: \"4f68d831-6076-4bf4-806d-742446d2dfbf\") "
Nov 01 03:27:58.448313 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:58.448291    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f68d831-6076-4bf4-806d-742446d2dfbf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f68d831-6076-4bf4-806d-742446d2dfbf-maas-api-openshift-ca\") pod \"4f68d831-6076-4bf4-806d-742446d2dfbf\" (UID: \"4f68d831-6076-4bf4-806d-742446d2dfbf\") "
Nov 01 03:27:58.448461 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:58.448441    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f68d831-6076-4bf4-806d-742446d2dfbf-kube-api-access-f7vfm\" (UniqueName: \"kubernetes.io/projected/4f68d831-6076-4bf4-806d-742446d2dfbf-kube-api-access-f7vfm\") pod \"4f68d831-6076-4bf4-806d-742446d2dfbf\" (UID: \"4f68d831-6076-4bf4-806d-742446d2dfbf\") "
Nov 01 03:27:58.448734 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:58.448704    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f68d831-6076-4bf4-806d-742446d2dfbf-maas-api-openshift-ca" pod "4f68d831-6076-4bf4-806d-742446d2dfbf" (UID: "4f68d831-6076-4bf4-806d-742446d2dfbf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:27:58.449147 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:58.449121    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f68d831-6076-4bf4-806d-742446d2dfbf-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:27:58.450541 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:58.450518    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f68d831-6076-4bf4-806d-742446d2dfbf-kube-api-access-f7vfm" pod "4f68d831-6076-4bf4-806d-742446d2dfbf" (UID: "4f68d831-6076-4bf4-806d-742446d2dfbf"). InnerVolumeSpecName "kube-api-access-f7vfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:27:58.451034 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:58.451011    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f68d831-6076-4bf4-806d-742446d2dfbf-maas-api-xks-ca" pod "4f68d831-6076-4bf4-806d-742446d2dfbf" (UID: "4f68d831-6076-4bf4-806d-742446d2dfbf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:27:58.550473 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:58.550440    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f68d831-6076-4bf4-806d-742446d2dfbf-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:27:58.550473 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:58.550472    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7vfm\" (UniqueName: \"kubernetes.io/projected/4f68d831-6076-4bf4-806d-742446d2dfbf-kube-api-access-f7vfm\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:27:59.142684 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:59.142646    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:27:59.199483 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:59.199451    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa25001-m46sl"
Nov 01 03:27:59.199680 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:59.199445    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afa25001-m46sl" event={"ID":"4f68d831-6076-4bf4-806d-742446d2dfbf","Type":"ContainerDied","Data":"4238ff42079a2ee08a25198831d2258d823d2f231c0654ae4f9d1e5468880482"}
Nov 01 03:27:59.199680 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:27:59.199610    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4238ff42079a2ee08a25198831d2258d823d2f231c0654ae4f9d1e5468880482"
Nov 01 03:28:00.312004 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:00.311968    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-124f550e-hrtsq"]
Nov 01 03:28:00.312713 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:00.312695    4930 cpu_manager.go:414] "containerMap: removing container" podUID="0f046a7c-23d0-4995-88a1-2046b50666e6" containerName="revoke-keys"
Nov 01 03:28:00.312713 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:00.312713    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f046a7c-23d0-4995-88a1-2046b50666e6" containerName="revoke-keys"
Nov 01 03:28:00.312841 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:00.312754    4930 cpu_manager.go:414] "containerMap: removing container" podUID="4f68d831-6076-4bf4-806d-742446d2dfbf" containerName="revoke-keys"
Nov 01 03:28:00.312841 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:00.312762    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f68d831-6076-4bf4-806d-742446d2dfbf" containerName="revoke-keys"
Nov 01 03:28:00.312913 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:00.312896    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f68d831-6076-4bf4-806d-742446d2dfbf" containerName="revoke-keys"
Nov 01 03:28:00.312945 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:00.312914    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f046a7c-23d0-4995-88a1-2046b50666e6" containerName="revoke-keys"
Nov 01 03:28:00.320934 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:00.320904    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-124f550e-hrtsq"
Nov 01 03:28:00.362045 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:00.362013    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-124f550e-hrtsq"]
Nov 01 03:28:00.373335 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:00.373303    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:28:00.373656 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:00.373622    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:28:00.373656 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:00.373653    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:28:00.374036 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:00.374000    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:28:00.392888 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:00.392861    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhg94\" (UniqueName: \"kubernetes.io/projected/17c52d6e-b715-45f7-9cab-31b383dbce20-kube-api-access-fhg94\") pod \"maas-api-revoke-sub-124f550e-hrtsq\" (UID: \"17c52d6e-b715-45f7-9cab-31b383dbce20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-124f550e-hrtsq"
Nov 01 03:28:00.393075 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:00.393049    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17c52d6e-b715-45f7-9cab-31b383dbce20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-124f550e-hrtsq\" (UID: \"17c52d6e-b715-45f7-9cab-31b383dbce20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-124f550e-hrtsq"
Nov 01 03:28:00.393307 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:00.393286    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17c52d6e-b715-45f7-9cab-31b383dbce20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-124f550e-hrtsq\" (UID: \"17c52d6e-b715-45f7-9cab-31b383dbce20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-124f550e-hrtsq"
Nov 01 03:28:00.495090 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:00.495052    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17c52d6e-b715-45f7-9cab-31b383dbce20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-124f550e-hrtsq\" (UID: \"17c52d6e-b715-45f7-9cab-31b383dbce20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-124f550e-hrtsq"
Nov 01 03:28:00.495267 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:00.495187    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17c52d6e-b715-45f7-9cab-31b383dbce20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-124f550e-hrtsq\" (UID: \"17c52d6e-b715-45f7-9cab-31b383dbce20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-124f550e-hrtsq"
Nov 01 03:28:00.495267 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:00.495228    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhg94\" (UniqueName: \"kubernetes.io/projected/17c52d6e-b715-45f7-9cab-31b383dbce20-kube-api-access-fhg94\") pod \"maas-api-revoke-sub-124f550e-hrtsq\" (UID: \"17c52d6e-b715-45f7-9cab-31b383dbce20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-124f550e-hrtsq"
Nov 01 03:28:00.495915 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:00.495891    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17c52d6e-b715-45f7-9cab-31b383dbce20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-124f550e-hrtsq\" (UID: \"17c52d6e-b715-45f7-9cab-31b383dbce20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-124f550e-hrtsq"
Nov 01 03:28:00.497647 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:00.497622    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17c52d6e-b715-45f7-9cab-31b383dbce20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-124f550e-hrtsq\" (UID: \"17c52d6e-b715-45f7-9cab-31b383dbce20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-124f550e-hrtsq"
Nov 01 03:28:00.556389 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:00.556350    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhg94\" (UniqueName: \"kubernetes.io/projected/17c52d6e-b715-45f7-9cab-31b383dbce20-kube-api-access-fhg94\") pod \"maas-api-revoke-sub-124f550e-hrtsq\" (UID: \"17c52d6e-b715-45f7-9cab-31b383dbce20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-124f550e-hrtsq"
Nov 01 03:28:00.637126 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:00.637091    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-124f550e-hrtsq"
Nov 01 03:28:00.943251 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:00.943171    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ed506bf-t2d2h"]
Nov 01 03:28:00.948999 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:00.948974    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed506bf-t2d2h"
Nov 01 03:28:01.005102 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:01.004356    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30aa6d42-2084-4a5c-a1c0-34037d291406-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ed506bf-t2d2h\" (UID: \"30aa6d42-2084-4a5c-a1c0-34037d291406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed506bf-t2d2h"
Nov 01 03:28:01.005102 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:01.004512    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w5d4\" (UniqueName: \"kubernetes.io/projected/30aa6d42-2084-4a5c-a1c0-34037d291406-kube-api-access-6w5d4\") pod \"maas-api-revoke-sub-6ed506bf-t2d2h\" (UID: \"30aa6d42-2084-4a5c-a1c0-34037d291406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed506bf-t2d2h"
Nov 01 03:28:01.005349 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:01.005281    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30aa6d42-2084-4a5c-a1c0-34037d291406-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ed506bf-t2d2h\" (UID: \"30aa6d42-2084-4a5c-a1c0-34037d291406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed506bf-t2d2h"
Nov 01 03:28:01.012598 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:01.009410    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ed506bf-t2d2h"]
Nov 01 03:28:01.033549 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:28:01.033518    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17c52d6e_b715_45f7_9cab_31b383dbce20.slice/crio-3b8017666ecb1aaedd1e5ed44cb1e6d6968ba2070a1c17e14536965cf192f98a WatchSource:0}: Error finding container 3b8017666ecb1aaedd1e5ed44cb1e6d6968ba2070a1c17e14536965cf192f98a: Status 404 returned error can't find the container with id 3b8017666ecb1aaedd1e5ed44cb1e6d6968ba2070a1c17e14536965cf192f98a
Nov 01 03:28:01.034167 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:01.034143    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-124f550e-hrtsq"]
Nov 01 03:28:01.107300 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:01.107268    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30aa6d42-2084-4a5c-a1c0-34037d291406-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ed506bf-t2d2h\" (UID: \"30aa6d42-2084-4a5c-a1c0-34037d291406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed506bf-t2d2h"
Nov 01 03:28:01.107473 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:01.107452    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30aa6d42-2084-4a5c-a1c0-34037d291406-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ed506bf-t2d2h\" (UID: \"30aa6d42-2084-4a5c-a1c0-34037d291406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed506bf-t2d2h"
Nov 01 03:28:01.107551 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:01.107533    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w5d4\" (UniqueName: \"kubernetes.io/projected/30aa6d42-2084-4a5c-a1c0-34037d291406-kube-api-access-6w5d4\") pod \"maas-api-revoke-sub-6ed506bf-t2d2h\" (UID: \"30aa6d42-2084-4a5c-a1c0-34037d291406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed506bf-t2d2h"
Nov 01 03:28:01.108091 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:01.108065    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30aa6d42-2084-4a5c-a1c0-34037d291406-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ed506bf-t2d2h\" (UID: \"30aa6d42-2084-4a5c-a1c0-34037d291406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed506bf-t2d2h"
Nov 01 03:28:01.109771 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:01.109753    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30aa6d42-2084-4a5c-a1c0-34037d291406-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ed506bf-t2d2h\" (UID: \"30aa6d42-2084-4a5c-a1c0-34037d291406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed506bf-t2d2h"
Nov 01 03:28:01.169667 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:01.169632    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w5d4\" (UniqueName: \"kubernetes.io/projected/30aa6d42-2084-4a5c-a1c0-34037d291406-kube-api-access-6w5d4\") pod \"maas-api-revoke-sub-6ed506bf-t2d2h\" (UID: \"30aa6d42-2084-4a5c-a1c0-34037d291406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed506bf-t2d2h"
Nov 01 03:28:01.247482 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:01.247403    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-124f550e-hrtsq" event={"ID":"17c52d6e-b715-45f7-9cab-31b383dbce20","Type":"ContainerStarted","Data":"3b8017666ecb1aaedd1e5ed44cb1e6d6968ba2070a1c17e14536965cf192f98a"}
Nov 01 03:28:01.268324 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:01.268287    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed506bf-t2d2h"
Nov 01 03:28:01.678330 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:01.678296    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ed506bf-t2d2h"]
Nov 01 03:28:01.682831 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:28:01.682799    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30aa6d42_2084_4a5c_a1c0_34037d291406.slice/crio-09233a436b16d2f3443a361a61f9cb01a55e28537464e356d2f0981aa4dc95f7 WatchSource:0}: Error finding container 09233a436b16d2f3443a361a61f9cb01a55e28537464e356d2f0981aa4dc95f7: Status 404 returned error can't find the container with id 09233a436b16d2f3443a361a61f9cb01a55e28537464e356d2f0981aa4dc95f7
Nov 01 03:28:02.260863 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:02.260766    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed506bf-t2d2h" event={"ID":"30aa6d42-2084-4a5c-a1c0-34037d291406","Type":"ContainerStarted","Data":"377247bfcd6ad86da38fbd46356f84da00db77ffce2ab256ed6fa9c07c8a3a40"}
Nov 01 03:28:02.261154 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:02.261123    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed506bf-t2d2h" event={"ID":"30aa6d42-2084-4a5c-a1c0-34037d291406","Type":"ContainerStarted","Data":"09233a436b16d2f3443a361a61f9cb01a55e28537464e356d2f0981aa4dc95f7"}
Nov 01 03:28:02.264178 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:02.264148    4930 generic.go:356] "Generic (PLEG): container finished" podID="17c52d6e-b715-45f7-9cab-31b383dbce20" containerID="53caacd8ae17df838ae25c003f702df8ff477df23952408adbb29225d4298599" exitCode=0
Nov 01 03:28:02.264317 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:02.264265    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-124f550e-hrtsq" event={"ID":"17c52d6e-b715-45f7-9cab-31b383dbce20","Type":"ContainerDied","Data":"53caacd8ae17df838ae25c003f702df8ff477df23952408adbb29225d4298599"}
Nov 01 03:28:02.379051 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:02.378983    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed506bf-t2d2h" podStartSLOduration=2.37896635 podStartE2EDuration="2.37896635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:28:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:28:02.377465213 +0000 UTC m=+1286.900471805" watchObservedRunningTime="2026-11-01 03:28:02.37896635 +0000 UTC m=+1286.901972938"
Nov 01 03:28:03.051418 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:03.051384    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-495d744b-wglnt"]
Nov 01 03:28:03.101259 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:03.101226    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-495d744b-wglnt"]
Nov 01 03:28:03.276048 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:03.276009    4930 generic.go:356] "Generic (PLEG): container finished" podID="30aa6d42-2084-4a5c-a1c0-34037d291406" containerID="377247bfcd6ad86da38fbd46356f84da00db77ffce2ab256ed6fa9c07c8a3a40" exitCode=0
Nov 01 03:28:03.276224 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:03.276100    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed506bf-t2d2h" event={"ID":"30aa6d42-2084-4a5c-a1c0-34037d291406","Type":"ContainerDied","Data":"377247bfcd6ad86da38fbd46356f84da00db77ffce2ab256ed6fa9c07c8a3a40"}
Nov 01 03:28:03.459661 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:03.459632    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-124f550e-hrtsq"
Nov 01 03:28:03.543338 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:03.543306    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17c52d6e-b715-45f7-9cab-31b383dbce20-kube-api-access-fhg94\" (UniqueName: \"kubernetes.io/projected/17c52d6e-b715-45f7-9cab-31b383dbce20-kube-api-access-fhg94\") pod \"17c52d6e-b715-45f7-9cab-31b383dbce20\" (UID: \"17c52d6e-b715-45f7-9cab-31b383dbce20\") "
Nov 01 03:28:03.543493 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:03.543405    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17c52d6e-b715-45f7-9cab-31b383dbce20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17c52d6e-b715-45f7-9cab-31b383dbce20-maas-api-xks-ca\") pod \"17c52d6e-b715-45f7-9cab-31b383dbce20\" (UID: \"17c52d6e-b715-45f7-9cab-31b383dbce20\") "
Nov 01 03:28:03.543624 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:03.543546    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17c52d6e-b715-45f7-9cab-31b383dbce20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17c52d6e-b715-45f7-9cab-31b383dbce20-maas-api-openshift-ca\") pod \"17c52d6e-b715-45f7-9cab-31b383dbce20\" (UID: \"17c52d6e-b715-45f7-9cab-31b383dbce20\") "
Nov 01 03:28:03.543860 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:03.543834    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17c52d6e-b715-45f7-9cab-31b383dbce20-maas-api-openshift-ca" pod "17c52d6e-b715-45f7-9cab-31b383dbce20" (UID: "17c52d6e-b715-45f7-9cab-31b383dbce20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:28:03.544158 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:03.544143    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17c52d6e-b715-45f7-9cab-31b383dbce20-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:03.545417 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:03.545391    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17c52d6e-b715-45f7-9cab-31b383dbce20-kube-api-access-fhg94" pod "17c52d6e-b715-45f7-9cab-31b383dbce20" (UID: "17c52d6e-b715-45f7-9cab-31b383dbce20"). InnerVolumeSpecName "kube-api-access-fhg94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:28:03.545514 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:03.545461    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17c52d6e-b715-45f7-9cab-31b383dbce20-maas-api-xks-ca" pod "17c52d6e-b715-45f7-9cab-31b383dbce20" (UID: "17c52d6e-b715-45f7-9cab-31b383dbce20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:28:03.625489 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:03.625450    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80c056a8-53c1-4778-af88-3a8ac446a6d1" path="/var/lib/kubelet/pods/80c056a8-53c1-4778-af88-3a8ac446a6d1/volumes"
Nov 01 03:28:03.645059 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:03.645037    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhg94\" (UniqueName: \"kubernetes.io/projected/17c52d6e-b715-45f7-9cab-31b383dbce20-kube-api-access-fhg94\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:03.645059 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:03.645060    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17c52d6e-b715-45f7-9cab-31b383dbce20-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:03.938074 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:03.937976    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:04.298907 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:04.298817    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-124f550e-hrtsq" event={"ID":"17c52d6e-b715-45f7-9cab-31b383dbce20","Type":"ContainerDied","Data":"3b8017666ecb1aaedd1e5ed44cb1e6d6968ba2070a1c17e14536965cf192f98a"}
Nov 01 03:28:04.299366 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:04.298907    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b8017666ecb1aaedd1e5ed44cb1e6d6968ba2070a1c17e14536965cf192f98a"
Nov 01 03:28:04.299366 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:04.298831    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-124f550e-hrtsq"
Nov 01 03:28:04.474275 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:04.474248    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed506bf-t2d2h"
Nov 01 03:28:04.557043 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:04.556968    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30aa6d42-2084-4a5c-a1c0-34037d291406-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30aa6d42-2084-4a5c-a1c0-34037d291406-maas-api-openshift-ca\") pod \"30aa6d42-2084-4a5c-a1c0-34037d291406\" (UID: \"30aa6d42-2084-4a5c-a1c0-34037d291406\") "
Nov 01 03:28:04.557209 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:04.557125    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30aa6d42-2084-4a5c-a1c0-34037d291406-kube-api-access-6w5d4\" (UniqueName: \"kubernetes.io/projected/30aa6d42-2084-4a5c-a1c0-34037d291406-kube-api-access-6w5d4\") pod \"30aa6d42-2084-4a5c-a1c0-34037d291406\" (UID: \"30aa6d42-2084-4a5c-a1c0-34037d291406\") "
Nov 01 03:28:04.557209 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:04.557176    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30aa6d42-2084-4a5c-a1c0-34037d291406-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30aa6d42-2084-4a5c-a1c0-34037d291406-maas-api-xks-ca\") pod \"30aa6d42-2084-4a5c-a1c0-34037d291406\" (UID: \"30aa6d42-2084-4a5c-a1c0-34037d291406\") "
Nov 01 03:28:04.557326 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:04.557299    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30aa6d42-2084-4a5c-a1c0-34037d291406-maas-api-openshift-ca" pod "30aa6d42-2084-4a5c-a1c0-34037d291406" (UID: "30aa6d42-2084-4a5c-a1c0-34037d291406"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:28:04.557860 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:04.557828    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30aa6d42-2084-4a5c-a1c0-34037d291406-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:04.559208 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:04.559186    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30aa6d42-2084-4a5c-a1c0-34037d291406-kube-api-access-6w5d4" pod "30aa6d42-2084-4a5c-a1c0-34037d291406" (UID: "30aa6d42-2084-4a5c-a1c0-34037d291406"). InnerVolumeSpecName "kube-api-access-6w5d4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:28:04.559315 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:04.559271    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30aa6d42-2084-4a5c-a1c0-34037d291406-maas-api-xks-ca" pod "30aa6d42-2084-4a5c-a1c0-34037d291406" (UID: "30aa6d42-2084-4a5c-a1c0-34037d291406"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:28:04.659381 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:04.659348    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w5d4\" (UniqueName: \"kubernetes.io/projected/30aa6d42-2084-4a5c-a1c0-34037d291406-kube-api-access-6w5d4\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:04.659381 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:04.659379    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30aa6d42-2084-4a5c-a1c0-34037d291406-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:05.051484 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:05.051448    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dee007ba-w6t4t"]
Nov 01 03:28:05.055993 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:05.055970    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dee007ba-w6t4t"]
Nov 01 03:28:05.308539 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:05.308452    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed506bf-t2d2h"
Nov 01 03:28:05.308539 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:05.308507    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ed506bf-t2d2h" event={"ID":"30aa6d42-2084-4a5c-a1c0-34037d291406","Type":"ContainerDied","Data":"09233a436b16d2f3443a361a61f9cb01a55e28537464e356d2f0981aa4dc95f7"}
Nov 01 03:28:05.309050 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:05.308572    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09233a436b16d2f3443a361a61f9cb01a55e28537464e356d2f0981aa4dc95f7"
Nov 01 03:28:05.623241 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:05.623210    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="505adf30-6661-40ae-969b-5936e9ae4879" path="/var/lib/kubelet/pods/505adf30-6661-40ae-969b-5936e9ae4879/volumes"
Nov 01 03:28:06.057530 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:06.057443    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d752352-2z46j"]
Nov 01 03:28:06.062021 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:06.061995    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d752352-2z46j"]
Nov 01 03:28:07.622831 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:07.622800    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a57688db-f33f-4e22-a356-48b95564e4ae" path="/var/lib/kubelet/pods/a57688db-f33f-4e22-a356-48b95564e4ae/volumes"
Nov 01 03:28:08.771032 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:08.770989    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:09.018194 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:09.018160    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-305021d0-6nlmc"]
Nov 01 03:28:09.018892 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:09.018876    4930 cpu_manager.go:414] "containerMap: removing container" podUID="17c52d6e-b715-45f7-9cab-31b383dbce20" containerName="revoke-keys"
Nov 01 03:28:09.018956 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:09.018893    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17c52d6e-b715-45f7-9cab-31b383dbce20" containerName="revoke-keys"
Nov 01 03:28:09.018956 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:09.018921    4930 cpu_manager.go:414] "containerMap: removing container" podUID="30aa6d42-2084-4a5c-a1c0-34037d291406" containerName="revoke-keys"
Nov 01 03:28:09.018956 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:09.018927    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30aa6d42-2084-4a5c-a1c0-34037d291406" containerName="revoke-keys"
Nov 01 03:28:09.019049 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:09.019028    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="17c52d6e-b715-45f7-9cab-31b383dbce20" containerName="revoke-keys"
Nov 01 03:28:09.019049 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:09.019039    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="30aa6d42-2084-4a5c-a1c0-34037d291406" containerName="revoke-keys"
Nov 01 03:28:09.035963 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:09.035893    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-305021d0-6nlmc"
Nov 01 03:28:09.088651 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:09.088628    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:28:09.088809 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:09.088794    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:28:09.136902 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:09.136867    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2242792-3f71-4878-9b72-4f2bac2c3f76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-305021d0-6nlmc\" (UID: \"d2242792-3f71-4878-9b72-4f2bac2c3f76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-305021d0-6nlmc"
Nov 01 03:28:09.137068 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:09.136981    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2242792-3f71-4878-9b72-4f2bac2c3f76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-305021d0-6nlmc\" (UID: \"d2242792-3f71-4878-9b72-4f2bac2c3f76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-305021d0-6nlmc"
Nov 01 03:28:09.137068 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:09.137024    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-488br\" (UniqueName: \"kubernetes.io/projected/d2242792-3f71-4878-9b72-4f2bac2c3f76-kube-api-access-488br\") pod \"maas-api-revoke-sub-305021d0-6nlmc\" (UID: \"d2242792-3f71-4878-9b72-4f2bac2c3f76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-305021d0-6nlmc"
Nov 01 03:28:09.138328 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:09.138302    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:28:09.138514 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:09.138495    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:28:09.143462 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:09.143438    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-305021d0-6nlmc"]
Nov 01 03:28:09.239414 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:09.239374    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2242792-3f71-4878-9b72-4f2bac2c3f76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-305021d0-6nlmc\" (UID: \"d2242792-3f71-4878-9b72-4f2bac2c3f76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-305021d0-6nlmc"
Nov 01 03:28:09.239627 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:09.239521    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2242792-3f71-4878-9b72-4f2bac2c3f76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-305021d0-6nlmc\" (UID: \"d2242792-3f71-4878-9b72-4f2bac2c3f76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-305021d0-6nlmc"
Nov 01 03:28:09.240482 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:09.240453    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2242792-3f71-4878-9b72-4f2bac2c3f76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-305021d0-6nlmc\" (UID: \"d2242792-3f71-4878-9b72-4f2bac2c3f76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-305021d0-6nlmc"
Nov 01 03:28:09.241132 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:09.241107    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-488br\" (UniqueName: \"kubernetes.io/projected/d2242792-3f71-4878-9b72-4f2bac2c3f76-kube-api-access-488br\") pod \"maas-api-revoke-sub-305021d0-6nlmc\" (UID: \"d2242792-3f71-4878-9b72-4f2bac2c3f76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-305021d0-6nlmc"
Nov 01 03:28:09.252940 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:09.252920    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2242792-3f71-4878-9b72-4f2bac2c3f76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-305021d0-6nlmc\" (UID: \"d2242792-3f71-4878-9b72-4f2bac2c3f76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-305021d0-6nlmc"
Nov 01 03:28:09.302401 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:09.302337    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-488br\" (UniqueName: \"kubernetes.io/projected/d2242792-3f71-4878-9b72-4f2bac2c3f76-kube-api-access-488br\") pod \"maas-api-revoke-sub-305021d0-6nlmc\" (UID: \"d2242792-3f71-4878-9b72-4f2bac2c3f76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-305021d0-6nlmc"
Nov 01 03:28:09.351606 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:09.351580    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-305021d0-6nlmc"
Nov 01 03:28:09.737354 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:28:09.737322    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2242792_3f71_4878_9b72_4f2bac2c3f76.slice/crio-dea04c3d7c605a00880b00fc56a8e1f23ad837eee9be84ad0e8b027957711b3c WatchSource:0}: Error finding container dea04c3d7c605a00880b00fc56a8e1f23ad837eee9be84ad0e8b027957711b3c: Status 404 returned error can't find the container with id dea04c3d7c605a00880b00fc56a8e1f23ad837eee9be84ad0e8b027957711b3c
Nov 01 03:28:09.738286 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:09.738209    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-305021d0-6nlmc"]
Nov 01 03:28:10.259175 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:10.259146    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-184a059e-jv2ln"]
Nov 01 03:28:10.263964 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:10.263922    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-184a059e-jv2ln"
Nov 01 03:28:10.322903 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:10.322837    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-184a059e-jv2ln"]
Nov 01 03:28:10.358682 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:10.358648    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc079b72-10cb-4437-afa0-74902f4ef55c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-184a059e-jv2ln\" (UID: \"dc079b72-10cb-4437-afa0-74902f4ef55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-184a059e-jv2ln"
Nov 01 03:28:10.358856 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:10.358705    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64jzx\" (UniqueName: \"kubernetes.io/projected/dc079b72-10cb-4437-afa0-74902f4ef55c-kube-api-access-64jzx\") pod \"maas-api-revoke-sub-184a059e-jv2ln\" (UID: \"dc079b72-10cb-4437-afa0-74902f4ef55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-184a059e-jv2ln"
Nov 01 03:28:10.359008 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:10.358979    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc079b72-10cb-4437-afa0-74902f4ef55c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-184a059e-jv2ln\" (UID: \"dc079b72-10cb-4437-afa0-74902f4ef55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-184a059e-jv2ln"
Nov 01 03:28:10.361444 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:10.361414    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-305021d0-6nlmc" event={"ID":"d2242792-3f71-4878-9b72-4f2bac2c3f76","Type":"ContainerStarted","Data":"e012cf7b52a7a0e003450fd4b0b745a6bfefe21319bcf0ceefe423df98e392e4"}
Nov 01 03:28:10.361540 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:10.361480    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-305021d0-6nlmc" event={"ID":"d2242792-3f71-4878-9b72-4f2bac2c3f76","Type":"ContainerStarted","Data":"dea04c3d7c605a00880b00fc56a8e1f23ad837eee9be84ad0e8b027957711b3c"}
Nov 01 03:28:10.410920 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:10.410891    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8f22c0d-827rx"]
Nov 01 03:28:10.415192 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:10.415175    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f22c0d-827rx"
Nov 01 03:28:10.460252 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:10.460226    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bfb43fb-7e73-4be2-80f2-e4361bfa096a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8f22c0d-827rx\" (UID: \"6bfb43fb-7e73-4be2-80f2-e4361bfa096a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f22c0d-827rx"
Nov 01 03:28:10.460392 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:10.460285    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc079b72-10cb-4437-afa0-74902f4ef55c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-184a059e-jv2ln\" (UID: \"dc079b72-10cb-4437-afa0-74902f4ef55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-184a059e-jv2ln"
Nov 01 03:28:10.460392 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:10.460356    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64jzx\" (UniqueName: \"kubernetes.io/projected/dc079b72-10cb-4437-afa0-74902f4ef55c-kube-api-access-64jzx\") pod \"maas-api-revoke-sub-184a059e-jv2ln\" (UID: \"dc079b72-10cb-4437-afa0-74902f4ef55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-184a059e-jv2ln"
Nov 01 03:28:10.460539 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:10.460397    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bfb43fb-7e73-4be2-80f2-e4361bfa096a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8f22c0d-827rx\" (UID: \"6bfb43fb-7e73-4be2-80f2-e4361bfa096a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f22c0d-827rx"
Nov 01 03:28:10.460642 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:10.460574    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2btq\" (UniqueName: \"kubernetes.io/projected/6bfb43fb-7e73-4be2-80f2-e4361bfa096a-kube-api-access-r2btq\") pod \"maas-api-revoke-sub-c8f22c0d-827rx\" (UID: \"6bfb43fb-7e73-4be2-80f2-e4361bfa096a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f22c0d-827rx"
Nov 01 03:28:10.460642 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:10.460629    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc079b72-10cb-4437-afa0-74902f4ef55c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-184a059e-jv2ln\" (UID: \"dc079b72-10cb-4437-afa0-74902f4ef55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-184a059e-jv2ln"
Nov 01 03:28:10.461225 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:10.461202    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc079b72-10cb-4437-afa0-74902f4ef55c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-184a059e-jv2ln\" (UID: \"dc079b72-10cb-4437-afa0-74902f4ef55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-184a059e-jv2ln"
Nov 01 03:28:10.463097 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:10.463071    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc079b72-10cb-4437-afa0-74902f4ef55c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-184a059e-jv2ln\" (UID: \"dc079b72-10cb-4437-afa0-74902f4ef55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-184a059e-jv2ln"
Nov 01 03:28:10.477269 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:10.477247    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8f22c0d-827rx"]
Nov 01 03:28:10.515844 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:10.515790    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-305021d0-6nlmc" podStartSLOduration=2.5157725060000002 podStartE2EDuration="2.515772506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:28:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:28:10.512990404 +0000 UTC m=+1295.035996992" watchObservedRunningTime="2026-11-01 03:28:10.515772506 +0000 UTC m=+1295.038779100"
Nov 01 03:28:10.539534 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:10.539410    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64jzx\" (UniqueName: \"kubernetes.io/projected/dc079b72-10cb-4437-afa0-74902f4ef55c-kube-api-access-64jzx\") pod \"maas-api-revoke-sub-184a059e-jv2ln\" (UID: \"dc079b72-10cb-4437-afa0-74902f4ef55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-184a059e-jv2ln"
Nov 01 03:28:10.562659 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:10.562635    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bfb43fb-7e73-4be2-80f2-e4361bfa096a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8f22c0d-827rx\" (UID: \"6bfb43fb-7e73-4be2-80f2-e4361bfa096a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f22c0d-827rx"
Nov 01 03:28:10.562854 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:10.562704    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bfb43fb-7e73-4be2-80f2-e4361bfa096a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8f22c0d-827rx\" (UID: \"6bfb43fb-7e73-4be2-80f2-e4361bfa096a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f22c0d-827rx"
Nov 01 03:28:10.562854 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:10.562779    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2btq\" (UniqueName: \"kubernetes.io/projected/6bfb43fb-7e73-4be2-80f2-e4361bfa096a-kube-api-access-r2btq\") pod \"maas-api-revoke-sub-c8f22c0d-827rx\" (UID: \"6bfb43fb-7e73-4be2-80f2-e4361bfa096a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f22c0d-827rx"
Nov 01 03:28:10.563517 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:10.563492    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bfb43fb-7e73-4be2-80f2-e4361bfa096a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8f22c0d-827rx\" (UID: \"6bfb43fb-7e73-4be2-80f2-e4361bfa096a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f22c0d-827rx"
Nov 01 03:28:10.565408 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:10.565385    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bfb43fb-7e73-4be2-80f2-e4361bfa096a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8f22c0d-827rx\" (UID: \"6bfb43fb-7e73-4be2-80f2-e4361bfa096a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f22c0d-827rx"
Nov 01 03:28:10.579186 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:10.579130    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-184a059e-jv2ln"
Nov 01 03:28:10.632323 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:10.631988    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2btq\" (UniqueName: \"kubernetes.io/projected/6bfb43fb-7e73-4be2-80f2-e4361bfa096a-kube-api-access-r2btq\") pod \"maas-api-revoke-sub-c8f22c0d-827rx\" (UID: \"6bfb43fb-7e73-4be2-80f2-e4361bfa096a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f22c0d-827rx"
Nov 01 03:28:10.729371 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:10.729343    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f22c0d-827rx"
Nov 01 03:28:10.979872 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:10.979840    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-184a059e-jv2ln"]
Nov 01 03:28:11.054895 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.054862    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4304e0c2-86r5h"]
Nov 01 03:28:11.059591 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.059545    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4304e0c2-86r5h"]
Nov 01 03:28:11.130010 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:28:11.129980    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bfb43fb_7e73_4be2_80f2_e4361bfa096a.slice/crio-9beead8a3dc3cfdcbc5455d2aa3ad9ccab0e636e06b43360aee58d1d200d3365 WatchSource:0}: Error finding container 9beead8a3dc3cfdcbc5455d2aa3ad9ccab0e636e06b43360aee58d1d200d3365: Status 404 returned error can't find the container with id 9beead8a3dc3cfdcbc5455d2aa3ad9ccab0e636e06b43360aee58d1d200d3365
Nov 01 03:28:11.133702 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.133673    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8f22c0d-827rx"]
Nov 01 03:28:11.372279 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.372245    4930 generic.go:356] "Generic (PLEG): container finished" podID="d2242792-3f71-4878-9b72-4f2bac2c3f76" containerID="e012cf7b52a7a0e003450fd4b0b745a6bfefe21319bcf0ceefe423df98e392e4" exitCode=0
Nov 01 03:28:11.372676 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.372326    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-305021d0-6nlmc" event={"ID":"d2242792-3f71-4878-9b72-4f2bac2c3f76","Type":"ContainerDied","Data":"e012cf7b52a7a0e003450fd4b0b745a6bfefe21319bcf0ceefe423df98e392e4"}
Nov 01 03:28:11.375958 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.375640    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f22c0d-827rx" event={"ID":"6bfb43fb-7e73-4be2-80f2-e4361bfa096a","Type":"ContainerStarted","Data":"9beead8a3dc3cfdcbc5455d2aa3ad9ccab0e636e06b43360aee58d1d200d3365"}
Nov 01 03:28:11.377519 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.377498    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-184a059e-jv2ln" event={"ID":"dc079b72-10cb-4437-afa0-74902f4ef55c","Type":"ContainerStarted","Data":"0ef445132fbfbb0ca11ed5a73d5be220e66aac2783119b55ed6a1ec8c8bda494"}
Nov 01 03:28:11.516953 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.516668    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz"]
Nov 01 03:28:11.521894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.521837    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz"
Nov 01 03:28:11.586711 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.586652    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27bad768-f9df-4649-a979-2a459327f07d-istio-podinfo\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz\" (UID: \"27bad768-f9df-4649-a979-2a459327f07d\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz"
Nov 01 03:28:11.586867 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.586795    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27bad768-f9df-4649-a979-2a459327f07d-workload-socket\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz\" (UID: \"27bad768-f9df-4649-a979-2a459327f07d\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz"
Nov 01 03:28:11.586929 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.586868    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7wxq\" (UniqueName: \"kubernetes.io/projected/27bad768-f9df-4649-a979-2a459327f07d-kube-api-access-m7wxq\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz\" (UID: \"27bad768-f9df-4649-a979-2a459327f07d\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz"
Nov 01 03:28:11.586929 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.586912    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27bad768-f9df-4649-a979-2a459327f07d-workload-certs\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz\" (UID: \"27bad768-f9df-4649-a979-2a459327f07d\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz"
Nov 01 03:28:11.587610 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.587090    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27bad768-f9df-4649-a979-2a459327f07d-istio-data\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz\" (UID: \"27bad768-f9df-4649-a979-2a459327f07d\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz"
Nov 01 03:28:11.587610 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.587115    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27bad768-f9df-4649-a979-2a459327f07d-istiod-ca-cert\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz\" (UID: \"27bad768-f9df-4649-a979-2a459327f07d\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz"
Nov 01 03:28:11.587610 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.587214    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27bad768-f9df-4649-a979-2a459327f07d-credential-socket\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz\" (UID: \"27bad768-f9df-4649-a979-2a459327f07d\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz"
Nov 01 03:28:11.587610 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.587273    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27bad768-f9df-4649-a979-2a459327f07d-istio-token\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz\" (UID: \"27bad768-f9df-4649-a979-2a459327f07d\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz"
Nov 01 03:28:11.587610 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.587392    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27bad768-f9df-4649-a979-2a459327f07d-istio-envoy\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz\" (UID: \"27bad768-f9df-4649-a979-2a459327f07d\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz"
Nov 01 03:28:11.625838 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.625792    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6da1345e-9204-4023-b610-f31e4c3c8250" path="/var/lib/kubelet/pods/6da1345e-9204-4023-b610-f31e4c3c8250/volumes"
Nov 01 03:28:11.626281 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.626264    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz"]
Nov 01 03:28:11.688992 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.688958    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27bad768-f9df-4649-a979-2a459327f07d-istio-podinfo\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz\" (UID: \"27bad768-f9df-4649-a979-2a459327f07d\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz"
Nov 01 03:28:11.689164 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.689004    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27bad768-f9df-4649-a979-2a459327f07d-workload-socket\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz\" (UID: \"27bad768-f9df-4649-a979-2a459327f07d\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz"
Nov 01 03:28:11.689164 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.689040    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7wxq\" (UniqueName: \"kubernetes.io/projected/27bad768-f9df-4649-a979-2a459327f07d-kube-api-access-m7wxq\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz\" (UID: \"27bad768-f9df-4649-a979-2a459327f07d\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz"
Nov 01 03:28:11.689164 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.689068    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27bad768-f9df-4649-a979-2a459327f07d-workload-certs\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz\" (UID: \"27bad768-f9df-4649-a979-2a459327f07d\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz"
Nov 01 03:28:11.689164 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.689163    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27bad768-f9df-4649-a979-2a459327f07d-istio-data\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz\" (UID: \"27bad768-f9df-4649-a979-2a459327f07d\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz"
Nov 01 03:28:11.689383 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.689185    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27bad768-f9df-4649-a979-2a459327f07d-istiod-ca-cert\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz\" (UID: \"27bad768-f9df-4649-a979-2a459327f07d\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz"
Nov 01 03:28:11.689383 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.689288    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27bad768-f9df-4649-a979-2a459327f07d-credential-socket\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz\" (UID: \"27bad768-f9df-4649-a979-2a459327f07d\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz"
Nov 01 03:28:11.689383 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.689347    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27bad768-f9df-4649-a979-2a459327f07d-istio-token\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz\" (UID: \"27bad768-f9df-4649-a979-2a459327f07d\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz"
Nov 01 03:28:11.689547 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.689502    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27bad768-f9df-4649-a979-2a459327f07d-istio-envoy\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz\" (UID: \"27bad768-f9df-4649-a979-2a459327f07d\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz"
Nov 01 03:28:11.689547 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.689513    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27bad768-f9df-4649-a979-2a459327f07d-workload-certs\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz\" (UID: \"27bad768-f9df-4649-a979-2a459327f07d\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz"
Nov 01 03:28:11.689681 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.689637    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27bad768-f9df-4649-a979-2a459327f07d-istio-data\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz\" (UID: \"27bad768-f9df-4649-a979-2a459327f07d\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz"
Nov 01 03:28:11.689810 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.689783    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27bad768-f9df-4649-a979-2a459327f07d-credential-socket\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz\" (UID: \"27bad768-f9df-4649-a979-2a459327f07d\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz"
Nov 01 03:28:11.689973 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.689953    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27bad768-f9df-4649-a979-2a459327f07d-workload-socket\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz\" (UID: \"27bad768-f9df-4649-a979-2a459327f07d\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz"
Nov 01 03:28:11.690145 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.690123    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27bad768-f9df-4649-a979-2a459327f07d-istiod-ca-cert\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz\" (UID: \"27bad768-f9df-4649-a979-2a459327f07d\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz"
Nov 01 03:28:11.691477 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.691453    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27bad768-f9df-4649-a979-2a459327f07d-istio-podinfo\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz\" (UID: \"27bad768-f9df-4649-a979-2a459327f07d\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz"
Nov 01 03:28:11.691589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.691526    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27bad768-f9df-4649-a979-2a459327f07d-istio-envoy\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz\" (UID: \"27bad768-f9df-4649-a979-2a459327f07d\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz"
Nov 01 03:28:11.721672 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.721647    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-210e6572-lxcdd"]
Nov 01 03:28:11.727147 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.727131    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-210e6572-lxcdd"
Nov 01 03:28:11.750531 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.750499    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27bad768-f9df-4649-a979-2a459327f07d-istio-token\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz\" (UID: \"27bad768-f9df-4649-a979-2a459327f07d\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz"
Nov 01 03:28:11.750531 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.750521    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7wxq\" (UniqueName: \"kubernetes.io/projected/27bad768-f9df-4649-a979-2a459327f07d-kube-api-access-m7wxq\") pod \"e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz\" (UID: \"27bad768-f9df-4649-a979-2a459327f07d\") " pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz"
Nov 01 03:28:11.786047 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.785980    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-210e6572-lxcdd"]
Nov 01 03:28:11.793593 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.791263    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dce274d0-64ca-459a-995e-651c9d548965-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-210e6572-lxcdd\" (UID: \"dce274d0-64ca-459a-995e-651c9d548965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-210e6572-lxcdd"
Nov 01 03:28:11.793593 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.791333    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dce274d0-64ca-459a-995e-651c9d548965-maas-api-xks-ca\") pod \"maas-api-revoke-sub-210e6572-lxcdd\" (UID: \"dce274d0-64ca-459a-995e-651c9d548965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-210e6572-lxcdd"
Nov 01 03:28:11.793593 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.791386    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trjx6\" (UniqueName: \"kubernetes.io/projected/dce274d0-64ca-459a-995e-651c9d548965-kube-api-access-trjx6\") pod \"maas-api-revoke-sub-210e6572-lxcdd\" (UID: \"dce274d0-64ca-459a-995e-651c9d548965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-210e6572-lxcdd"
Nov 01 03:28:11.840018 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.839985    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz"
Nov 01 03:28:11.893182 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.893143    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dce274d0-64ca-459a-995e-651c9d548965-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-210e6572-lxcdd\" (UID: \"dce274d0-64ca-459a-995e-651c9d548965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-210e6572-lxcdd"
Nov 01 03:28:11.893326 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.893231    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dce274d0-64ca-459a-995e-651c9d548965-maas-api-xks-ca\") pod \"maas-api-revoke-sub-210e6572-lxcdd\" (UID: \"dce274d0-64ca-459a-995e-651c9d548965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-210e6572-lxcdd"
Nov 01 03:28:11.893326 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.893288    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trjx6\" (UniqueName: \"kubernetes.io/projected/dce274d0-64ca-459a-995e-651c9d548965-kube-api-access-trjx6\") pod \"maas-api-revoke-sub-210e6572-lxcdd\" (UID: \"dce274d0-64ca-459a-995e-651c9d548965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-210e6572-lxcdd"
Nov 01 03:28:11.894021 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.893996    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dce274d0-64ca-459a-995e-651c9d548965-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-210e6572-lxcdd\" (UID: \"dce274d0-64ca-459a-995e-651c9d548965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-210e6572-lxcdd"
Nov 01 03:28:11.895595 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.895575    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dce274d0-64ca-459a-995e-651c9d548965-maas-api-xks-ca\") pod \"maas-api-revoke-sub-210e6572-lxcdd\" (UID: \"dce274d0-64ca-459a-995e-651c9d548965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-210e6572-lxcdd"
Nov 01 03:28:11.957432 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:11.957395    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trjx6\" (UniqueName: \"kubernetes.io/projected/dce274d0-64ca-459a-995e-651c9d548965-kube-api-access-trjx6\") pod \"maas-api-revoke-sub-210e6572-lxcdd\" (UID: \"dce274d0-64ca-459a-995e-651c9d548965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-210e6572-lxcdd"
Nov 01 03:28:12.042071 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:12.042004    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-210e6572-lxcdd"
Nov 01 03:28:12.279205 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:28:12.279171    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27bad768_f9df_4649_a979_2a459327f07d.slice/crio-463ea2ab4b419a8f31d8c067d7bb7498b579fd9aea88334b74bab09c59bc9401 WatchSource:0}: Error finding container 463ea2ab4b419a8f31d8c067d7bb7498b579fd9aea88334b74bab09c59bc9401: Status 404 returned error can't find the container with id 463ea2ab4b419a8f31d8c067d7bb7498b579fd9aea88334b74bab09c59bc9401
Nov 01 03:28:12.281005 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:12.280926    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz"]
Nov 01 03:28:12.281961 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:12.281930    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:28:12.282042 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:12.281990    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:28:12.282042 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:12.282019    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:28:12.393028 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:12.392989    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz" event={"ID":"27bad768-f9df-4649-a979-2a459327f07d","Type":"ContainerStarted","Data":"1b6710396ea1c3a864a6e4972e728fe7c165c385b57dfe70cc5e911fa7144143"}
Nov 01 03:28:12.393513 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:12.393064    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz" event={"ID":"27bad768-f9df-4649-a979-2a459327f07d","Type":"ContainerStarted","Data":"463ea2ab4b419a8f31d8c067d7bb7498b579fd9aea88334b74bab09c59bc9401"}
Nov 01 03:28:12.395180 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:12.395152    4930 generic.go:356] "Generic (PLEG): container finished" podID="6bfb43fb-7e73-4be2-80f2-e4361bfa096a" containerID="d24c1319b99a481b37ea62b6c11af07f77c38deea31ee734b3acc3bfef240f0b" exitCode=0
Nov 01 03:28:12.395328 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:12.395229    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f22c0d-827rx" event={"ID":"6bfb43fb-7e73-4be2-80f2-e4361bfa096a","Type":"ContainerDied","Data":"d24c1319b99a481b37ea62b6c11af07f77c38deea31ee734b3acc3bfef240f0b"}
Nov 01 03:28:12.397209 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:12.397185    4930 generic.go:356] "Generic (PLEG): container finished" podID="dc079b72-10cb-4437-afa0-74902f4ef55c" containerID="f535b409ee6a3d8ad78701a400ab7dd32ed34fd46dd733a41e3891a71990d5fd" exitCode=0
Nov 01 03:28:12.397323 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:12.397251    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-184a059e-jv2ln" event={"ID":"dc079b72-10cb-4437-afa0-74902f4ef55c","Type":"ContainerDied","Data":"f535b409ee6a3d8ad78701a400ab7dd32ed34fd46dd733a41e3891a71990d5fd"}
Nov 01 03:28:12.431905 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:12.431870    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-210e6572-lxcdd"]
Nov 01 03:28:12.521265 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:12.521192    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz" podStartSLOduration=1.5211728340000001 podStartE2EDuration="1.521172834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:28:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:28:12.518862446 +0000 UTC m=+1297.041869054" watchObservedRunningTime="2026-11-01 03:28:12.521172834 +0000 UTC m=+1297.044179424"
Nov 01 03:28:12.539872 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:12.531841    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:12.580108 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:12.580081    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-305021d0-6nlmc"
Nov 01 03:28:12.706961 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:12.706928    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2242792-3f71-4878-9b72-4f2bac2c3f76-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2242792-3f71-4878-9b72-4f2bac2c3f76-maas-api-xks-ca\") pod \"d2242792-3f71-4878-9b72-4f2bac2c3f76\" (UID: \"d2242792-3f71-4878-9b72-4f2bac2c3f76\") "
Nov 01 03:28:12.707143 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:12.707104    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2242792-3f71-4878-9b72-4f2bac2c3f76-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2242792-3f71-4878-9b72-4f2bac2c3f76-maas-api-openshift-ca\") pod \"d2242792-3f71-4878-9b72-4f2bac2c3f76\" (UID: \"d2242792-3f71-4878-9b72-4f2bac2c3f76\") "
Nov 01 03:28:12.707298 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:12.707279    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2242792-3f71-4878-9b72-4f2bac2c3f76-kube-api-access-488br\" (UniqueName: \"kubernetes.io/projected/d2242792-3f71-4878-9b72-4f2bac2c3f76-kube-api-access-488br\") pod \"d2242792-3f71-4878-9b72-4f2bac2c3f76\" (UID: \"d2242792-3f71-4878-9b72-4f2bac2c3f76\") "
Nov 01 03:28:12.707636 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:12.707490    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2242792-3f71-4878-9b72-4f2bac2c3f76-maas-api-openshift-ca" pod "d2242792-3f71-4878-9b72-4f2bac2c3f76" (UID: "d2242792-3f71-4878-9b72-4f2bac2c3f76"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:28:12.708496 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:12.708176    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2242792-3f71-4878-9b72-4f2bac2c3f76-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:12.709503 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:12.709477    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2242792-3f71-4878-9b72-4f2bac2c3f76-maas-api-xks-ca" pod "d2242792-3f71-4878-9b72-4f2bac2c3f76" (UID: "d2242792-3f71-4878-9b72-4f2bac2c3f76"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:28:12.709915 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:12.709890    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2242792-3f71-4878-9b72-4f2bac2c3f76-kube-api-access-488br" pod "d2242792-3f71-4878-9b72-4f2bac2c3f76" (UID: "d2242792-3f71-4878-9b72-4f2bac2c3f76"). InnerVolumeSpecName "kube-api-access-488br". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:28:12.809688 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:12.809647    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-488br\" (UniqueName: \"kubernetes.io/projected/d2242792-3f71-4878-9b72-4f2bac2c3f76-kube-api-access-488br\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:12.809688 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:12.809684    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2242792-3f71-4878-9b72-4f2bac2c3f76-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:12.840733 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:12.840689    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz"
Nov 01 03:28:13.407323 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:13.407283    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-210e6572-lxcdd" event={"ID":"dce274d0-64ca-459a-995e-651c9d548965","Type":"ContainerStarted","Data":"57f461a3d368806b07d30d1874fce81388c3d0c267e67df0f8b44dd42a15e219"}
Nov 01 03:28:13.407747 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:13.407348    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-210e6572-lxcdd" event={"ID":"dce274d0-64ca-459a-995e-651c9d548965","Type":"ContainerStarted","Data":"89c407090eeb5e068eeff4e125a8473ede1ff278fabeb393e207800b20f89b67"}
Nov 01 03:28:13.408969 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:13.408945    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-305021d0-6nlmc"
Nov 01 03:28:13.409077 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:13.408961    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-305021d0-6nlmc" event={"ID":"d2242792-3f71-4878-9b72-4f2bac2c3f76","Type":"ContainerDied","Data":"dea04c3d7c605a00880b00fc56a8e1f23ad837eee9be84ad0e8b027957711b3c"}
Nov 01 03:28:13.409077 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:13.409011    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dea04c3d7c605a00880b00fc56a8e1f23ad837eee9be84ad0e8b027957711b3c"
Nov 01 03:28:13.529906 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:13.528410    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-210e6572-lxcdd" podStartSLOduration=2.528389361 podStartE2EDuration="2.528389361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:28:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:28:13.527321578 +0000 UTC m=+1298.050328167" watchObservedRunningTime="2026-11-01 03:28:13.528389361 +0000 UTC m=+1298.051395951"
Nov 01 03:28:13.643110 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:13.643084    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-184a059e-jv2ln"
Nov 01 03:28:13.716057 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:13.716032    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f22c0d-827rx"
Nov 01 03:28:13.727058 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:13.727034    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc079b72-10cb-4437-afa0-74902f4ef55c-kube-api-access-64jzx\" (UniqueName: \"kubernetes.io/projected/dc079b72-10cb-4437-afa0-74902f4ef55c-kube-api-access-64jzx\") pod \"dc079b72-10cb-4437-afa0-74902f4ef55c\" (UID: \"dc079b72-10cb-4437-afa0-74902f4ef55c\") "
Nov 01 03:28:13.727191 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:13.727176    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc079b72-10cb-4437-afa0-74902f4ef55c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc079b72-10cb-4437-afa0-74902f4ef55c-maas-api-xks-ca\") pod \"dc079b72-10cb-4437-afa0-74902f4ef55c\" (UID: \"dc079b72-10cb-4437-afa0-74902f4ef55c\") "
Nov 01 03:28:13.727338 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:13.727327    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc079b72-10cb-4437-afa0-74902f4ef55c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc079b72-10cb-4437-afa0-74902f4ef55c-maas-api-openshift-ca\") pod \"dc079b72-10cb-4437-afa0-74902f4ef55c\" (UID: \"dc079b72-10cb-4437-afa0-74902f4ef55c\") "
Nov 01 03:28:13.727676 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:13.727650    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc079b72-10cb-4437-afa0-74902f4ef55c-maas-api-openshift-ca" pod "dc079b72-10cb-4437-afa0-74902f4ef55c" (UID: "dc079b72-10cb-4437-afa0-74902f4ef55c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:28:13.727822 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:13.727809    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc079b72-10cb-4437-afa0-74902f4ef55c-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:13.729516 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:13.729490    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc079b72-10cb-4437-afa0-74902f4ef55c-maas-api-xks-ca" pod "dc079b72-10cb-4437-afa0-74902f4ef55c" (UID: "dc079b72-10cb-4437-afa0-74902f4ef55c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:28:13.729715 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:13.729648    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc079b72-10cb-4437-afa0-74902f4ef55c-kube-api-access-64jzx" pod "dc079b72-10cb-4437-afa0-74902f4ef55c" (UID: "dc079b72-10cb-4437-afa0-74902f4ef55c"). InnerVolumeSpecName "kube-api-access-64jzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:28:13.802858 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:13.802665    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:13.828549 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:13.828514    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bfb43fb-7e73-4be2-80f2-e4361bfa096a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bfb43fb-7e73-4be2-80f2-e4361bfa096a-maas-api-openshift-ca\") pod \"6bfb43fb-7e73-4be2-80f2-e4361bfa096a\" (UID: \"6bfb43fb-7e73-4be2-80f2-e4361bfa096a\") "
Nov 01 03:28:13.828801 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:13.828674    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bfb43fb-7e73-4be2-80f2-e4361bfa096a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bfb43fb-7e73-4be2-80f2-e4361bfa096a-maas-api-xks-ca\") pod \"6bfb43fb-7e73-4be2-80f2-e4361bfa096a\" (UID: \"6bfb43fb-7e73-4be2-80f2-e4361bfa096a\") "
Nov 01 03:28:13.828801 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:13.828698    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bfb43fb-7e73-4be2-80f2-e4361bfa096a-kube-api-access-r2btq\" (UniqueName: \"kubernetes.io/projected/6bfb43fb-7e73-4be2-80f2-e4361bfa096a-kube-api-access-r2btq\") pod \"6bfb43fb-7e73-4be2-80f2-e4361bfa096a\" (UID: \"6bfb43fb-7e73-4be2-80f2-e4361bfa096a\") "
Nov 01 03:28:13.829188 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:13.829168    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64jzx\" (UniqueName: \"kubernetes.io/projected/dc079b72-10cb-4437-afa0-74902f4ef55c-kube-api-access-64jzx\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:13.829188 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:13.829187    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc079b72-10cb-4437-afa0-74902f4ef55c-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:13.829821 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:13.829797    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bfb43fb-7e73-4be2-80f2-e4361bfa096a-maas-api-openshift-ca" pod "6bfb43fb-7e73-4be2-80f2-e4361bfa096a" (UID: "6bfb43fb-7e73-4be2-80f2-e4361bfa096a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:28:13.832269 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:13.832239    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bfb43fb-7e73-4be2-80f2-e4361bfa096a-maas-api-xks-ca" pod "6bfb43fb-7e73-4be2-80f2-e4361bfa096a" (UID: "6bfb43fb-7e73-4be2-80f2-e4361bfa096a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:28:13.832359 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:13.832245    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bfb43fb-7e73-4be2-80f2-e4361bfa096a-kube-api-access-r2btq" pod "6bfb43fb-7e73-4be2-80f2-e4361bfa096a" (UID: "6bfb43fb-7e73-4be2-80f2-e4361bfa096a"). InnerVolumeSpecName "kube-api-access-r2btq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:28:13.841124 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:13.841086    4930 patch_prober.go:29] interesting pod/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 01 03:28:13.841220 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:13.841174    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz" podUID="27bad768-f9df-4649-a979-2a459327f07d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 01 03:28:13.931041 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:13.931006    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bfb43fb-7e73-4be2-80f2-e4361bfa096a-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:13.931041 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:13.931037    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bfb43fb-7e73-4be2-80f2-e4361bfa096a-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:13.931041 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:13.931046    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2btq\" (UniqueName: \"kubernetes.io/projected/6bfb43fb-7e73-4be2-80f2-e4361bfa096a-kube-api-access-r2btq\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:14.419901 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:14.419865    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f22c0d-827rx" event={"ID":"6bfb43fb-7e73-4be2-80f2-e4361bfa096a","Type":"ContainerDied","Data":"9beead8a3dc3cfdcbc5455d2aa3ad9ccab0e636e06b43360aee58d1d200d3365"}
Nov 01 03:28:14.420328 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:14.419926    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9beead8a3dc3cfdcbc5455d2aa3ad9ccab0e636e06b43360aee58d1d200d3365"
Nov 01 03:28:14.420328 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:14.419877    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8f22c0d-827rx"
Nov 01 03:28:14.421547 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:14.421516    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-184a059e-jv2ln"
Nov 01 03:28:14.421547 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:14.421539    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-184a059e-jv2ln" event={"ID":"dc079b72-10cb-4437-afa0-74902f4ef55c","Type":"ContainerDied","Data":"0ef445132fbfbb0ca11ed5a73d5be220e66aac2783119b55ed6a1ec8c8bda494"}
Nov 01 03:28:14.421759 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:14.421593    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ef445132fbfbb0ca11ed5a73d5be220e66aac2783119b55ed6a1ec8c8bda494"
Nov 01 03:28:14.423396 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:14.423368    4930 generic.go:356] "Generic (PLEG): container finished" podID="dce274d0-64ca-459a-995e-651c9d548965" containerID="57f461a3d368806b07d30d1874fce81388c3d0c267e67df0f8b44dd42a15e219" exitCode=0
Nov 01 03:28:14.423515 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:14.423444    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-210e6572-lxcdd" event={"ID":"dce274d0-64ca-459a-995e-651c9d548965","Type":"ContainerDied","Data":"57f461a3d368806b07d30d1874fce81388c3d0c267e67df0f8b44dd42a15e219"}
Nov 01 03:28:14.840528 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:14.840498    4930 patch_prober.go:29] interesting pod/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 01 03:28:14.840759 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:14.840544    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz" podUID="27bad768-f9df-4649-a979-2a459327f07d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 01 03:28:15.051172 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:15.051129    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-034045af-xq7ns"]
Nov 01 03:28:15.056095 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:15.056067    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-034045af-xq7ns"]
Nov 01 03:28:15.643434 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:15.643402    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8805a580-8cc7-483b-b266-d6db7193c90f" path="/var/lib/kubelet/pods/8805a580-8cc7-483b-b266-d6db7193c90f/volumes"
Nov 01 03:28:15.692367 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:15.692295    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-210e6572-lxcdd"
Nov 01 03:28:15.757827 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:15.757791    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dce274d0-64ca-459a-995e-651c9d548965-kube-api-access-trjx6\" (UniqueName: \"kubernetes.io/projected/dce274d0-64ca-459a-995e-651c9d548965-kube-api-access-trjx6\") pod \"dce274d0-64ca-459a-995e-651c9d548965\" (UID: \"dce274d0-64ca-459a-995e-651c9d548965\") "
Nov 01 03:28:15.758215 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:15.758185    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dce274d0-64ca-459a-995e-651c9d548965-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dce274d0-64ca-459a-995e-651c9d548965-maas-api-openshift-ca\") pod \"dce274d0-64ca-459a-995e-651c9d548965\" (UID: \"dce274d0-64ca-459a-995e-651c9d548965\") "
Nov 01 03:28:15.758414 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:15.758397    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dce274d0-64ca-459a-995e-651c9d548965-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dce274d0-64ca-459a-995e-651c9d548965-maas-api-xks-ca\") pod \"dce274d0-64ca-459a-995e-651c9d548965\" (UID: \"dce274d0-64ca-459a-995e-651c9d548965\") "
Nov 01 03:28:15.778578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:15.772292    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dce274d0-64ca-459a-995e-651c9d548965-maas-api-openshift-ca" pod "dce274d0-64ca-459a-995e-651c9d548965" (UID: "dce274d0-64ca-459a-995e-651c9d548965"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:28:15.794636 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:15.792603    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce274d0-64ca-459a-995e-651c9d548965-kube-api-access-trjx6" pod "dce274d0-64ca-459a-995e-651c9d548965" (UID: "dce274d0-64ca-459a-995e-651c9d548965"). InnerVolumeSpecName "kube-api-access-trjx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:28:15.794636 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:15.794466    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dce274d0-64ca-459a-995e-651c9d548965-maas-api-xks-ca" pod "dce274d0-64ca-459a-995e-651c9d548965" (UID: "dce274d0-64ca-459a-995e-651c9d548965"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:28:15.841238 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:15.841206    4930 patch_prober.go:29] interesting pod/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 01 03:28:15.841380 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:15.841263    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz" podUID="27bad768-f9df-4649-a979-2a459327f07d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 01 03:28:15.866578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:15.865893    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dce274d0-64ca-459a-995e-651c9d548965-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:15.866578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:15.865931    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dce274d0-64ca-459a-995e-651c9d548965-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:15.866578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:15.865947    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trjx6\" (UniqueName: \"kubernetes.io/projected/dce274d0-64ca-459a-995e-651c9d548965-kube-api-access-trjx6\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:16.444420 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:16.444389    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-210e6572-lxcdd" event={"ID":"dce274d0-64ca-459a-995e-651c9d548965","Type":"ContainerDied","Data":"89c407090eeb5e068eeff4e125a8473ede1ff278fabeb393e207800b20f89b67"}
Nov 01 03:28:16.444420 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:16.444408    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-210e6572-lxcdd"
Nov 01 03:28:16.444420 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:16.444437    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89c407090eeb5e068eeff4e125a8473ede1ff278fabeb393e207800b20f89b67"
Nov 01 03:28:16.841096 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:16.841065    4930 patch_prober.go:29] interesting pod/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 01 03:28:16.841526 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:16.841127    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz" podUID="27bad768-f9df-4649-a979-2a459327f07d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 01 03:28:17.051090 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:17.051056    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1a9ce86-9jznn"]
Nov 01 03:28:17.055713 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:17.055683    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1a9ce86-9jznn"]
Nov 01 03:28:17.622840 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:17.622805    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eefc364f-9d31-456c-b95c-0fcebe66bfc5" path="/var/lib/kubelet/pods/eefc364f-9d31-456c-b95c-0fcebe66bfc5/volumes"
Nov 01 03:28:17.841187 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:17.841152    4930 patch_prober.go:29] interesting pod/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 01 03:28:17.841614 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:17.841207    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz" podUID="27bad768-f9df-4649-a979-2a459327f07d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 01 03:28:18.745434 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:18.745400    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:18.841037 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:18.841004    4930 patch_prober.go:29] interesting pod/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 01 03:28:18.841211 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:18.841069    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz" podUID="27bad768-f9df-4649-a979-2a459327f07d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 01 03:28:18.902713 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:18.902676    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53513363-zgf2q"]
Nov 01 03:28:18.903680 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:18.903659    4930 cpu_manager.go:414] "containerMap: removing container" podUID="dce274d0-64ca-459a-995e-651c9d548965" containerName="revoke-keys"
Nov 01 03:28:18.903739 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:18.903685    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dce274d0-64ca-459a-995e-651c9d548965" containerName="revoke-keys"
Nov 01 03:28:18.903739 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:18.903719    4930 cpu_manager.go:414] "containerMap: removing container" podUID="dc079b72-10cb-4437-afa0-74902f4ef55c" containerName="revoke-keys"
Nov 01 03:28:18.903739 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:18.903728    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc079b72-10cb-4437-afa0-74902f4ef55c" containerName="revoke-keys"
Nov 01 03:28:18.903857 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:18.903741    4930 cpu_manager.go:414] "containerMap: removing container" podUID="6bfb43fb-7e73-4be2-80f2-e4361bfa096a" containerName="revoke-keys"
Nov 01 03:28:18.903857 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:18.903749    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bfb43fb-7e73-4be2-80f2-e4361bfa096a" containerName="revoke-keys"
Nov 01 03:28:18.903857 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:18.903798    4930 cpu_manager.go:414] "containerMap: removing container" podUID="d2242792-3f71-4878-9b72-4f2bac2c3f76" containerName="revoke-keys"
Nov 01 03:28:18.903857 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:18.903807    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2242792-3f71-4878-9b72-4f2bac2c3f76" containerName="revoke-keys"
Nov 01 03:28:18.903970 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:18.903952    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc079b72-10cb-4437-afa0-74902f4ef55c" containerName="revoke-keys"
Nov 01 03:28:18.904000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:18.903968    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bfb43fb-7e73-4be2-80f2-e4361bfa096a" containerName="revoke-keys"
Nov 01 03:28:18.904000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:18.903989    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2242792-3f71-4878-9b72-4f2bac2c3f76" containerName="revoke-keys"
Nov 01 03:28:18.904064 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:18.904005    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="dce274d0-64ca-459a-995e-651c9d548965" containerName="revoke-keys"
Nov 01 03:28:18.909618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:18.909594    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53513363-zgf2q"
Nov 01 03:28:18.966411 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:18.966376    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53513363-zgf2q"]
Nov 01 03:28:19.005048 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:19.004964    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:28:19.007330 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:19.007306    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:28:19.007480 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:19.007384    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:28:19.007534 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:19.007492    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:28:19.031087 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:19.031057    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8l6k\" (UniqueName: \"kubernetes.io/projected/c8c55607-460a-4562-b07e-1f0274d9f917-kube-api-access-t8l6k\") pod \"maas-api-revoke-sub-53513363-zgf2q\" (UID: \"c8c55607-460a-4562-b07e-1f0274d9f917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53513363-zgf2q"
Nov 01 03:28:19.031184 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:19.031112    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8c55607-460a-4562-b07e-1f0274d9f917-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53513363-zgf2q\" (UID: \"c8c55607-460a-4562-b07e-1f0274d9f917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53513363-zgf2q"
Nov 01 03:28:19.031316 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:19.031293    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8c55607-460a-4562-b07e-1f0274d9f917-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53513363-zgf2q\" (UID: \"c8c55607-460a-4562-b07e-1f0274d9f917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53513363-zgf2q"
Nov 01 03:28:19.128524 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:28:19.128492    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:28:19.132440 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:19.132416    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8c55607-460a-4562-b07e-1f0274d9f917-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53513363-zgf2q\" (UID: \"c8c55607-460a-4562-b07e-1f0274d9f917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53513363-zgf2q"
Nov 01 03:28:19.132633 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:19.132615    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8l6k\" (UniqueName: \"kubernetes.io/projected/c8c55607-460a-4562-b07e-1f0274d9f917-kube-api-access-t8l6k\") pod \"maas-api-revoke-sub-53513363-zgf2q\" (UID: \"c8c55607-460a-4562-b07e-1f0274d9f917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53513363-zgf2q"
Nov 01 03:28:19.132681 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:19.132662    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8c55607-460a-4562-b07e-1f0274d9f917-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53513363-zgf2q\" (UID: \"c8c55607-460a-4562-b07e-1f0274d9f917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53513363-zgf2q"
Nov 01 03:28:19.133229 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:19.133207    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8c55607-460a-4562-b07e-1f0274d9f917-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53513363-zgf2q\" (UID: \"c8c55607-460a-4562-b07e-1f0274d9f917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53513363-zgf2q"
Nov 01 03:28:19.134939 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:19.134918    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8c55607-460a-4562-b07e-1f0274d9f917-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53513363-zgf2q\" (UID: \"c8c55607-460a-4562-b07e-1f0274d9f917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53513363-zgf2q"
Nov 01 03:28:19.193936 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:19.193904    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8l6k\" (UniqueName: \"kubernetes.io/projected/c8c55607-460a-4562-b07e-1f0274d9f917-kube-api-access-t8l6k\") pod \"maas-api-revoke-sub-53513363-zgf2q\" (UID: \"c8c55607-460a-4562-b07e-1f0274d9f917\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53513363-zgf2q"
Nov 01 03:28:19.223684 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:19.223657    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53513363-zgf2q"
Nov 01 03:28:19.636239 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:19.636206    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53513363-zgf2q"]
Nov 01 03:28:19.636859 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:28:19.636835    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8c55607_460a_4562_b07e_1f0274d9f917.slice/crio-3ccca2abc5a0c99cf5146ababcfc8bac54ecf25c3897ee3c0b7bb6774e3678ce WatchSource:0}: Error finding container 3ccca2abc5a0c99cf5146ababcfc8bac54ecf25c3897ee3c0b7bb6774e3678ce: Status 404 returned error can't find the container with id 3ccca2abc5a0c99cf5146ababcfc8bac54ecf25c3897ee3c0b7bb6774e3678ce
Nov 01 03:28:19.872155 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:19.872104    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz"
Nov 01 03:28:19.873788 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:19.873729    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz"
Nov 01 03:28:19.874550 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:19.874532    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz"
Nov 01 03:28:20.499796 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:20.499756    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53513363-zgf2q" event={"ID":"c8c55607-460a-4562-b07e-1f0274d9f917","Type":"ContainerStarted","Data":"10b1639cadf8c0b2abc646e911d3038a647213e55265c226e86496ade4a58d52"}
Nov 01 03:28:20.500003 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:20.499822    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53513363-zgf2q" event={"ID":"c8c55607-460a-4562-b07e-1f0274d9f917","Type":"ContainerStarted","Data":"3ccca2abc5a0c99cf5146ababcfc8bac54ecf25c3897ee3c0b7bb6774e3678ce"}
Nov 01 03:28:20.616877 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:20.616828    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53513363-zgf2q" podStartSLOduration=2.616802573 podStartE2EDuration="2.616802573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:28:20.61469075 +0000 UTC m=+1305.137697355" watchObservedRunningTime="2026-11-01 03:28:20.616802573 +0000 UTC m=+1305.139809161"
Nov 01 03:28:21.162970 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:21.162935    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-518af558-8p2xt"]
Nov 01 03:28:21.168529 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:21.168504    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-518af558-8p2xt"
Nov 01 03:28:21.226666 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:21.226634    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-518af558-8p2xt"]
Nov 01 03:28:21.286488 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:21.286449    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc5cffef-d29f-4ade-b650-f8904b356bcd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-518af558-8p2xt\" (UID: \"fc5cffef-d29f-4ade-b650-f8904b356bcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518af558-8p2xt"
Nov 01 03:28:21.286706 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:21.286650    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khdwc\" (UniqueName: \"kubernetes.io/projected/fc5cffef-d29f-4ade-b650-f8904b356bcd-kube-api-access-khdwc\") pod \"maas-api-revoke-sub-518af558-8p2xt\" (UID: \"fc5cffef-d29f-4ade-b650-f8904b356bcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518af558-8p2xt"
Nov 01 03:28:21.286777 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:21.286731    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc5cffef-d29f-4ade-b650-f8904b356bcd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-518af558-8p2xt\" (UID: \"fc5cffef-d29f-4ade-b650-f8904b356bcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518af558-8p2xt"
Nov 01 03:28:21.389000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:21.388958    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc5cffef-d29f-4ade-b650-f8904b356bcd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-518af558-8p2xt\" (UID: \"fc5cffef-d29f-4ade-b650-f8904b356bcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518af558-8p2xt"
Nov 01 03:28:21.389187 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:21.389154    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khdwc\" (UniqueName: \"kubernetes.io/projected/fc5cffef-d29f-4ade-b650-f8904b356bcd-kube-api-access-khdwc\") pod \"maas-api-revoke-sub-518af558-8p2xt\" (UID: \"fc5cffef-d29f-4ade-b650-f8904b356bcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518af558-8p2xt"
Nov 01 03:28:21.389266 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:21.389234    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc5cffef-d29f-4ade-b650-f8904b356bcd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-518af558-8p2xt\" (UID: \"fc5cffef-d29f-4ade-b650-f8904b356bcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518af558-8p2xt"
Nov 01 03:28:21.390279 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:21.390250    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc5cffef-d29f-4ade-b650-f8904b356bcd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-518af558-8p2xt\" (UID: \"fc5cffef-d29f-4ade-b650-f8904b356bcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518af558-8p2xt"
Nov 01 03:28:21.392223 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:21.392169    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc5cffef-d29f-4ade-b650-f8904b356bcd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-518af558-8p2xt\" (UID: \"fc5cffef-d29f-4ade-b650-f8904b356bcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518af558-8p2xt"
Nov 01 03:28:21.450968 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:21.450905    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khdwc\" (UniqueName: \"kubernetes.io/projected/fc5cffef-d29f-4ade-b650-f8904b356bcd-kube-api-access-khdwc\") pod \"maas-api-revoke-sub-518af558-8p2xt\" (UID: \"fc5cffef-d29f-4ade-b650-f8904b356bcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518af558-8p2xt"
Nov 01 03:28:21.494020 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:21.493738    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-518af558-8p2xt"
Nov 01 03:28:21.520479 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:21.520445    4930 generic.go:356] "Generic (PLEG): container finished" podID="c8c55607-460a-4562-b07e-1f0274d9f917" containerID="10b1639cadf8c0b2abc646e911d3038a647213e55265c226e86496ade4a58d52" exitCode=0
Nov 01 03:28:21.521866 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:21.521837    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53513363-zgf2q" event={"ID":"c8c55607-460a-4562-b07e-1f0274d9f917","Type":"ContainerDied","Data":"10b1639cadf8c0b2abc646e911d3038a647213e55265c226e86496ade4a58d52"}
Nov 01 03:28:21.892673 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:28:21.892638    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc5cffef_d29f_4ade_b650_f8904b356bcd.slice/crio-1126836cd19bf9e0914227ce4f9b402bc2d34602aa1dcdf1b0a4372cea594013 WatchSource:0}: Error finding container 1126836cd19bf9e0914227ce4f9b402bc2d34602aa1dcdf1b0a4372cea594013: Status 404 returned error can't find the container with id 1126836cd19bf9e0914227ce4f9b402bc2d34602aa1dcdf1b0a4372cea594013
Nov 01 03:28:21.893320 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:21.893296    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-518af558-8p2xt"]
Nov 01 03:28:22.531407 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:22.531304    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-518af558-8p2xt" event={"ID":"fc5cffef-d29f-4ade-b650-f8904b356bcd","Type":"ContainerStarted","Data":"3084384ef164cba47fcb5329747dafb6028a38b677778baf73d2537582b1a0d8"}
Nov 01 03:28:22.531932 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:22.531407    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-518af558-8p2xt" event={"ID":"fc5cffef-d29f-4ade-b650-f8904b356bcd","Type":"ContainerStarted","Data":"1126836cd19bf9e0914227ce4f9b402bc2d34602aa1dcdf1b0a4372cea594013"}
Nov 01 03:28:22.654129 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:22.654057    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-518af558-8p2xt" podStartSLOduration=1.65403904 podStartE2EDuration="1.65403904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:28:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:28:22.65117224 +0000 UTC m=+1307.174178853" watchObservedRunningTime="2026-11-01 03:28:22.65403904 +0000 UTC m=+1307.177045630"
Nov 01 03:28:22.713126 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:22.713094    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53513363-zgf2q"
Nov 01 03:28:22.815358 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:22.815273    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8c55607-460a-4562-b07e-1f0274d9f917-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8c55607-460a-4562-b07e-1f0274d9f917-maas-api-xks-ca\") pod \"c8c55607-460a-4562-b07e-1f0274d9f917\" (UID: \"c8c55607-460a-4562-b07e-1f0274d9f917\") "
Nov 01 03:28:22.815592 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:22.815530    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8c55607-460a-4562-b07e-1f0274d9f917-kube-api-access-t8l6k\" (UniqueName: \"kubernetes.io/projected/c8c55607-460a-4562-b07e-1f0274d9f917-kube-api-access-t8l6k\") pod \"c8c55607-460a-4562-b07e-1f0274d9f917\" (UID: \"c8c55607-460a-4562-b07e-1f0274d9f917\") "
Nov 01 03:28:22.815681 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:22.815642    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8c55607-460a-4562-b07e-1f0274d9f917-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8c55607-460a-4562-b07e-1f0274d9f917-maas-api-openshift-ca\") pod \"c8c55607-460a-4562-b07e-1f0274d9f917\" (UID: \"c8c55607-460a-4562-b07e-1f0274d9f917\") "
Nov 01 03:28:22.816010 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:22.815981    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8c55607-460a-4562-b07e-1f0274d9f917-maas-api-openshift-ca" pod "c8c55607-460a-4562-b07e-1f0274d9f917" (UID: "c8c55607-460a-4562-b07e-1f0274d9f917"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:28:22.816241 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:22.816221    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8c55607-460a-4562-b07e-1f0274d9f917-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:22.817296 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:22.817271    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8c55607-460a-4562-b07e-1f0274d9f917-maas-api-xks-ca" pod "c8c55607-460a-4562-b07e-1f0274d9f917" (UID: "c8c55607-460a-4562-b07e-1f0274d9f917"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:28:22.817628 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:22.817604    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8c55607-460a-4562-b07e-1f0274d9f917-kube-api-access-t8l6k" pod "c8c55607-460a-4562-b07e-1f0274d9f917" (UID: "c8c55607-460a-4562-b07e-1f0274d9f917"). InnerVolumeSpecName "kube-api-access-t8l6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:28:22.917418 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:22.917386    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8l6k\" (UniqueName: \"kubernetes.io/projected/c8c55607-460a-4562-b07e-1f0274d9f917-kube-api-access-t8l6k\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:22.917418 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:22.917418    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8c55607-460a-4562-b07e-1f0274d9f917-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:23.542785 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:23.542756    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53513363-zgf2q"
Nov 01 03:28:23.543234 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:23.542781    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53513363-zgf2q" event={"ID":"c8c55607-460a-4562-b07e-1f0274d9f917","Type":"ContainerDied","Data":"3ccca2abc5a0c99cf5146ababcfc8bac54ecf25c3897ee3c0b7bb6774e3678ce"}
Nov 01 03:28:23.543234 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:23.542839    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ccca2abc5a0c99cf5146ababcfc8bac54ecf25c3897ee3c0b7bb6774e3678ce"
Nov 01 03:28:23.544762 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:23.544739    4930 generic.go:356] "Generic (PLEG): container finished" podID="fc5cffef-d29f-4ade-b650-f8904b356bcd" containerID="3084384ef164cba47fcb5329747dafb6028a38b677778baf73d2537582b1a0d8" exitCode=0
Nov 01 03:28:23.544897 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:23.544825    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-518af558-8p2xt" event={"ID":"fc5cffef-d29f-4ade-b650-f8904b356bcd","Type":"ContainerDied","Data":"3084384ef164cba47fcb5329747dafb6028a38b677778baf73d2537582b1a0d8"}
Nov 01 03:28:23.564393 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:23.564363    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:24.419248 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:24.419213    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10863eb9-6zd52"]
Nov 01 03:28:24.419879 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:24.419864    4930 cpu_manager.go:414] "containerMap: removing container" podUID="c8c55607-460a-4562-b07e-1f0274d9f917" containerName="revoke-keys"
Nov 01 03:28:24.419934 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:24.419880    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8c55607-460a-4562-b07e-1f0274d9f917" containerName="revoke-keys"
Nov 01 03:28:24.420006 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:24.419997    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8c55607-460a-4562-b07e-1f0274d9f917" containerName="revoke-keys"
Nov 01 03:28:24.423665 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:24.423646    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10863eb9-6zd52"
Nov 01 03:28:24.482626 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:24.482579    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10863eb9-6zd52"]
Nov 01 03:28:24.544673 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:24.544641    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsgl4\" (UniqueName: \"kubernetes.io/projected/3934c25e-4360-44ca-a355-abb425ebed69-kube-api-access-tsgl4\") pod \"maas-api-revoke-sub-10863eb9-6zd52\" (UID: \"3934c25e-4360-44ca-a355-abb425ebed69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10863eb9-6zd52"
Nov 01 03:28:24.545085 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:24.544698    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3934c25e-4360-44ca-a355-abb425ebed69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10863eb9-6zd52\" (UID: \"3934c25e-4360-44ca-a355-abb425ebed69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10863eb9-6zd52"
Nov 01 03:28:24.545085 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:24.544822    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3934c25e-4360-44ca-a355-abb425ebed69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10863eb9-6zd52\" (UID: \"3934c25e-4360-44ca-a355-abb425ebed69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10863eb9-6zd52"
Nov 01 03:28:24.646940 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:24.646904    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsgl4\" (UniqueName: \"kubernetes.io/projected/3934c25e-4360-44ca-a355-abb425ebed69-kube-api-access-tsgl4\") pod \"maas-api-revoke-sub-10863eb9-6zd52\" (UID: \"3934c25e-4360-44ca-a355-abb425ebed69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10863eb9-6zd52"
Nov 01 03:28:24.647168 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:24.646979    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3934c25e-4360-44ca-a355-abb425ebed69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10863eb9-6zd52\" (UID: \"3934c25e-4360-44ca-a355-abb425ebed69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10863eb9-6zd52"
Nov 01 03:28:24.647168 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:24.647050    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3934c25e-4360-44ca-a355-abb425ebed69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10863eb9-6zd52\" (UID: \"3934c25e-4360-44ca-a355-abb425ebed69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10863eb9-6zd52"
Nov 01 03:28:24.647934 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:24.647905    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3934c25e-4360-44ca-a355-abb425ebed69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10863eb9-6zd52\" (UID: \"3934c25e-4360-44ca-a355-abb425ebed69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10863eb9-6zd52"
Nov 01 03:28:24.650587 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:24.650520    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3934c25e-4360-44ca-a355-abb425ebed69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10863eb9-6zd52\" (UID: \"3934c25e-4360-44ca-a355-abb425ebed69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10863eb9-6zd52"
Nov 01 03:28:24.708185 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:24.708158    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsgl4\" (UniqueName: \"kubernetes.io/projected/3934c25e-4360-44ca-a355-abb425ebed69-kube-api-access-tsgl4\") pod \"maas-api-revoke-sub-10863eb9-6zd52\" (UID: \"3934c25e-4360-44ca-a355-abb425ebed69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10863eb9-6zd52"
Nov 01 03:28:24.743579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:24.738030    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10863eb9-6zd52"
Nov 01 03:28:24.768665 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:24.751756    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-518af558-8p2xt"
Nov 01 03:28:24.902583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:24.890350    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc5cffef-d29f-4ade-b650-f8904b356bcd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc5cffef-d29f-4ade-b650-f8904b356bcd-maas-api-xks-ca\") pod \"fc5cffef-d29f-4ade-b650-f8904b356bcd\" (UID: \"fc5cffef-d29f-4ade-b650-f8904b356bcd\") "
Nov 01 03:28:24.902583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:24.890658    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc5cffef-d29f-4ade-b650-f8904b356bcd-kube-api-access-khdwc\" (UniqueName: \"kubernetes.io/projected/fc5cffef-d29f-4ade-b650-f8904b356bcd-kube-api-access-khdwc\") pod \"fc5cffef-d29f-4ade-b650-f8904b356bcd\" (UID: \"fc5cffef-d29f-4ade-b650-f8904b356bcd\") "
Nov 01 03:28:24.902583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:24.890797    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc5cffef-d29f-4ade-b650-f8904b356bcd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc5cffef-d29f-4ade-b650-f8904b356bcd-maas-api-openshift-ca\") pod \"fc5cffef-d29f-4ade-b650-f8904b356bcd\" (UID: \"fc5cffef-d29f-4ade-b650-f8904b356bcd\") "
Nov 01 03:28:24.902583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:24.892170    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc5cffef-d29f-4ade-b650-f8904b356bcd-maas-api-openshift-ca" pod "fc5cffef-d29f-4ade-b650-f8904b356bcd" (UID: "fc5cffef-d29f-4ade-b650-f8904b356bcd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:28:24.934859 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:24.934799    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc5cffef-d29f-4ade-b650-f8904b356bcd-kube-api-access-khdwc" pod "fc5cffef-d29f-4ade-b650-f8904b356bcd" (UID: "fc5cffef-d29f-4ade-b650-f8904b356bcd"). InnerVolumeSpecName "kube-api-access-khdwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:28:24.936648 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:24.936017    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc5cffef-d29f-4ade-b650-f8904b356bcd-maas-api-xks-ca" pod "fc5cffef-d29f-4ade-b650-f8904b356bcd" (UID: "fc5cffef-d29f-4ade-b650-f8904b356bcd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:28:25.036628 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:25.035250    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khdwc\" (UniqueName: \"kubernetes.io/projected/fc5cffef-d29f-4ade-b650-f8904b356bcd-kube-api-access-khdwc\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:25.036628 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:25.035291    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc5cffef-d29f-4ade-b650-f8904b356bcd-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:25.036628 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:25.035310    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc5cffef-d29f-4ade-b650-f8904b356bcd-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:25.515060 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:25.512902    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10863eb9-6zd52"]
Nov 01 03:28:25.948579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:25.941605    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-518af558-8p2xt" event={"ID":"fc5cffef-d29f-4ade-b650-f8904b356bcd","Type":"ContainerDied","Data":"1126836cd19bf9e0914227ce4f9b402bc2d34602aa1dcdf1b0a4372cea594013"}
Nov 01 03:28:25.948579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:25.941676    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1126836cd19bf9e0914227ce4f9b402bc2d34602aa1dcdf1b0a4372cea594013"
Nov 01 03:28:25.948579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:25.941769    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-518af558-8p2xt"
Nov 01 03:28:26.000580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:25.982229    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10863eb9-6zd52" event={"ID":"3934c25e-4360-44ca-a355-abb425ebed69","Type":"ContainerStarted","Data":"1feb0e3e6102475b4d0eb11f778a67799bc5448c8cf767889032b1287c3d1d74"}
Nov 01 03:28:26.732645 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:26.729181    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:26.889579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:26.889095    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:27.050646 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:27.046962    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10863eb9-6zd52" event={"ID":"3934c25e-4360-44ca-a355-abb425ebed69","Type":"ContainerStarted","Data":"34d3ff014710376aa82f5101f2ba09f3e036e33b36cd963a24d1af2ef1524e16"}
Nov 01 03:28:27.234604 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:27.212133    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm" podUID="75d142b3-7172-4b57-bc07-a2f404c4656e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": context deadline exceeded"
Nov 01 03:28:27.236370 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:27.234910    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps" podUID="d658772c-b072-4513-a639-d19a94c78525" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:27.236370 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:27.235305    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp" podUID="667240b0-64a7-45cd-84fe-9cd7e0d8a301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 01 03:28:27.236370 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:27.235853    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2" podUID="e2483a06-2f4b-4a1e-af4e-a63eb50d29bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:27.236370 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:27.236101    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" podUID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded"
Nov 01 03:28:27.622590 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:27.619642    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj" podUID="b0e32cb3-ac18-4c1f-819c-f1139a88bfea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 01 03:28:27.651591 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:27.648975    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8" podUID="88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:27.663230 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:27.662056    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm" podUID="b2637bc6-3828-4b75-9783-5415519403a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:27.667427 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:27.667393    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7" podUID="1ce834cc-d34d-46cc-a081-4d2cc261becb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:28.050959 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:28.050836    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8" podUID="88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": context deadline exceeded"
Nov 01 03:28:28.185580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:28.184532    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n" podUID="d2511772-66f9-40a0-9166-87c4345485c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:28.215725 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:28.215675    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm" podUID="75d142b3-7172-4b57-bc07-a2f404c4656e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": context deadline exceeded"
Nov 01 03:28:28.224592 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:28.216938    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:28:28.224592 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:28.219266    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf" podUID="40881383-61bb-4fc8-9c75-f8ffce4834c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:28.224592 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:28.219526    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw" podUID="b6d688b8-b911-4a6a-a842-2b0fb7a75e83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:28.359967 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:28.357668    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps" podUID="d658772c-b072-4513-a639-d19a94c78525" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 01 03:28:28.359967 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:28.357699    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2" podUID="e2483a06-2f4b-4a1e-af4e-a63eb50d29bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": context deadline exceeded"
Nov 01 03:28:28.359967 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:28.357933    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" podUID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded"
Nov 01 03:28:28.359967 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:28.358812    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:28:28.359967 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:28.358872    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"
Nov 01 03:28:28.359967 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:28.358912    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:28:28.447581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:28.360885    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t" podUID="13930c1e-2200-4412-b8d3-ae3e4126a851" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 01 03:28:28.447581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:28.361063    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp" podUID="667240b0-64a7-45cd-84fe-9cd7e0d8a301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 01 03:28:28.447581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:28.361246    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp"
Nov 01 03:28:28.447581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:28.371812    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps" podUID="d658772c-b072-4513-a639-d19a94c78525" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:28.549287 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:28.549238    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp" podUID="667240b0-64a7-45cd-84fe-9cd7e0d8a301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": context deadline exceeded"
Nov 01 03:28:28.738769 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:28.738658    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8" podUID="88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Nov 01 03:28:28.749602 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:28.747856    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:28:28.810353 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:28.810201    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm" podUID="b2637bc6-3828-4b75-9783-5415519403a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:28.828898 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:28.828854    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj" podUID="b0e32cb3-ac18-4c1f-819c-f1139a88bfea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 01 03:28:28.845216 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:28.837704    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"
Nov 01 03:28:28.845529 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:28.845508    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:28:28.927347 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:28.927297    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7" podUID="1ce834cc-d34d-46cc-a081-4d2cc261becb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Nov 01 03:28:28.937095 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:28.937046    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"
Nov 01 03:28:29.009981 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:28.939323    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7" podUID="1ce834cc-d34d-46cc-a081-4d2cc261becb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": context deadline exceeded"
Nov 01 03:28:29.107526 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:29.097948    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10863eb9-6zd52" podStartSLOduration=5.097925747 podStartE2EDuration="5.097925747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:28:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:28:27.203870676 +0000 UTC m=+1311.726877265" watchObservedRunningTime="2026-11-01 03:28:29.097925747 +0000 UTC m=+1313.620932336"
Nov 01 03:28:29.199730 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:29.199683    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n" podUID="d2511772-66f9-40a0-9166-87c4345485c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:29.200651 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:29.200626    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"
Nov 01 03:28:29.228581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:29.222741    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf" podUID="40881383-61bb-4fc8-9c75-f8ffce4834c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Nov 01 03:28:29.229274 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:29.229235    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm" podUID="75d142b3-7172-4b57-bc07-a2f404c4656e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:29.229889 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:29.229867    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:28:29.255827 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:29.254851    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw" podUID="b6d688b8-b911-4a6a-a842-2b0fb7a75e83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:29.271850 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:29.271735    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:28:29.380598 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:29.377893    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp" podUID="667240b0-64a7-45cd-84fe-9cd7e0d8a301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:29.380598 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:29.378313    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2" podUID="e2483a06-2f4b-4a1e-af4e-a63eb50d29bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": context deadline exceeded"
Nov 01 03:28:29.381254 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:29.381215    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t" podUID="13930c1e-2200-4412-b8d3-ae3e4126a851" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:29.385820 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:29.385787    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:28:29.523798 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:29.523686    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps" podUID="d658772c-b072-4513-a639-d19a94c78525" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 01 03:28:29.524095 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:29.524072    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" podUID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded"
Nov 01 03:28:29.829638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:29.804189    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8" podUID="88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:29.857612 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:29.845954    4930 generic.go:356] "Generic (PLEG): container finished" podID="3934c25e-4360-44ca-a355-abb425ebed69" containerID="34d3ff014710376aa82f5101f2ba09f3e036e33b36cd963a24d1af2ef1524e16" exitCode=0
Nov 01 03:28:29.882211 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:29.882156    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj" podUID="b0e32cb3-ac18-4c1f-819c-f1139a88bfea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:29.987588 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:29.962886    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7" podUID="1ce834cc-d34d-46cc-a081-4d2cc261becb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Nov 01 03:28:30.041689 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:30.015228    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm" podUID="b2637bc6-3828-4b75-9783-5415519403a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 01 03:28:30.094182 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:30.094138    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj" podUID="b0e32cb3-ac18-4c1f-819c-f1139a88bfea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:30.205712 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:30.205663    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n" podUID="d2511772-66f9-40a0-9166-87c4345485c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:30.237873 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:30.237818    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf" podUID="40881383-61bb-4fc8-9c75-f8ffce4834c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:30.366645 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:30.240100    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm" podUID="75d142b3-7172-4b57-bc07-a2f404c4656e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:30.507581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:30.240821    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10863eb9-6zd52" event={"ID":"3934c25e-4360-44ca-a355-abb425ebed69","Type":"ContainerDied","Data":"34d3ff014710376aa82f5101f2ba09f3e036e33b36cd963a24d1af2ef1524e16"}
Nov 01 03:28:30.537683 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:30.403649    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t" podUID="13930c1e-2200-4412-b8d3-ae3e4126a851" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:30.549580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:30.457710    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw" podUID="b6d688b8-b911-4a6a-a842-2b0fb7a75e83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:30.593578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:30.474452    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp" podUID="667240b0-64a7-45cd-84fe-9cd7e0d8a301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 01 03:28:30.593578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:30.477373    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2" podUID="e2483a06-2f4b-4a1e-af4e-a63eb50d29bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": context deadline exceeded"
Nov 01 03:28:30.593578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:30.540680    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps" podUID="d658772c-b072-4513-a639-d19a94c78525" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 01 03:28:30.731638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:30.728152    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" podUID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:30.840510 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:30.840425    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8" podUID="88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:30.898620 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:30.898542    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj" podUID="b0e32cb3-ac18-4c1f-819c-f1139a88bfea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:30.994752 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:30.994656    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7" podUID="1ce834cc-d34d-46cc-a081-4d2cc261becb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:31.072963 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:31.048127    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm" podUID="b2637bc6-3828-4b75-9783-5415519403a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 01 03:28:31.127576 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:31.127145    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n" podUID="d2511772-66f9-40a0-9166-87c4345485c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": context deadline exceeded"
Nov 01 03:28:31.128489 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:31.128394    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:31.252539 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:31.252427    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n" podUID="d2511772-66f9-40a0-9166-87c4345485c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 01 03:28:31.353362 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:31.353311    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf" podUID="40881383-61bb-4fc8-9c75-f8ffce4834c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Nov 01 03:28:31.506908 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:31.505187    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm" podUID="75d142b3-7172-4b57-bc07-a2f404c4656e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": context deadline exceeded"
Nov 01 03:28:31.538772 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:31.538712    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t" podUID="13930c1e-2200-4412-b8d3-ae3e4126a851" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:31.541062 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:31.541027    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2" podUID="e2483a06-2f4b-4a1e-af4e-a63eb50d29bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:31.636852 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:31.636808    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps" podUID="d658772c-b072-4513-a639-d19a94c78525" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 01 03:28:31.676131 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:31.646865    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2" podUID="e2483a06-2f4b-4a1e-af4e-a63eb50d29bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:31.676131 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:31.647161    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw" podUID="b6d688b8-b911-4a6a-a842-2b0fb7a75e83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 01 03:28:31.676131 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:31.647344    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp" podUID="667240b0-64a7-45cd-84fe-9cd7e0d8a301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 01 03:28:31.769722 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:31.769622    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" podUID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded"
Nov 01 03:28:31.856118 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:31.856048    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8" podUID="88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:31.899892 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:31.899844    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj" podUID="b0e32cb3-ac18-4c1f-819c-f1139a88bfea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:32.001298 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:32.001252    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7" podUID="1ce834cc-d34d-46cc-a081-4d2cc261becb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Nov 01 03:28:32.147931 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:32.147086    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm" podUID="b2637bc6-3828-4b75-9783-5415519403a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:32.280936 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:32.280891    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n" podUID="d2511772-66f9-40a0-9166-87c4345485c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:32.385187 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:32.385135    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf" podUID="40881383-61bb-4fc8-9c75-f8ffce4834c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Nov 01 03:28:32.956618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:32.938783    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj" podUID="b0e32cb3-ac18-4c1f-819c-f1139a88bfea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 01 03:28:32.956871 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:32.715837    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps" podUID="d658772c-b072-4513-a639-d19a94c78525" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:32.973685 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:32.715868    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2" podUID="e2483a06-2f4b-4a1e-af4e-a63eb50d29bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": context deadline exceeded"
Nov 01 03:28:33.006726 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:32.715915    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw" podUID="b6d688b8-b911-4a6a-a842-2b0fb7a75e83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 01 03:28:33.022669 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:32.716796    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t" podUID="13930c1e-2200-4412-b8d3-ae3e4126a851" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:33.092596 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:32.716870    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm" podUID="75d142b3-7172-4b57-bc07-a2f404c4656e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": context deadline exceeded"
Nov 01 03:28:33.092596 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:32.797056    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" podUID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:33.092596 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:32.885945    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8" podUID="88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Nov 01 03:28:33.435068 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:33.435008    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n" podUID="d2511772-66f9-40a0-9166-87c4345485c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 01 03:28:33.791663 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:32.715692    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp" podUID="667240b0-64a7-45cd-84fe-9cd7e0d8a301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:33.791663 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:33.660503    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-64vp4" podUID="16d41e60-8b45-447c-aec0-798c5712ece0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:33.791663 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:33.672054    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm" podUID="b2637bc6-3828-4b75-9783-5415519403a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 01 03:28:33.791663 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:33.677622    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7" podUID="1ce834cc-d34d-46cc-a081-4d2cc261becb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:33.791663 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:33.746705    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" podUID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:33.939876 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:33.939826    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj" podUID="b0e32cb3-ac18-4c1f-819c-f1139a88bfea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:33.942723 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:33.942685    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf" podUID="40881383-61bb-4fc8-9c75-f8ffce4834c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:34.068765 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:33.973797    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps" podUID="d658772c-b072-4513-a639-d19a94c78525" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 01 03:28:34.069147 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:33.985256    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2" podUID="e2483a06-2f4b-4a1e-af4e-a63eb50d29bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:34.072664 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:34.031494    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t" podUID="13930c1e-2200-4412-b8d3-ae3e4126a851" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 01 03:28:34.097578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:34.031581    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw" podUID="b6d688b8-b911-4a6a-a842-2b0fb7a75e83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:34.097994 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:34.064716    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8" podUID="88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:34.167583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:34.064802    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm" podUID="75d142b3-7172-4b57-bc07-a2f404c4656e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:34.171622 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:34.064830    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" podUID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded"
Nov 01 03:28:34.307158 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:34.307063    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:34.568744 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:34.568631    4930 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 03:28:34.569336 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:34.569281    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw" podUID="9cd5db87-0172-46ce-b856-cfc6366d7cb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:34.669861 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:34.669812    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-5rwkr" podUID="53a7d5be-9872-49c5-9f21-f1e2f55d444a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:34.683523 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:34.683474    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-qhchk" podUID="98c81f0f-a540-4758-a3fa-e46f639e89d6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:34.796723 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:34.796680    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n" podUID="d2511772-66f9-40a0-9166-87c4345485c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 01 03:28:34.797420 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:34.797374    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp" podUID="667240b0-64a7-45cd-84fe-9cd7e0d8a301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 01 03:28:34.843511 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:34.797649    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm" podUID="b2637bc6-3828-4b75-9783-5415519403a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 01 03:28:34.844107 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:34.797713    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7" podUID="1ce834cc-d34d-46cc-a081-4d2cc261becb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:34.897484 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:28:34.894855    4930 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.222s"
Nov 01 03:28:34.920171 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:34.920141    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10863eb9-6zd52"
Nov 01 03:28:34.924363 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:34.924303    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10863eb9-6zd52" event={"ID":"3934c25e-4360-44ca-a355-abb425ebed69","Type":"ContainerDied","Data":"1feb0e3e6102475b4d0eb11f778a67799bc5448c8cf767889032b1287c3d1d74"}
Nov 01 03:28:35.014752 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.014714    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1feb0e3e6102475b4d0eb11f778a67799bc5448c8cf767889032b1287c3d1d74"
Nov 01 03:28:35.014951 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:34.959301    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj" podUID="b0e32cb3-ac18-4c1f-819c-f1139a88bfea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:35.015935 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:34.956821    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3934c25e-4360-44ca-a355-abb425ebed69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3934c25e-4360-44ca-a355-abb425ebed69-maas-api-xks-ca\") pod \"3934c25e-4360-44ca-a355-abb425ebed69\" (UID: \"3934c25e-4360-44ca-a355-abb425ebed69\") "
Nov 01 03:28:35.016193 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.016173    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3934c25e-4360-44ca-a355-abb425ebed69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3934c25e-4360-44ca-a355-abb425ebed69-maas-api-openshift-ca\") pod \"3934c25e-4360-44ca-a355-abb425ebed69\" (UID: \"3934c25e-4360-44ca-a355-abb425ebed69\") "
Nov 01 03:28:35.069868 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.069819    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps" podUID="d658772c-b072-4513-a639-d19a94c78525" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:35.102633 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.101780    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf" podUID="40881383-61bb-4fc8-9c75-f8ffce4834c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:35.126579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.125773    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw" podUID="b6d688b8-b911-4a6a-a842-2b0fb7a75e83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:35.158099 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.158052    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t" podUID="13930c1e-2200-4412-b8d3-ae3e4126a851" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:35.160434 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.160406    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3934c25e-4360-44ca-a355-abb425ebed69-kube-api-access-tsgl4\" (UniqueName: \"kubernetes.io/projected/3934c25e-4360-44ca-a355-abb425ebed69-kube-api-access-tsgl4\") pod \"3934c25e-4360-44ca-a355-abb425ebed69\" (UID: \"3934c25e-4360-44ca-a355-abb425ebed69\") "
Nov 01 03:28:35.339794 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.339739    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8" podUID="88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:35.341261 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.341229    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2" podUID="e2483a06-2f4b-4a1e-af4e-a63eb50d29bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:35.342247 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.342222    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3934c25e-4360-44ca-a355-abb425ebed69-maas-api-openshift-ca" pod "3934c25e-4360-44ca-a355-abb425ebed69" (UID: "3934c25e-4360-44ca-a355-abb425ebed69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:28:35.370005 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.353678    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm" podUID="75d142b3-7172-4b57-bc07-a2f404c4656e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": context deadline exceeded"
Nov 01 03:28:35.370005 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.357403    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" podUID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded"
Nov 01 03:28:35.404576 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.399549    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f2040c8-csxkb"]
Nov 01 03:28:35.404576 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.399628    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f2040c8-csxkb"]
Nov 01 03:28:35.404576 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.399668    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx"]
Nov 01 03:28:35.404576 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.399916    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx" podUID="fc508d65-2f4b-4cf1-aec1-8d229baef624" containerName="istio-proxy" containerID="cri-o://12df7a95ac96d8046f62b0152f8172986d41576d330ef368406262d186b53926" gracePeriod=30
Nov 01 03:28:35.405534 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.405436    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf" podUID="40881383-61bb-4fc8-9c75-f8ffce4834c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:35.422573 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.407195    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3934c25e-4360-44ca-a355-abb425ebed69-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:35.445894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.411219    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts"]
Nov 01 03:28:35.445894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.436355    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts" podUID="4ba1754d-0939-4b6a-b6d0-69155eb4cfa0" containerName="istio-proxy" containerID="cri-o://a77bf021a96e4721e04519717998f814ff214ba344930518070f2ec898d1903a" gracePeriod=30
Nov 01 03:28:35.445894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.431695    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3934c25e-4360-44ca-a355-abb425ebed69-maas-api-xks-ca" pod "3934c25e-4360-44ca-a355-abb425ebed69" (UID: "3934c25e-4360-44ca-a355-abb425ebed69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:28:35.445894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.438915    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:28:35.445894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.438963    4930 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 03:28:35.445894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.439007    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n" podUID="72c2d7cf-fac4-4d80-a5f1-82ae300a255a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:35.445894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.440077    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7"]
Nov 01 03:28:35.445894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.440257    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7" podUID="e325fcbf-d504-4923-b0fd-809005447f3e" containerName="istio-proxy" containerID="cri-o://8fc9c28599b028d464ecb875484b3d5d8c6db7cb616e7c96ba2fd6d8e019dd75" gracePeriod=30
Nov 01 03:28:35.445894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.443028    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp"
Nov 01 03:28:35.445894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.443117    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3934c25e-4360-44ca-a355-abb425ebed69-kube-api-access-tsgl4" pod "3934c25e-4360-44ca-a355-abb425ebed69" (UID: "3934c25e-4360-44ca-a355-abb425ebed69"). InnerVolumeSpecName "kube-api-access-tsgl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:28:35.456674 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.455649    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k"]
Nov 01 03:28:35.456674 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.455840    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k" podUID="85529ed7-e2d9-470f-a897-1711fd4abfa0" containerName="istio-proxy" containerID="cri-o://9018f3823ac66b77bb260a122a93c861dad45b752d4a6b6c361e24e975bbcc51" gracePeriod=30
Nov 01 03:28:35.497089 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.495633    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm" podUID="b2637bc6-3828-4b75-9783-5415519403a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:35.542673 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.537637    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3934c25e-4360-44ca-a355-abb425ebed69-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:35.542673 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.537667    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsgl4\" (UniqueName: \"kubernetes.io/projected/3934c25e-4360-44ca-a355-abb425ebed69-kube-api-access-tsgl4\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:35.563574 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.556690    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw" podUID="b6d688b8-b911-4a6a-a842-2b0fb7a75e83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:28:35.563574 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.559010    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:28:35.575585 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.571001    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:28:35.575585 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.573135    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:28:35.663620 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.659330    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5370e16f-3d6e-4e90-9f29-dcafa5110b74" path="/var/lib/kubelet/pods/5370e16f-3d6e-4e90-9f29-dcafa5110b74/volumes"
Nov 01 03:28:35.663620 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.660165    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"
Nov 01 03:28:35.663620 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.660227    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:28:35.663620 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.660276    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"
Nov 01 03:28:35.663620 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.660322    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"
Nov 01 03:28:35.672868 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.672842    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:28:35.703754 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.703726    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"
Nov 01 03:28:35.704059 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.704045    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:28:35.707880 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.707858    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:28:35.932101 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.932028    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10863eb9-6zd52"
Nov 01 03:28:35.985213 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.985183    4930 scope.go:122] "RemoveContainer" containerID="2a6dfd371c11f0e968ad6c03efe78333b31774ac158ce3e54933f448c2bfc64d"
Nov 01 03:28:35.999179 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:35.999156    4930 scope.go:122] "RemoveContainer" containerID="a34cc1bfa7a268f120bcbb3db0038e16d8d2064c5eb930c60cd72019601db001"
Nov 01 03:28:36.010050 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:36.010026    4930 scope.go:122] "RemoveContainer" containerID="694d2edc6a51ea2a838cfd9ddc26ff654b87e680bbcd8145dc2ead2a4d2fff30"
Nov 01 03:28:36.020037 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:36.020015    4930 scope.go:122] "RemoveContainer" containerID="40e51fcaf088ad0baf41f77b1aaf25966e82157368190aaa052cfb29c5fadc56"
Nov 01 03:28:36.030147 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:36.030129    4930 scope.go:122] "RemoveContainer" containerID="02ac00a89967f15549a1a180f55aeea5c3aea43b3eac0a260fada7bb3daa836f"
Nov 01 03:28:36.039881 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:36.039861    4930 scope.go:122] "RemoveContainer" containerID="59f1c118b65f5ae4087b1408fe234e5034851a3a2d8129b434b5c92148697f2c"
Nov 01 03:28:36.048870 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:36.048853    4930 scope.go:122] "RemoveContainer" containerID="5e9bd79aad194779114270da6c2341c46d670157ceeb244bfa6ecd7282ca5546"
Nov 01 03:28:36.386577 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:36.386533    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:36.908102 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:36.908069    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"]
Nov 01 03:28:36.908938 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:36.908916    4930 cpu_manager.go:414] "containerMap: removing container" podUID="3934c25e-4360-44ca-a355-abb425ebed69" containerName="revoke-keys"
Nov 01 03:28:36.908938 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:36.908938    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3934c25e-4360-44ca-a355-abb425ebed69" containerName="revoke-keys"
Nov 01 03:28:36.909101 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:36.908997    4930 cpu_manager.go:414] "containerMap: removing container" podUID="fc5cffef-d29f-4ade-b650-f8904b356bcd" containerName="revoke-keys"
Nov 01 03:28:36.909101 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:36.909006    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc5cffef-d29f-4ade-b650-f8904b356bcd" containerName="revoke-keys"
Nov 01 03:28:36.909219 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:36.909204    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc5cffef-d29f-4ade-b650-f8904b356bcd" containerName="revoke-keys"
Nov 01 03:28:36.909269 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:36.909232    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="3934c25e-4360-44ca-a355-abb425ebed69" containerName="revoke-keys"
Nov 01 03:28:36.914528 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:36.914508    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"
Nov 01 03:28:36.971501 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:36.971466    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d5c42938-openshift-default-dockercfg-8wtr9\""
Nov 01 03:28:36.984295 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:36.984214    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fd59ab7-fd50-4ce9-a290-7f018204eb66-istio-token\") pod \"e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f\" (UID: \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\") " pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"
Nov 01 03:28:36.984295 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:36.984288    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fd59ab7-fd50-4ce9-a290-7f018204eb66-workload-certs\") pod \"e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f\" (UID: \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\") " pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"
Nov 01 03:28:36.984514 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:36.984326    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fd59ab7-fd50-4ce9-a290-7f018204eb66-istiod-ca-cert\") pod \"e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f\" (UID: \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\") " pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"
Nov 01 03:28:36.984514 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:36.984379    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd59ab7-fd50-4ce9-a290-7f018204eb66-workload-socket\") pod \"e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f\" (UID: \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\") " pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"
Nov 01 03:28:36.984514 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:36.984417    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fd59ab7-fd50-4ce9-a290-7f018204eb66-istio-envoy\") pod \"e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f\" (UID: \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\") " pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"
Nov 01 03:28:36.984514 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:36.984470    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fd59ab7-fd50-4ce9-a290-7f018204eb66-istio-podinfo\") pod \"e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f\" (UID: \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\") " pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"
Nov 01 03:28:36.984762 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:36.984598    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-247st\" (UniqueName: \"kubernetes.io/projected/0fd59ab7-fd50-4ce9-a290-7f018204eb66-kube-api-access-247st\") pod \"e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f\" (UID: \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\") " pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"
Nov 01 03:28:36.984762 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:36.984711    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd59ab7-fd50-4ce9-a290-7f018204eb66-credential-socket\") pod \"e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f\" (UID: \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\") " pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"
Nov 01 03:28:36.984861 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:36.984773    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fd59ab7-fd50-4ce9-a290-7f018204eb66-istio-data\") pod \"e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f\" (UID: \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\") " pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"
Nov 01 03:28:36.985963 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:36.985937    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"]
Nov 01 03:28:37.012117 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:37.012029    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:37.097796 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:37.095361    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fd59ab7-fd50-4ce9-a290-7f018204eb66-istio-token\") pod \"e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f\" (UID: \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\") " pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"
Nov 01 03:28:37.097796 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:37.095464    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fd59ab7-fd50-4ce9-a290-7f018204eb66-workload-certs\") pod \"e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f\" (UID: \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\") " pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"
Nov 01 03:28:37.097796 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:37.095497    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fd59ab7-fd50-4ce9-a290-7f018204eb66-istiod-ca-cert\") pod \"e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f\" (UID: \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\") " pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"
Nov 01 03:28:37.097796 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:37.095550    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd59ab7-fd50-4ce9-a290-7f018204eb66-workload-socket\") pod \"e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f\" (UID: \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\") " pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"
Nov 01 03:28:37.097796 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:37.095601    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fd59ab7-fd50-4ce9-a290-7f018204eb66-istio-envoy\") pod \"e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f\" (UID: \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\") " pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"
Nov 01 03:28:37.097796 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:37.095660    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fd59ab7-fd50-4ce9-a290-7f018204eb66-istio-podinfo\") pod \"e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f\" (UID: \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\") " pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"
Nov 01 03:28:37.097796 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:37.095789    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-247st\" (UniqueName: \"kubernetes.io/projected/0fd59ab7-fd50-4ce9-a290-7f018204eb66-kube-api-access-247st\") pod \"e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f\" (UID: \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\") " pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"
Nov 01 03:28:37.097796 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:37.095885    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd59ab7-fd50-4ce9-a290-7f018204eb66-credential-socket\") pod \"e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f\" (UID: \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\") " pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"
Nov 01 03:28:37.097796 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:37.095948    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fd59ab7-fd50-4ce9-a290-7f018204eb66-istio-data\") pod \"e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f\" (UID: \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\") " pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"
Nov 01 03:28:37.183628 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:37.100475    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd59ab7-fd50-4ce9-a290-7f018204eb66-workload-socket\") pod \"e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f\" (UID: \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\") " pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"
Nov 01 03:28:37.183628 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:37.101074    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fd59ab7-fd50-4ce9-a290-7f018204eb66-workload-certs\") pod \"e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f\" (UID: \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\") " pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"
Nov 01 03:28:37.183628 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:37.101779    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fd59ab7-fd50-4ce9-a290-7f018204eb66-istiod-ca-cert\") pod \"e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f\" (UID: \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\") " pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"
Nov 01 03:28:37.183628 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:37.150999    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd59ab7-fd50-4ce9-a290-7f018204eb66-credential-socket\") pod \"e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f\" (UID: \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\") " pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"
Nov 01 03:28:37.183628 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:37.174347    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fd59ab7-fd50-4ce9-a290-7f018204eb66-istio-data\") pod \"e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f\" (UID: \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\") " pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"
Nov 01 03:28:37.229657 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:37.204996    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:37.430430 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:37.430376    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fd59ab7-fd50-4ce9-a290-7f018204eb66-istio-podinfo\") pod \"e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f\" (UID: \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\") " pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"
Nov 01 03:28:37.430639 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:37.430454    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fd59ab7-fd50-4ce9-a290-7f018204eb66-istio-envoy\") pod \"e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f\" (UID: \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\") " pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"
Nov 01 03:28:37.430898 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:37.430869    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fd59ab7-fd50-4ce9-a290-7f018204eb66-istio-token\") pod \"e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f\" (UID: \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\") " pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"
Nov 01 03:28:37.435725 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:37.435132    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-247st\" (UniqueName: \"kubernetes.io/projected/0fd59ab7-fd50-4ce9-a290-7f018204eb66-kube-api-access-247st\") pod \"e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f\" (UID: \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\") " pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"
Nov 01 03:28:37.470392 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:37.470337    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:37.528905 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:37.528863    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"
Nov 01 03:28:37.664590 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:37.664269    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:38.120668 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:38.120638    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"]
Nov 01 03:28:38.121674 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:28:38.121651    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fd59ab7_fd50_4ce9_a290_7f018204eb66.slice/crio-3cf6635dca64f1c4bd30b0f2bb9f9b976bddc1a8044399ac43de6486eb51d30d WatchSource:0}: Error finding container 3cf6635dca64f1c4bd30b0f2bb9f9b976bddc1a8044399ac43de6486eb51d30d: Status 404 returned error can't find the container with id 3cf6635dca64f1c4bd30b0f2bb9f9b976bddc1a8044399ac43de6486eb51d30d
Nov 01 03:28:38.124309 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:38.124279    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:28:38.124387 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:38.124343    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:28:38.124387 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:38.124372    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:28:38.255733 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:38.255701    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:38.427603 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:38.427480    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:39.017266 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:39.017223    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f" event={"ID":"0fd59ab7-fd50-4ce9-a290-7f018204eb66","Type":"ContainerStarted","Data":"fb093f377b7645fc576a77c9f9f21a1b0c51c1d7a97d12fc4c2a174636fa2df8"}
Nov 01 03:28:39.017482 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:39.017296    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f" event={"ID":"0fd59ab7-fd50-4ce9-a290-7f018204eb66","Type":"ContainerStarted","Data":"3cf6635dca64f1c4bd30b0f2bb9f9b976bddc1a8044399ac43de6486eb51d30d"}
Nov 01 03:28:39.261992 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:39.261953    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:39.371758 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:39.324606    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f" podStartSLOduration=3.324584709 podStartE2EDuration="3.324584709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:28:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:28:39.17935888 +0000 UTC m=+1323.702365469" watchObservedRunningTime="2026-11-01 03:28:39.324584709 +0000 UTC m=+1323.847591297"
Nov 01 03:28:39.542658 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:39.540658    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"
Nov 01 03:28:39.566575 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:39.544282    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:39.566575 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:39.558997    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"
Nov 01 03:28:39.935311 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:39.925175    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-367a7f78-w4t5v"]
Nov 01 03:28:39.935311 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:39.930870    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-367a7f78-w4t5v"
Nov 01 03:28:39.946708 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:39.946680    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:28:39.946839 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:39.946732    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7" podUID="e325fcbf-d504-4923-b0fd-809005447f3e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:28:39.984734 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:39.984690    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:28:39.984894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:39.984798    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:28:39.984894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:39.984879    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:28:39.985209 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:39.985191    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:28:39.986477 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:39.986455    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-367a7f78-w4t5v"]
Nov 01 03:28:40.029513 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.029479    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"
Nov 01 03:28:40.030696 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.030673    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"
Nov 01 03:28:40.035487 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.035458    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db76ac41-529c-4927-b1fc-b029877dd3fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-367a7f78-w4t5v\" (UID: \"db76ac41-529c-4927-b1fc-b029877dd3fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367a7f78-w4t5v"
Nov 01 03:28:40.035648 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.035488    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4lxk\" (UniqueName: \"kubernetes.io/projected/db76ac41-529c-4927-b1fc-b029877dd3fc-kube-api-access-c4lxk\") pod \"maas-api-revoke-sub-367a7f78-w4t5v\" (UID: \"db76ac41-529c-4927-b1fc-b029877dd3fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367a7f78-w4t5v"
Nov 01 03:28:40.035775 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.035747    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db76ac41-529c-4927-b1fc-b029877dd3fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-367a7f78-w4t5v\" (UID: \"db76ac41-529c-4927-b1fc-b029877dd3fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367a7f78-w4t5v"
Nov 01 03:28:40.137712 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.137673    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db76ac41-529c-4927-b1fc-b029877dd3fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-367a7f78-w4t5v\" (UID: \"db76ac41-529c-4927-b1fc-b029877dd3fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367a7f78-w4t5v"
Nov 01 03:28:40.137866 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.137722    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4lxk\" (UniqueName: \"kubernetes.io/projected/db76ac41-529c-4927-b1fc-b029877dd3fc-kube-api-access-c4lxk\") pod \"maas-api-revoke-sub-367a7f78-w4t5v\" (UID: \"db76ac41-529c-4927-b1fc-b029877dd3fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367a7f78-w4t5v"
Nov 01 03:28:40.137952 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.137930    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db76ac41-529c-4927-b1fc-b029877dd3fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-367a7f78-w4t5v\" (UID: \"db76ac41-529c-4927-b1fc-b029877dd3fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367a7f78-w4t5v"
Nov 01 03:28:40.139236 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.139204    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db76ac41-529c-4927-b1fc-b029877dd3fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-367a7f78-w4t5v\" (UID: \"db76ac41-529c-4927-b1fc-b029877dd3fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367a7f78-w4t5v"
Nov 01 03:28:40.141681 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.141079    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db76ac41-529c-4927-b1fc-b029877dd3fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-367a7f78-w4t5v\" (UID: \"db76ac41-529c-4927-b1fc-b029877dd3fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367a7f78-w4t5v"
Nov 01 03:28:40.215417 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.215323    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4lxk\" (UniqueName: \"kubernetes.io/projected/db76ac41-529c-4927-b1fc-b029877dd3fc-kube-api-access-c4lxk\") pod \"maas-api-revoke-sub-367a7f78-w4t5v\" (UID: \"db76ac41-529c-4927-b1fc-b029877dd3fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367a7f78-w4t5v"
Nov 01 03:28:40.255282 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.255248    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-367a7f78-w4t5v"
Nov 01 03:28:40.322740 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.322681    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a07782b1-wjp8r"]
Nov 01 03:28:40.348063 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.343260    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07782b1-wjp8r"
Nov 01 03:28:40.416756 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.416727    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a07782b1-wjp8r"]
Nov 01 03:28:40.444641 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.444608    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540c82b7-8782-49b8-86a4-576e10e21357-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a07782b1-wjp8r\" (UID: \"540c82b7-8782-49b8-86a4-576e10e21357\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07782b1-wjp8r"
Nov 01 03:28:40.444838 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.444819    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540c82b7-8782-49b8-86a4-576e10e21357-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a07782b1-wjp8r\" (UID: \"540c82b7-8782-49b8-86a4-576e10e21357\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07782b1-wjp8r"
Nov 01 03:28:40.444968 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.444954    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwsld\" (UniqueName: \"kubernetes.io/projected/540c82b7-8782-49b8-86a4-576e10e21357-kube-api-access-xwsld\") pod \"maas-api-revoke-sub-a07782b1-wjp8r\" (UID: \"540c82b7-8782-49b8-86a4-576e10e21357\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07782b1-wjp8r"
Nov 01 03:28:40.548624 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.546270    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwsld\" (UniqueName: \"kubernetes.io/projected/540c82b7-8782-49b8-86a4-576e10e21357-kube-api-access-xwsld\") pod \"maas-api-revoke-sub-a07782b1-wjp8r\" (UID: \"540c82b7-8782-49b8-86a4-576e10e21357\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07782b1-wjp8r"
Nov 01 03:28:40.548624 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.546948    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540c82b7-8782-49b8-86a4-576e10e21357-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a07782b1-wjp8r\" (UID: \"540c82b7-8782-49b8-86a4-576e10e21357\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07782b1-wjp8r"
Nov 01 03:28:40.548624 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.547010    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540c82b7-8782-49b8-86a4-576e10e21357-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a07782b1-wjp8r\" (UID: \"540c82b7-8782-49b8-86a4-576e10e21357\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07782b1-wjp8r"
Nov 01 03:28:40.548624 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.547781    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540c82b7-8782-49b8-86a4-576e10e21357-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a07782b1-wjp8r\" (UID: \"540c82b7-8782-49b8-86a4-576e10e21357\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07782b1-wjp8r"
Nov 01 03:28:40.558002 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.557958    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540c82b7-8782-49b8-86a4-576e10e21357-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a07782b1-wjp8r\" (UID: \"540c82b7-8782-49b8-86a4-576e10e21357\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07782b1-wjp8r"
Nov 01 03:28:40.616391 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.616176    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwsld\" (UniqueName: \"kubernetes.io/projected/540c82b7-8782-49b8-86a4-576e10e21357-kube-api-access-xwsld\") pod \"maas-api-revoke-sub-a07782b1-wjp8r\" (UID: \"540c82b7-8782-49b8-86a4-576e10e21357\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07782b1-wjp8r"
Nov 01 03:28:40.672244 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.671902    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07782b1-wjp8r"
Nov 01 03:28:40.740826 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.740793    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx"
Nov 01 03:28:40.803165 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.803126    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-367a7f78-w4t5v"]
Nov 01 03:28:40.803818 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:28:40.803786    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb76ac41_529c_4927_b1fc_b029877dd3fc.slice/crio-34a41bbc02247c1a7175ecc32e7a3cabb6ef9f3bf018db2a26eba3d090b70de9 WatchSource:0}: Error finding container 34a41bbc02247c1a7175ecc32e7a3cabb6ef9f3bf018db2a26eba3d090b70de9: Status 404 returned error can't find the container with id 34a41bbc02247c1a7175ecc32e7a3cabb6ef9f3bf018db2a26eba3d090b70de9
Nov 01 03:28:40.854762 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.853071    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc508d65-2f4b-4cf1-aec1-8d229baef624-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc508d65-2f4b-4cf1-aec1-8d229baef624-workload-socket\") pod \"fc508d65-2f4b-4cf1-aec1-8d229baef624\" (UID: \"fc508d65-2f4b-4cf1-aec1-8d229baef624\") "
Nov 01 03:28:40.854762 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.853198    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc508d65-2f4b-4cf1-aec1-8d229baef624-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc508d65-2f4b-4cf1-aec1-8d229baef624-istio-envoy\") pod \"fc508d65-2f4b-4cf1-aec1-8d229baef624\" (UID: \"fc508d65-2f4b-4cf1-aec1-8d229baef624\") "
Nov 01 03:28:40.854762 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.853245    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc508d65-2f4b-4cf1-aec1-8d229baef624-istio-token\" (UniqueName: \"kubernetes.io/projected/fc508d65-2f4b-4cf1-aec1-8d229baef624-istio-token\") pod \"fc508d65-2f4b-4cf1-aec1-8d229baef624\" (UID: \"fc508d65-2f4b-4cf1-aec1-8d229baef624\") "
Nov 01 03:28:40.854762 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.853319    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc508d65-2f4b-4cf1-aec1-8d229baef624-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc508d65-2f4b-4cf1-aec1-8d229baef624-workload-certs\") pod \"fc508d65-2f4b-4cf1-aec1-8d229baef624\" (UID: \"fc508d65-2f4b-4cf1-aec1-8d229baef624\") "
Nov 01 03:28:40.854762 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.853362    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc508d65-2f4b-4cf1-aec1-8d229baef624-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc508d65-2f4b-4cf1-aec1-8d229baef624-istio-data\") pod \"fc508d65-2f4b-4cf1-aec1-8d229baef624\" (UID: \"fc508d65-2f4b-4cf1-aec1-8d229baef624\") "
Nov 01 03:28:40.854762 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.853429    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc508d65-2f4b-4cf1-aec1-8d229baef624-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc508d65-2f4b-4cf1-aec1-8d229baef624-istio-podinfo\") pod \"fc508d65-2f4b-4cf1-aec1-8d229baef624\" (UID: \"fc508d65-2f4b-4cf1-aec1-8d229baef624\") "
Nov 01 03:28:40.854762 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.853618    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc508d65-2f4b-4cf1-aec1-8d229baef624-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc508d65-2f4b-4cf1-aec1-8d229baef624-credential-socket\") pod \"fc508d65-2f4b-4cf1-aec1-8d229baef624\" (UID: \"fc508d65-2f4b-4cf1-aec1-8d229baef624\") "
Nov 01 03:28:40.854762 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.854214    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc508d65-2f4b-4cf1-aec1-8d229baef624-workload-certs" pod "fc508d65-2f4b-4cf1-aec1-8d229baef624" (UID: "fc508d65-2f4b-4cf1-aec1-8d229baef624"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:28:40.854762 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.854434    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc508d65-2f4b-4cf1-aec1-8d229baef624-workload-socket" pod "fc508d65-2f4b-4cf1-aec1-8d229baef624" (UID: "fc508d65-2f4b-4cf1-aec1-8d229baef624"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:28:40.854762 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.854730    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc508d65-2f4b-4cf1-aec1-8d229baef624-kube-api-access-nzkdx\" (UniqueName: \"kubernetes.io/projected/fc508d65-2f4b-4cf1-aec1-8d229baef624-kube-api-access-nzkdx\") pod \"fc508d65-2f4b-4cf1-aec1-8d229baef624\" (UID: \"fc508d65-2f4b-4cf1-aec1-8d229baef624\") "
Nov 01 03:28:40.855833 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.854826    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc508d65-2f4b-4cf1-aec1-8d229baef624-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc508d65-2f4b-4cf1-aec1-8d229baef624-istiod-ca-cert\") pod \"fc508d65-2f4b-4cf1-aec1-8d229baef624\" (UID: \"fc508d65-2f4b-4cf1-aec1-8d229baef624\") "
Nov 01 03:28:40.858373 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.856687    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc508d65-2f4b-4cf1-aec1-8d229baef624-istio-data" pod "fc508d65-2f4b-4cf1-aec1-8d229baef624" (UID: "fc508d65-2f4b-4cf1-aec1-8d229baef624"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:28:40.858373 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.856887    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc508d65-2f4b-4cf1-aec1-8d229baef624-credential-socket" pod "fc508d65-2f4b-4cf1-aec1-8d229baef624" (UID: "fc508d65-2f4b-4cf1-aec1-8d229baef624"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:28:40.858373 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.857507    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc508d65-2f4b-4cf1-aec1-8d229baef624-istiod-ca-cert" pod "fc508d65-2f4b-4cf1-aec1-8d229baef624" (UID: "fc508d65-2f4b-4cf1-aec1-8d229baef624"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:28:40.858373 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.858013    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc508d65-2f4b-4cf1-aec1-8d229baef624-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:40.858373 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.858037    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc508d65-2f4b-4cf1-aec1-8d229baef624-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:40.858373 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.858050    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc508d65-2f4b-4cf1-aec1-8d229baef624-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:40.858373 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.858063    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc508d65-2f4b-4cf1-aec1-8d229baef624-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:40.858373 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.858077    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc508d65-2f4b-4cf1-aec1-8d229baef624-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:40.867177 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.867125    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc508d65-2f4b-4cf1-aec1-8d229baef624-istio-podinfo" pod "fc508d65-2f4b-4cf1-aec1-8d229baef624" (UID: "fc508d65-2f4b-4cf1-aec1-8d229baef624"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:28:40.867414 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.867274    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc508d65-2f4b-4cf1-aec1-8d229baef624-istio-envoy" pod "fc508d65-2f4b-4cf1-aec1-8d229baef624" (UID: "fc508d65-2f4b-4cf1-aec1-8d229baef624"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:28:40.868027 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.867985    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc508d65-2f4b-4cf1-aec1-8d229baef624-istio-token" pod "fc508d65-2f4b-4cf1-aec1-8d229baef624" (UID: "fc508d65-2f4b-4cf1-aec1-8d229baef624"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:28:40.869770 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.869734    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc508d65-2f4b-4cf1-aec1-8d229baef624-kube-api-access-nzkdx" pod "fc508d65-2f4b-4cf1-aec1-8d229baef624" (UID: "fc508d65-2f4b-4cf1-aec1-8d229baef624"). InnerVolumeSpecName "kube-api-access-nzkdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:28:40.889031 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.888949    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7"
Nov 01 03:28:40.958024 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.957984    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts"
Nov 01 03:28:40.959539 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.959505    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e325fcbf-d504-4923-b0fd-809005447f3e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e325fcbf-d504-4923-b0fd-809005447f3e-istio-podinfo\") pod \"e325fcbf-d504-4923-b0fd-809005447f3e\" (UID: \"e325fcbf-d504-4923-b0fd-809005447f3e\") "
Nov 01 03:28:40.959762 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.959659    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e325fcbf-d504-4923-b0fd-809005447f3e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e325fcbf-d504-4923-b0fd-809005447f3e-istio-envoy\") pod \"e325fcbf-d504-4923-b0fd-809005447f3e\" (UID: \"e325fcbf-d504-4923-b0fd-809005447f3e\") "
Nov 01 03:28:40.959835 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.959794    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e325fcbf-d504-4923-b0fd-809005447f3e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e325fcbf-d504-4923-b0fd-809005447f3e-workload-certs\") pod \"e325fcbf-d504-4923-b0fd-809005447f3e\" (UID: \"e325fcbf-d504-4923-b0fd-809005447f3e\") "
Nov 01 03:28:40.959958 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.959924    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e325fcbf-d504-4923-b0fd-809005447f3e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e325fcbf-d504-4923-b0fd-809005447f3e-credential-socket\") pod \"e325fcbf-d504-4923-b0fd-809005447f3e\" (UID: \"e325fcbf-d504-4923-b0fd-809005447f3e\") "
Nov 01 03:28:40.960033 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.959994    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e325fcbf-d504-4923-b0fd-809005447f3e-istio-token\" (UniqueName: \"kubernetes.io/projected/e325fcbf-d504-4923-b0fd-809005447f3e-istio-token\") pod \"e325fcbf-d504-4923-b0fd-809005447f3e\" (UID: \"e325fcbf-d504-4923-b0fd-809005447f3e\") "
Nov 01 03:28:40.960126 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.960046    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e325fcbf-d504-4923-b0fd-809005447f3e-kube-api-access-djtps\" (UniqueName: \"kubernetes.io/projected/e325fcbf-d504-4923-b0fd-809005447f3e-kube-api-access-djtps\") pod \"e325fcbf-d504-4923-b0fd-809005447f3e\" (UID: \"e325fcbf-d504-4923-b0fd-809005447f3e\") "
Nov 01 03:28:40.960187 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.960147    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e325fcbf-d504-4923-b0fd-809005447f3e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e325fcbf-d504-4923-b0fd-809005447f3e-workload-socket\") pod \"e325fcbf-d504-4923-b0fd-809005447f3e\" (UID: \"e325fcbf-d504-4923-b0fd-809005447f3e\") "
Nov 01 03:28:40.960265 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.960230    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e325fcbf-d504-4923-b0fd-809005447f3e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e325fcbf-d504-4923-b0fd-809005447f3e-istiod-ca-cert\") pod \"e325fcbf-d504-4923-b0fd-809005447f3e\" (UID: \"e325fcbf-d504-4923-b0fd-809005447f3e\") "
Nov 01 03:28:40.960455 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.960242    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e325fcbf-d504-4923-b0fd-809005447f3e-credential-socket" pod "e325fcbf-d504-4923-b0fd-809005447f3e" (UID: "e325fcbf-d504-4923-b0fd-809005447f3e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:28:40.960646 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.960618    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e325fcbf-d504-4923-b0fd-809005447f3e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e325fcbf-d504-4923-b0fd-809005447f3e-istio-data\") pod \"e325fcbf-d504-4923-b0fd-809005447f3e\" (UID: \"e325fcbf-d504-4923-b0fd-809005447f3e\") "
Nov 01 03:28:40.961047 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.961020    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e325fcbf-d504-4923-b0fd-809005447f3e-istio-data" pod "e325fcbf-d504-4923-b0fd-809005447f3e" (UID: "e325fcbf-d504-4923-b0fd-809005447f3e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:28:40.962082 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.962058    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e325fcbf-d504-4923-b0fd-809005447f3e-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:40.962201 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.962095    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc508d65-2f4b-4cf1-aec1-8d229baef624-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:40.962201 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.962111    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc508d65-2f4b-4cf1-aec1-8d229baef624-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:40.962201 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.962127    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e325fcbf-d504-4923-b0fd-809005447f3e-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:40.962201 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.962148    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc508d65-2f4b-4cf1-aec1-8d229baef624-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:40.962201 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.962165    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzkdx\" (UniqueName: \"kubernetes.io/projected/fc508d65-2f4b-4cf1-aec1-8d229baef624-kube-api-access-nzkdx\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:40.962479 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.962373    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e325fcbf-d504-4923-b0fd-809005447f3e-workload-certs" pod "e325fcbf-d504-4923-b0fd-809005447f3e" (UID: "e325fcbf-d504-4923-b0fd-809005447f3e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:28:40.966228 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.966205    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e325fcbf-d504-4923-b0fd-809005447f3e-istiod-ca-cert" pod "e325fcbf-d504-4923-b0fd-809005447f3e" (UID: "e325fcbf-d504-4923-b0fd-809005447f3e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:28:40.966661 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:40.966630    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e325fcbf-d504-4923-b0fd-809005447f3e-workload-socket" pod "e325fcbf-d504-4923-b0fd-809005447f3e" (UID: "e325fcbf-d504-4923-b0fd-809005447f3e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:28:41.058249 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.058162    4930 generic.go:356] "Generic (PLEG): container finished" podID="e325fcbf-d504-4923-b0fd-809005447f3e" containerID="8fc9c28599b028d464ecb875484b3d5d8c6db7cb616e7c96ba2fd6d8e019dd75" exitCode=0
Nov 01 03:28:41.058530 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.058264    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7"
Nov 01 03:28:41.058849 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.058285    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7" event={"ID":"e325fcbf-d504-4923-b0fd-809005447f3e","Type":"ContainerDied","Data":"8fc9c28599b028d464ecb875484b3d5d8c6db7cb616e7c96ba2fd6d8e019dd75"}
Nov 01 03:28:41.060138 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.060113    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7" event={"ID":"e325fcbf-d504-4923-b0fd-809005447f3e","Type":"ContainerDied","Data":"8e413152a817d3ada4a843bd3d358795dae38c02fd99191119538cde21d25c6f"}
Nov 01 03:28:41.060278 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.060267    4930 scope.go:122] "RemoveContainer" containerID="8fc9c28599b028d464ecb875484b3d5d8c6db7cb616e7c96ba2fd6d8e019dd75"
Nov 01 03:28:41.060977 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.060952    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e325fcbf-d504-4923-b0fd-809005447f3e-istio-envoy" pod "e325fcbf-d504-4923-b0fd-809005447f3e" (UID: "e325fcbf-d504-4923-b0fd-809005447f3e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:28:41.061646 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.061621    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e325fcbf-d504-4923-b0fd-809005447f3e-istio-token" pod "e325fcbf-d504-4923-b0fd-809005447f3e" (UID: "e325fcbf-d504-4923-b0fd-809005447f3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:28:41.061950 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.061797    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e325fcbf-d504-4923-b0fd-809005447f3e-kube-api-access-djtps" pod "e325fcbf-d504-4923-b0fd-809005447f3e" (UID: "e325fcbf-d504-4923-b0fd-809005447f3e"). InnerVolumeSpecName "kube-api-access-djtps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:28:41.062070 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.061860    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e325fcbf-d504-4923-b0fd-809005447f3e-istio-podinfo" pod "e325fcbf-d504-4923-b0fd-809005447f3e" (UID: "e325fcbf-d504-4923-b0fd-809005447f3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:28:41.063489 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.063153    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-istio-token\" (UniqueName: \"kubernetes.io/projected/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-istio-token\") pod \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\" (UID: \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\") "
Nov 01 03:28:41.063749 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.063729    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-credential-socket\") pod \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\" (UID: \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\") "
Nov 01 03:28:41.064066 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.064046    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-istio-data\") pod \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\" (UID: \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\") "
Nov 01 03:28:41.064203 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.064189    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-workload-certs\") pod \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\" (UID: \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\") "
Nov 01 03:28:41.064330 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.064317    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-istio-podinfo\") pod \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\" (UID: \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\") "
Nov 01 03:28:41.064453 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.064439    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-kube-api-access-ltsb7\" (UniqueName: \"kubernetes.io/projected/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-kube-api-access-ltsb7\") pod \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\" (UID: \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\") "
Nov 01 03:28:41.064800 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.064762    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-istiod-ca-cert\") pod \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\" (UID: \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\") "
Nov 01 03:28:41.065225 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.065207    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e325fcbf-d504-4923-b0fd-809005447f3e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e325fcbf-d504-4923-b0fd-809005447f3e-istio-podinfo\") pod \"e325fcbf-d504-4923-b0fd-809005447f3e\" (UID: \"e325fcbf-d504-4923-b0fd-809005447f3e\") "
Nov 01 03:28:41.065420 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.065389    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-workload-socket\") pod \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\" (UID: \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\") "
Nov 01 03:28:41.065579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.065540    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-istio-envoy\") pod \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\" (UID: \"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0\") "
Nov 01 03:28:41.067256 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.066764    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-workload-socket" pod "4ba1754d-0939-4b6a-b6d0-69155eb4cfa0" (UID: "4ba1754d-0939-4b6a-b6d0-69155eb4cfa0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:28:41.067256 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.067011    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-credential-socket" pod "4ba1754d-0939-4b6a-b6d0-69155eb4cfa0" (UID: "4ba1754d-0939-4b6a-b6d0-69155eb4cfa0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:28:41.067256 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.067225    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-workload-certs" pod "4ba1754d-0939-4b6a-b6d0-69155eb4cfa0" (UID: "4ba1754d-0939-4b6a-b6d0-69155eb4cfa0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:28:41.067986 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.067939    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-istio-data" pod "4ba1754d-0939-4b6a-b6d0-69155eb4cfa0" (UID: "4ba1754d-0939-4b6a-b6d0-69155eb4cfa0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:28:41.068910 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.068676    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-istiod-ca-cert" pod "4ba1754d-0939-4b6a-b6d0-69155eb4cfa0" (UID: "4ba1754d-0939-4b6a-b6d0-69155eb4cfa0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:28:41.071311 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.071286    4930 generic.go:356] "Generic (PLEG): container finished" podID="85529ed7-e2d9-470f-a897-1711fd4abfa0" containerID="9018f3823ac66b77bb260a122a93c861dad45b752d4a6b6c361e24e975bbcc51" exitCode=0
Nov 01 03:28:41.071944 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.071921    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k" event={"ID":"85529ed7-e2d9-470f-a897-1711fd4abfa0","Type":"ContainerDied","Data":"9018f3823ac66b77bb260a122a93c861dad45b752d4a6b6c361e24e975bbcc51"}
Nov 01 03:28:41.073165 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.070012    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e325fcbf-d504-4923-b0fd-809005447f3e-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:41.073292 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.073275    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:41.073398 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.073384    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djtps\" (UniqueName: \"kubernetes.io/projected/e325fcbf-d504-4923-b0fd-809005447f3e-kube-api-access-djtps\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:41.073512 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.073500    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e325fcbf-d504-4923-b0fd-809005447f3e-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:41.074138 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.074119    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e325fcbf-d504-4923-b0fd-809005447f3e-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:41.074266 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.074253    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:41.074365 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.074352    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e325fcbf-d504-4923-b0fd-809005447f3e-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:41.074447 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.074436    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:41.074526 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.074515    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:41.074714 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.074701    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e325fcbf-d504-4923-b0fd-809005447f3e-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:41.078914 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.078135    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:41.078914 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:28:41.072230    4930 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/e325fcbf-d504-4923-b0fd-809005447f3e/volumes/kubernetes.io~downward-api/istio-podinfo
Nov 01 03:28:41.078914 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.078163    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e325fcbf-d504-4923-b0fd-809005447f3e-istio-podinfo" pod "e325fcbf-d504-4923-b0fd-809005447f3e" (UID: "e325fcbf-d504-4923-b0fd-809005447f3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:28:41.078914 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.078063    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-istio-envoy" pod "4ba1754d-0939-4b6a-b6d0-69155eb4cfa0" (UID: "4ba1754d-0939-4b6a-b6d0-69155eb4cfa0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:28:41.080295 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.079272    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-istio-token" pod "4ba1754d-0939-4b6a-b6d0-69155eb4cfa0" (UID: "4ba1754d-0939-4b6a-b6d0-69155eb4cfa0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:28:41.080295 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.079497    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-kube-api-access-ltsb7" pod "4ba1754d-0939-4b6a-b6d0-69155eb4cfa0" (UID: "4ba1754d-0939-4b6a-b6d0-69155eb4cfa0"). InnerVolumeSpecName "kube-api-access-ltsb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:28:41.080295 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.079653    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-istio-podinfo" pod "4ba1754d-0939-4b6a-b6d0-69155eb4cfa0" (UID: "4ba1754d-0939-4b6a-b6d0-69155eb4cfa0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:28:41.080295 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.079699    4930 generic.go:356] "Generic (PLEG): container finished" podID="4ba1754d-0939-4b6a-b6d0-69155eb4cfa0" containerID="a77bf021a96e4721e04519717998f814ff214ba344930518070f2ec898d1903a" exitCode=0
Nov 01 03:28:41.080295 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.079769    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts" event={"ID":"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0","Type":"ContainerDied","Data":"a77bf021a96e4721e04519717998f814ff214ba344930518070f2ec898d1903a"}
Nov 01 03:28:41.080295 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.079818    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts" event={"ID":"4ba1754d-0939-4b6a-b6d0-69155eb4cfa0","Type":"ContainerDied","Data":"af32df15583507f107ac567a95a663ecac427550458434bed52352e0f7b0bf0c"}
Nov 01 03:28:41.080295 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.079895    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts"
Nov 01 03:28:41.084382 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.084363    4930 scope.go:122] "RemoveContainer" containerID="8fc9c28599b028d464ecb875484b3d5d8c6db7cb616e7c96ba2fd6d8e019dd75"
Nov 01 03:28:41.085548 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:28:41.085252    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fc9c28599b028d464ecb875484b3d5d8c6db7cb616e7c96ba2fd6d8e019dd75\": container with ID starting with 8fc9c28599b028d464ecb875484b3d5d8c6db7cb616e7c96ba2fd6d8e019dd75 not found: ID does not exist" containerID="8fc9c28599b028d464ecb875484b3d5d8c6db7cb616e7c96ba2fd6d8e019dd75"
Nov 01 03:28:41.085548 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.085287    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fc9c28599b028d464ecb875484b3d5d8c6db7cb616e7c96ba2fd6d8e019dd75"} err="failed to get container status \"8fc9c28599b028d464ecb875484b3d5d8c6db7cb616e7c96ba2fd6d8e019dd75\": rpc error: code = NotFound desc = could not find container \"8fc9c28599b028d464ecb875484b3d5d8c6db7cb616e7c96ba2fd6d8e019dd75\": container with ID starting with 8fc9c28599b028d464ecb875484b3d5d8c6db7cb616e7c96ba2fd6d8e019dd75 not found: ID does not exist"
Nov 01 03:28:41.085548 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.085311    4930 scope.go:122] "RemoveContainer" containerID="a77bf021a96e4721e04519717998f814ff214ba344930518070f2ec898d1903a"
Nov 01 03:28:41.085548 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.085485    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-367a7f78-w4t5v" event={"ID":"db76ac41-529c-4927-b1fc-b029877dd3fc","Type":"ContainerStarted","Data":"34a41bbc02247c1a7175ecc32e7a3cabb6ef9f3bf018db2a26eba3d090b70de9"}
Nov 01 03:28:41.090848 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.088585    4930 generic.go:356] "Generic (PLEG): container finished" podID="fc508d65-2f4b-4cf1-aec1-8d229baef624" containerID="12df7a95ac96d8046f62b0152f8172986d41576d330ef368406262d186b53926" exitCode=0
Nov 01 03:28:41.090848 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.089789    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx"
Nov 01 03:28:41.090848 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.090320    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx" event={"ID":"fc508d65-2f4b-4cf1-aec1-8d229baef624","Type":"ContainerDied","Data":"12df7a95ac96d8046f62b0152f8172986d41576d330ef368406262d186b53926"}
Nov 01 03:28:41.090848 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.090396    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx" event={"ID":"fc508d65-2f4b-4cf1-aec1-8d229baef624","Type":"ContainerDied","Data":"5d84ec1cbc8fcc73a9e88a2ad8bcd8041110d6f85eef08b441b1102707f3716a"}
Nov 01 03:28:41.096601 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.096532    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k"
Nov 01 03:28:41.107197 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.107178    4930 scope.go:122] "RemoveContainer" containerID="a77bf021a96e4721e04519717998f814ff214ba344930518070f2ec898d1903a"
Nov 01 03:28:41.107656 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:28:41.107631    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a77bf021a96e4721e04519717998f814ff214ba344930518070f2ec898d1903a\": container with ID starting with a77bf021a96e4721e04519717998f814ff214ba344930518070f2ec898d1903a not found: ID does not exist" containerID="a77bf021a96e4721e04519717998f814ff214ba344930518070f2ec898d1903a"
Nov 01 03:28:41.107745 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.107662    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a77bf021a96e4721e04519717998f814ff214ba344930518070f2ec898d1903a"} err="failed to get container status \"a77bf021a96e4721e04519717998f814ff214ba344930518070f2ec898d1903a\": rpc error: code = NotFound desc = could not find container \"a77bf021a96e4721e04519717998f814ff214ba344930518070f2ec898d1903a\": container with ID starting with a77bf021a96e4721e04519717998f814ff214ba344930518070f2ec898d1903a not found: ID does not exist"
Nov 01 03:28:41.107745 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.107685    4930 scope.go:122] "RemoveContainer" containerID="12df7a95ac96d8046f62b0152f8172986d41576d330ef368406262d186b53926"
Nov 01 03:28:41.120145 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.120127    4930 scope.go:122] "RemoveContainer" containerID="12df7a95ac96d8046f62b0152f8172986d41576d330ef368406262d186b53926"
Nov 01 03:28:41.120422 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:28:41.120404    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12df7a95ac96d8046f62b0152f8172986d41576d330ef368406262d186b53926\": container with ID starting with 12df7a95ac96d8046f62b0152f8172986d41576d330ef368406262d186b53926 not found: ID does not exist" containerID="12df7a95ac96d8046f62b0152f8172986d41576d330ef368406262d186b53926"
Nov 01 03:28:41.120496 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.120427    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12df7a95ac96d8046f62b0152f8172986d41576d330ef368406262d186b53926"} err="failed to get container status \"12df7a95ac96d8046f62b0152f8172986d41576d330ef368406262d186b53926\": rpc error: code = NotFound desc = could not find container \"12df7a95ac96d8046f62b0152f8172986d41576d330ef368406262d186b53926\": container with ID starting with 12df7a95ac96d8046f62b0152f8172986d41576d330ef368406262d186b53926 not found: ID does not exist"
Nov 01 03:28:41.180169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.180136    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85529ed7-e2d9-470f-a897-1711fd4abfa0-credential-socket" pod "85529ed7-e2d9-470f-a897-1711fd4abfa0" (UID: "85529ed7-e2d9-470f-a897-1711fd4abfa0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:28:41.180289 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.180204    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85529ed7-e2d9-470f-a897-1711fd4abfa0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85529ed7-e2d9-470f-a897-1711fd4abfa0-credential-socket\") pod \"85529ed7-e2d9-470f-a897-1711fd4abfa0\" (UID: \"85529ed7-e2d9-470f-a897-1711fd4abfa0\") "
Nov 01 03:28:41.180358 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.180283    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85529ed7-e2d9-470f-a897-1711fd4abfa0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85529ed7-e2d9-470f-a897-1711fd4abfa0-istiod-ca-cert\") pod \"85529ed7-e2d9-470f-a897-1711fd4abfa0\" (UID: \"85529ed7-e2d9-470f-a897-1711fd4abfa0\") "
Nov 01 03:28:41.180791 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.180765    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85529ed7-e2d9-470f-a897-1711fd4abfa0-istiod-ca-cert" pod "85529ed7-e2d9-470f-a897-1711fd4abfa0" (UID: "85529ed7-e2d9-470f-a897-1711fd4abfa0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:28:41.180946 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.180926    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85529ed7-e2d9-470f-a897-1711fd4abfa0-istio-token\" (UniqueName: \"kubernetes.io/projected/85529ed7-e2d9-470f-a897-1711fd4abfa0-istio-token\") pod \"85529ed7-e2d9-470f-a897-1711fd4abfa0\" (UID: \"85529ed7-e2d9-470f-a897-1711fd4abfa0\") "
Nov 01 03:28:41.181438 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.181416    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85529ed7-e2d9-470f-a897-1711fd4abfa0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85529ed7-e2d9-470f-a897-1711fd4abfa0-istio-envoy\") pod \"85529ed7-e2d9-470f-a897-1711fd4abfa0\" (UID: \"85529ed7-e2d9-470f-a897-1711fd4abfa0\") "
Nov 01 03:28:41.181575 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.181531    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85529ed7-e2d9-470f-a897-1711fd4abfa0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85529ed7-e2d9-470f-a897-1711fd4abfa0-workload-socket\") pod \"85529ed7-e2d9-470f-a897-1711fd4abfa0\" (UID: \"85529ed7-e2d9-470f-a897-1711fd4abfa0\") "
Nov 01 03:28:41.181879 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.181859    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85529ed7-e2d9-470f-a897-1711fd4abfa0-kube-api-access-zhnzk\" (UniqueName: \"kubernetes.io/projected/85529ed7-e2d9-470f-a897-1711fd4abfa0-kube-api-access-zhnzk\") pod \"85529ed7-e2d9-470f-a897-1711fd4abfa0\" (UID: \"85529ed7-e2d9-470f-a897-1711fd4abfa0\") "
Nov 01 03:28:41.182205 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.182024    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85529ed7-e2d9-470f-a897-1711fd4abfa0-workload-socket" pod "85529ed7-e2d9-470f-a897-1711fd4abfa0" (UID: "85529ed7-e2d9-470f-a897-1711fd4abfa0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:28:41.182351 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.182330    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85529ed7-e2d9-470f-a897-1711fd4abfa0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85529ed7-e2d9-470f-a897-1711fd4abfa0-workload-certs\") pod \"85529ed7-e2d9-470f-a897-1711fd4abfa0\" (UID: \"85529ed7-e2d9-470f-a897-1711fd4abfa0\") "
Nov 01 03:28:41.182566 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.182530    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85529ed7-e2d9-470f-a897-1711fd4abfa0-workload-certs" pod "85529ed7-e2d9-470f-a897-1711fd4abfa0" (UID: "85529ed7-e2d9-470f-a897-1711fd4abfa0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:28:41.182724 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.182705    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85529ed7-e2d9-470f-a897-1711fd4abfa0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85529ed7-e2d9-470f-a897-1711fd4abfa0-istio-podinfo\") pod \"85529ed7-e2d9-470f-a897-1711fd4abfa0\" (UID: \"85529ed7-e2d9-470f-a897-1711fd4abfa0\") "
Nov 01 03:28:41.182982 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.182964    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85529ed7-e2d9-470f-a897-1711fd4abfa0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85529ed7-e2d9-470f-a897-1711fd4abfa0-istio-data\") pod \"85529ed7-e2d9-470f-a897-1711fd4abfa0\" (UID: \"85529ed7-e2d9-470f-a897-1711fd4abfa0\") "
Nov 01 03:28:41.183266 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.183232    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85529ed7-e2d9-470f-a897-1711fd4abfa0-istio-data" pod "85529ed7-e2d9-470f-a897-1711fd4abfa0" (UID: "85529ed7-e2d9-470f-a897-1711fd4abfa0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:28:41.184546 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.184252    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85529ed7-e2d9-470f-a897-1711fd4abfa0-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:41.184546 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.184293    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85529ed7-e2d9-470f-a897-1711fd4abfa0-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:41.184546 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.184310    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e325fcbf-d504-4923-b0fd-809005447f3e-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:41.184546 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.184324    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:41.184546 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.184341    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:41.184546 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.184368    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85529ed7-e2d9-470f-a897-1711fd4abfa0-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:41.184546 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.184383    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85529ed7-e2d9-470f-a897-1711fd4abfa0-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:41.184546 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.184397    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:41.184546 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.184414    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltsb7\" (UniqueName: \"kubernetes.io/projected/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0-kube-api-access-ltsb7\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:41.184546 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.184429    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85529ed7-e2d9-470f-a897-1711fd4abfa0-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:41.186056 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.185954    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85529ed7-e2d9-470f-a897-1711fd4abfa0-kube-api-access-zhnzk" pod "85529ed7-e2d9-470f-a897-1711fd4abfa0" (UID: "85529ed7-e2d9-470f-a897-1711fd4abfa0"). InnerVolumeSpecName "kube-api-access-zhnzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:28:41.186056 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.186025    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85529ed7-e2d9-470f-a897-1711fd4abfa0-istio-envoy" pod "85529ed7-e2d9-470f-a897-1711fd4abfa0" (UID: "85529ed7-e2d9-470f-a897-1711fd4abfa0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:28:41.187402 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.187369    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85529ed7-e2d9-470f-a897-1711fd4abfa0-istio-token" pod "85529ed7-e2d9-470f-a897-1711fd4abfa0" (UID: "85529ed7-e2d9-470f-a897-1711fd4abfa0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:28:41.187491 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.187450    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85529ed7-e2d9-470f-a897-1711fd4abfa0-istio-podinfo" pod "85529ed7-e2d9-470f-a897-1711fd4abfa0" (UID: "85529ed7-e2d9-470f-a897-1711fd4abfa0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:28:41.307236 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.307123    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85529ed7-e2d9-470f-a897-1711fd4abfa0-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:41.307236 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.307166    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhnzk\" (UniqueName: \"kubernetes.io/projected/85529ed7-e2d9-470f-a897-1711fd4abfa0-kube-api-access-zhnzk\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:41.307236 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.307186    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85529ed7-e2d9-470f-a897-1711fd4abfa0-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:41.307236 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.307199    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85529ed7-e2d9-470f-a897-1711fd4abfa0-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:41.325988 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.324622    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:41.341966 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.331821    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts"]
Nov 01 03:28:41.342625 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.342535    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-c29ts"]
Nov 01 03:28:41.350228 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:28:41.349955    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod540c82b7_8782_49b8_86a4_576e10e21357.slice/crio-f4d4e199529c45632ed9a014cf723dadf6d9f6a8b8e47d0895477c74bc8a8874 WatchSource:0}: Error finding container f4d4e199529c45632ed9a014cf723dadf6d9f6a8b8e47d0895477c74bc8a8874: Status 404 returned error can't find the container with id f4d4e199529c45632ed9a014cf723dadf6d9f6a8b8e47d0895477c74bc8a8874
Nov 01 03:28:41.350228 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.350171    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a07782b1-wjp8r"]
Nov 01 03:28:41.582911 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.582774    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:41.595254 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.595222    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx"]
Nov 01 03:28:41.603452 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.603424    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-f4djx"]
Nov 01 03:28:41.622470 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.622443    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ba1754d-0939-4b6a-b6d0-69155eb4cfa0" path="/var/lib/kubelet/pods/4ba1754d-0939-4b6a-b6d0-69155eb4cfa0/volumes"
Nov 01 03:28:41.622982 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.622964    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc508d65-2f4b-4cf1-aec1-8d229baef624" path="/var/lib/kubelet/pods/fc508d65-2f4b-4cf1-aec1-8d229baef624/volumes"
Nov 01 03:28:41.711511 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.711477    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7"]
Nov 01 03:28:41.716129 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:41.716108    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-jmwc7"]
Nov 01 03:28:42.100167 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:42.100137    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k"
Nov 01 03:28:42.100167 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:42.100155    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k" event={"ID":"85529ed7-e2d9-470f-a897-1711fd4abfa0","Type":"ContainerDied","Data":"fb296b8c820b2b92da89c3db86b0c01eb0f77914a763b71d1970f5fc6894431e"}
Nov 01 03:28:42.100410 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:42.100209    4930 scope.go:122] "RemoveContainer" containerID="9018f3823ac66b77bb260a122a93c861dad45b752d4a6b6c361e24e975bbcc51"
Nov 01 03:28:42.102140 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:42.102112    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07782b1-wjp8r" event={"ID":"540c82b7-8782-49b8-86a4-576e10e21357","Type":"ContainerStarted","Data":"931bac320b3dc51128321b2345535c08786c3efeeac9cae74efa2e481a242ae4"}
Nov 01 03:28:42.102263 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:42.102162    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07782b1-wjp8r" event={"ID":"540c82b7-8782-49b8-86a4-576e10e21357","Type":"ContainerStarted","Data":"f4d4e199529c45632ed9a014cf723dadf6d9f6a8b8e47d0895477c74bc8a8874"}
Nov 01 03:28:42.105069 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:42.105046    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-367a7f78-w4t5v" event={"ID":"db76ac41-529c-4927-b1fc-b029877dd3fc","Type":"ContainerStarted","Data":"8e7db56a5440b1f328b63460da48e4ac3a0ac853b07292ecd5e53e1f0fa3f5aa"}
Nov 01 03:28:42.268994 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:42.268964    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k"]
Nov 01 03:28:42.273639 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:42.273614    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xpl7k"]
Nov 01 03:28:42.395150 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:42.395057    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-367a7f78-w4t5v" podStartSLOduration=3.395042382 podStartE2EDuration="3.395042382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:28:42.391809307 +0000 UTC m=+1326.914815895" watchObservedRunningTime="2026-11-01 03:28:42.395042382 +0000 UTC m=+1326.918048969"
Nov 01 03:28:42.512108 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:42.512057    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07782b1-wjp8r" podStartSLOduration=2.5120435 podStartE2EDuration="2.5120435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:28:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:28:42.509235388 +0000 UTC m=+1327.032241981" watchObservedRunningTime="2026-11-01 03:28:42.5120435 +0000 UTC m=+1327.035050087"
Nov 01 03:28:42.578677 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:42.578644    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:42.748648 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:42.748529    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj"]
Nov 01 03:28:42.749230 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:42.749212    4930 cpu_manager.go:414] "containerMap: removing container" podUID="85529ed7-e2d9-470f-a897-1711fd4abfa0" containerName="istio-proxy"
Nov 01 03:28:42.749230 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:42.749229    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85529ed7-e2d9-470f-a897-1711fd4abfa0" containerName="istio-proxy"
Nov 01 03:28:42.749382 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:42.749265    4930 cpu_manager.go:414] "containerMap: removing container" podUID="4ba1754d-0939-4b6a-b6d0-69155eb4cfa0" containerName="istio-proxy"
Nov 01 03:28:42.749382 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:42.749270    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ba1754d-0939-4b6a-b6d0-69155eb4cfa0" containerName="istio-proxy"
Nov 01 03:28:42.749382 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:42.749288    4930 cpu_manager.go:414] "containerMap: removing container" podUID="e325fcbf-d504-4923-b0fd-809005447f3e" containerName="istio-proxy"
Nov 01 03:28:42.749382 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:42.749293    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e325fcbf-d504-4923-b0fd-809005447f3e" containerName="istio-proxy"
Nov 01 03:28:42.749382 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:42.749306    4930 cpu_manager.go:414] "containerMap: removing container" podUID="fc508d65-2f4b-4cf1-aec1-8d229baef624" containerName="istio-proxy"
Nov 01 03:28:42.749382 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:42.749311    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc508d65-2f4b-4cf1-aec1-8d229baef624" containerName="istio-proxy"
Nov 01 03:28:42.749713 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:42.749409    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc508d65-2f4b-4cf1-aec1-8d229baef624" containerName="istio-proxy"
Nov 01 03:28:42.749713 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:42.749421    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="85529ed7-e2d9-470f-a897-1711fd4abfa0" containerName="istio-proxy"
Nov 01 03:28:42.749713 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:42.749439    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ba1754d-0939-4b6a-b6d0-69155eb4cfa0" containerName="istio-proxy"
Nov 01 03:28:42.749713 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:42.749450    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="e325fcbf-d504-4923-b0fd-809005447f3e" containerName="istio-proxy"
Nov 01 03:28:42.753463 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:42.753440    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj"
Nov 01 03:28:42.834199 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:42.834169    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj"]
Nov 01 03:28:42.940908 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:42.940875    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9a70ec2-21fd-40ae-8d18-025a67248fce-istio-token\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj\" (UID: \"e9a70ec2-21fd-40ae-8d18-025a67248fce\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj"
Nov 01 03:28:42.940908 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:42.940915    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9a70ec2-21fd-40ae-8d18-025a67248fce-istio-envoy\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj\" (UID: \"e9a70ec2-21fd-40ae-8d18-025a67248fce\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj"
Nov 01 03:28:42.941128 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:42.940939    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9a70ec2-21fd-40ae-8d18-025a67248fce-istio-podinfo\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj\" (UID: \"e9a70ec2-21fd-40ae-8d18-025a67248fce\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj"
Nov 01 03:28:42.941128 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:42.941007    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9a70ec2-21fd-40ae-8d18-025a67248fce-workload-certs\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj\" (UID: \"e9a70ec2-21fd-40ae-8d18-025a67248fce\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj"
Nov 01 03:28:42.941128 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:42.941030    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9a70ec2-21fd-40ae-8d18-025a67248fce-istiod-ca-cert\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj\" (UID: \"e9a70ec2-21fd-40ae-8d18-025a67248fce\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj"
Nov 01 03:28:42.941292 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:42.941145    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a70ec2-21fd-40ae-8d18-025a67248fce-workload-socket\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj\" (UID: \"e9a70ec2-21fd-40ae-8d18-025a67248fce\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj"
Nov 01 03:28:42.941292 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:42.941203    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9a70ec2-21fd-40ae-8d18-025a67248fce-istio-data\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj\" (UID: \"e9a70ec2-21fd-40ae-8d18-025a67248fce\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj"
Nov 01 03:28:42.941455 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:42.941436    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a70ec2-21fd-40ae-8d18-025a67248fce-credential-socket\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj\" (UID: \"e9a70ec2-21fd-40ae-8d18-025a67248fce\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj"
Nov 01 03:28:42.941516 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:42.941485    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59tv7\" (UniqueName: \"kubernetes.io/projected/e9a70ec2-21fd-40ae-8d18-025a67248fce-kube-api-access-59tv7\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj\" (UID: \"e9a70ec2-21fd-40ae-8d18-025a67248fce\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj"
Nov 01 03:28:43.043088 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.043019    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9a70ec2-21fd-40ae-8d18-025a67248fce-istio-envoy\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj\" (UID: \"e9a70ec2-21fd-40ae-8d18-025a67248fce\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj"
Nov 01 03:28:43.043088 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.043051    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9a70ec2-21fd-40ae-8d18-025a67248fce-istio-podinfo\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj\" (UID: \"e9a70ec2-21fd-40ae-8d18-025a67248fce\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj"
Nov 01 03:28:43.043088 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.043071    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9a70ec2-21fd-40ae-8d18-025a67248fce-workload-certs\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj\" (UID: \"e9a70ec2-21fd-40ae-8d18-025a67248fce\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj"
Nov 01 03:28:43.043367 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.043199    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9a70ec2-21fd-40ae-8d18-025a67248fce-istiod-ca-cert\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj\" (UID: \"e9a70ec2-21fd-40ae-8d18-025a67248fce\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj"
Nov 01 03:28:43.043367 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.043309    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a70ec2-21fd-40ae-8d18-025a67248fce-workload-socket\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj\" (UID: \"e9a70ec2-21fd-40ae-8d18-025a67248fce\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj"
Nov 01 03:28:43.043367 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.043354    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9a70ec2-21fd-40ae-8d18-025a67248fce-workload-certs\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj\" (UID: \"e9a70ec2-21fd-40ae-8d18-025a67248fce\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj"
Nov 01 03:28:43.043515 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.043370    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9a70ec2-21fd-40ae-8d18-025a67248fce-istio-data\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj\" (UID: \"e9a70ec2-21fd-40ae-8d18-025a67248fce\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj"
Nov 01 03:28:43.043616 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.043600    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a70ec2-21fd-40ae-8d18-025a67248fce-credential-socket\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj\" (UID: \"e9a70ec2-21fd-40ae-8d18-025a67248fce\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj"
Nov 01 03:28:43.043674 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.043629    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9a70ec2-21fd-40ae-8d18-025a67248fce-istio-data\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj\" (UID: \"e9a70ec2-21fd-40ae-8d18-025a67248fce\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj"
Nov 01 03:28:43.043674 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.043652    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59tv7\" (UniqueName: \"kubernetes.io/projected/e9a70ec2-21fd-40ae-8d18-025a67248fce-kube-api-access-59tv7\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj\" (UID: \"e9a70ec2-21fd-40ae-8d18-025a67248fce\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj"
Nov 01 03:28:43.043777 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.043740    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a70ec2-21fd-40ae-8d18-025a67248fce-workload-socket\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj\" (UID: \"e9a70ec2-21fd-40ae-8d18-025a67248fce\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj"
Nov 01 03:28:43.043833 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.043799    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9a70ec2-21fd-40ae-8d18-025a67248fce-istio-token\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj\" (UID: \"e9a70ec2-21fd-40ae-8d18-025a67248fce\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj"
Nov 01 03:28:43.043907 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.043883    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a70ec2-21fd-40ae-8d18-025a67248fce-credential-socket\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj\" (UID: \"e9a70ec2-21fd-40ae-8d18-025a67248fce\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj"
Nov 01 03:28:43.044086 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.043925    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9a70ec2-21fd-40ae-8d18-025a67248fce-istiod-ca-cert\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj\" (UID: \"e9a70ec2-21fd-40ae-8d18-025a67248fce\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj"
Nov 01 03:28:43.045487 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.045470    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9a70ec2-21fd-40ae-8d18-025a67248fce-istio-envoy\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj\" (UID: \"e9a70ec2-21fd-40ae-8d18-025a67248fce\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj"
Nov 01 03:28:43.045732 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.045712    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9a70ec2-21fd-40ae-8d18-025a67248fce-istio-podinfo\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj\" (UID: \"e9a70ec2-21fd-40ae-8d18-025a67248fce\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj"
Nov 01 03:28:43.110886 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.110858    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59tv7\" (UniqueName: \"kubernetes.io/projected/e9a70ec2-21fd-40ae-8d18-025a67248fce-kube-api-access-59tv7\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj\" (UID: \"e9a70ec2-21fd-40ae-8d18-025a67248fce\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj"
Nov 01 03:28:43.113087 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.113059    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9a70ec2-21fd-40ae-8d18-025a67248fce-istio-token\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj\" (UID: \"e9a70ec2-21fd-40ae-8d18-025a67248fce\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj"
Nov 01 03:28:43.118685 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.118665    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvgwp"]
Nov 01 03:28:43.121094 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.121068    4930 generic.go:356] "Generic (PLEG): container finished" podID="540c82b7-8782-49b8-86a4-576e10e21357" containerID="931bac320b3dc51128321b2345535c08786c3efeeac9cae74efa2e481a242ae4" exitCode=0
Nov 01 03:28:43.123031 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.123007    4930 generic.go:356] "Generic (PLEG): container finished" podID="db76ac41-529c-4927-b1fc-b029877dd3fc" containerID="8e7db56a5440b1f328b63460da48e4ac3a0ac853b07292ecd5e53e1f0fa3f5aa" exitCode=0
Nov 01 03:28:43.124473 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.124449    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07782b1-wjp8r" event={"ID":"540c82b7-8782-49b8-86a4-576e10e21357","Type":"ContainerDied","Data":"931bac320b3dc51128321b2345535c08786c3efeeac9cae74efa2e481a242ae4"}
Nov 01 03:28:43.124597 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.124482    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-367a7f78-w4t5v" event={"ID":"db76ac41-529c-4927-b1fc-b029877dd3fc","Type":"ContainerDied","Data":"8e7db56a5440b1f328b63460da48e4ac3a0ac853b07292ecd5e53e1f0fa3f5aa"}
Nov 01 03:28:43.124597 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.124551    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvgwp"
Nov 01 03:28:43.185690 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.185659    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 01 03:28:43.185690 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.185660    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 01 03:28:43.185847 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.185669    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 01 03:28:43.185847 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.185700    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-8w9bn\""
Nov 01 03:28:43.187799 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.187780    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvgwp"]
Nov 01 03:28:43.190974 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.190943    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:43.247222 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.247194    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bvgwp\" (UID: \"c7161bfc-1b3c-4245-a0e7-c35a7fe69be1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvgwp"
Nov 01 03:28:43.247354 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.247235    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bvgwp\" (UID: \"c7161bfc-1b3c-4245-a0e7-c35a7fe69be1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvgwp"
Nov 01 03:28:43.247354 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.247314    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bvgwp\" (UID: \"c7161bfc-1b3c-4245-a0e7-c35a7fe69be1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvgwp"
Nov 01 03:28:43.247449 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.247411    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bvgwp\" (UID: \"c7161bfc-1b3c-4245-a0e7-c35a7fe69be1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvgwp"
Nov 01 03:28:43.247567 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.247537    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7j7l\" (UniqueName: \"kubernetes.io/projected/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-kube-api-access-m7j7l\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bvgwp\" (UID: \"c7161bfc-1b3c-4245-a0e7-c35a7fe69be1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvgwp"
Nov 01 03:28:43.247703 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.247683    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bvgwp\" (UID: \"c7161bfc-1b3c-4245-a0e7-c35a7fe69be1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvgwp"
Nov 01 03:28:43.247807 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.247790    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bvgwp\" (UID: \"c7161bfc-1b3c-4245-a0e7-c35a7fe69be1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvgwp"
Nov 01 03:28:43.349049 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.349024    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bvgwp\" (UID: \"c7161bfc-1b3c-4245-a0e7-c35a7fe69be1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvgwp"
Nov 01 03:28:43.349168 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.349073    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bvgwp\" (UID: \"c7161bfc-1b3c-4245-a0e7-c35a7fe69be1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvgwp"
Nov 01 03:28:43.349168 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.349112    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bvgwp\" (UID: \"c7161bfc-1b3c-4245-a0e7-c35a7fe69be1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvgwp"
Nov 01 03:28:43.349262 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.349200    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7j7l\" (UniqueName: \"kubernetes.io/projected/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-kube-api-access-m7j7l\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bvgwp\" (UID: \"c7161bfc-1b3c-4245-a0e7-c35a7fe69be1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvgwp"
Nov 01 03:28:43.349262 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.349250    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bvgwp\" (UID: \"c7161bfc-1b3c-4245-a0e7-c35a7fe69be1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvgwp"
Nov 01 03:28:43.349354 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.349293    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bvgwp\" (UID: \"c7161bfc-1b3c-4245-a0e7-c35a7fe69be1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvgwp"
Nov 01 03:28:43.349354 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.349333    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bvgwp\" (UID: \"c7161bfc-1b3c-4245-a0e7-c35a7fe69be1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvgwp"
Nov 01 03:28:43.350025 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.349994    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bvgwp\" (UID: \"c7161bfc-1b3c-4245-a0e7-c35a7fe69be1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvgwp"
Nov 01 03:28:43.351649 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.351627    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bvgwp\" (UID: \"c7161bfc-1b3c-4245-a0e7-c35a7fe69be1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvgwp"
Nov 01 03:28:43.351744 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.351676    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bvgwp\" (UID: \"c7161bfc-1b3c-4245-a0e7-c35a7fe69be1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvgwp"
Nov 01 03:28:43.351822 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.351761    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bvgwp\" (UID: \"c7161bfc-1b3c-4245-a0e7-c35a7fe69be1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvgwp"
Nov 01 03:28:43.351878 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.351846    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bvgwp\" (UID: \"c7161bfc-1b3c-4245-a0e7-c35a7fe69be1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvgwp"
Nov 01 03:28:43.368137 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.368117    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj"
Nov 01 03:28:43.412282 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.412245    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bvgwp\" (UID: \"c7161bfc-1b3c-4245-a0e7-c35a7fe69be1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvgwp"
Nov 01 03:28:43.418441 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.417451    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7j7l\" (UniqueName: \"kubernetes.io/projected/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-kube-api-access-m7j7l\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bvgwp\" (UID: \"c7161bfc-1b3c-4245-a0e7-c35a7fe69be1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvgwp"
Nov 01 03:28:43.443976 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.443950    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4"]
Nov 01 03:28:43.446920 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.446896    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvgwp"
Nov 01 03:28:43.451799 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.451778    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4"
Nov 01 03:28:43.521312 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.521260    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4"]
Nov 01 03:28:43.556167 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.552628    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53f53373-f6f8-43db-a686-ab971adec842-istio-token\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4\" (UID: \"53f53373-f6f8-43db-a686-ab971adec842\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4"
Nov 01 03:28:43.556167 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.552725    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f53373-f6f8-43db-a686-ab971adec842-workload-socket\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4\" (UID: \"53f53373-f6f8-43db-a686-ab971adec842\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4"
Nov 01 03:28:43.556167 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.552887    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53f53373-f6f8-43db-a686-ab971adec842-istio-podinfo\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4\" (UID: \"53f53373-f6f8-43db-a686-ab971adec842\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4"
Nov 01 03:28:43.556167 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.553025    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53f53373-f6f8-43db-a686-ab971adec842-istio-envoy\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4\" (UID: \"53f53373-f6f8-43db-a686-ab971adec842\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4"
Nov 01 03:28:43.560137 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.553058    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f53373-f6f8-43db-a686-ab971adec842-credential-socket\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4\" (UID: \"53f53373-f6f8-43db-a686-ab971adec842\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4"
Nov 01 03:28:43.560137 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.558591    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53f53373-f6f8-43db-a686-ab971adec842-istiod-ca-cert\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4\" (UID: \"53f53373-f6f8-43db-a686-ab971adec842\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4"
Nov 01 03:28:43.560137 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.558685    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m52bc\" (UniqueName: \"kubernetes.io/projected/53f53373-f6f8-43db-a686-ab971adec842-kube-api-access-m52bc\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4\" (UID: \"53f53373-f6f8-43db-a686-ab971adec842\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4"
Nov 01 03:28:43.560137 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.558716    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53f53373-f6f8-43db-a686-ab971adec842-istio-data\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4\" (UID: \"53f53373-f6f8-43db-a686-ab971adec842\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4"
Nov 01 03:28:43.560137 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.558796    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53f53373-f6f8-43db-a686-ab971adec842-workload-certs\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4\" (UID: \"53f53373-f6f8-43db-a686-ab971adec842\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4"
Nov 01 03:28:43.625578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.625483    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85529ed7-e2d9-470f-a897-1711fd4abfa0" path="/var/lib/kubelet/pods/85529ed7-e2d9-470f-a897-1711fd4abfa0/volumes"
Nov 01 03:28:43.626102 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.626085    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e325fcbf-d504-4923-b0fd-809005447f3e" path="/var/lib/kubelet/pods/e325fcbf-d504-4923-b0fd-809005447f3e/volumes"
Nov 01 03:28:43.660718 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.660675    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53f53373-f6f8-43db-a686-ab971adec842-istio-token\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4\" (UID: \"53f53373-f6f8-43db-a686-ab971adec842\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4"
Nov 01 03:28:43.660718 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.660715    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f53373-f6f8-43db-a686-ab971adec842-workload-socket\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4\" (UID: \"53f53373-f6f8-43db-a686-ab971adec842\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4"
Nov 01 03:28:43.660966 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.660763    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53f53373-f6f8-43db-a686-ab971adec842-istio-podinfo\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4\" (UID: \"53f53373-f6f8-43db-a686-ab971adec842\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4"
Nov 01 03:28:43.660966 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.660844    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53f53373-f6f8-43db-a686-ab971adec842-istio-envoy\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4\" (UID: \"53f53373-f6f8-43db-a686-ab971adec842\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4"
Nov 01 03:28:43.660966 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.660873    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f53373-f6f8-43db-a686-ab971adec842-credential-socket\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4\" (UID: \"53f53373-f6f8-43db-a686-ab971adec842\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4"
Nov 01 03:28:43.660966 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.660926    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53f53373-f6f8-43db-a686-ab971adec842-istiod-ca-cert\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4\" (UID: \"53f53373-f6f8-43db-a686-ab971adec842\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4"
Nov 01 03:28:43.661181 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.660981    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m52bc\" (UniqueName: \"kubernetes.io/projected/53f53373-f6f8-43db-a686-ab971adec842-kube-api-access-m52bc\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4\" (UID: \"53f53373-f6f8-43db-a686-ab971adec842\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4"
Nov 01 03:28:43.661245 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.661174    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53f53373-f6f8-43db-a686-ab971adec842-istio-data\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4\" (UID: \"53f53373-f6f8-43db-a686-ab971adec842\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4"
Nov 01 03:28:43.661245 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.661208    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f53373-f6f8-43db-a686-ab971adec842-workload-socket\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4\" (UID: \"53f53373-f6f8-43db-a686-ab971adec842\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4"
Nov 01 03:28:43.661245 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.661233    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53f53373-f6f8-43db-a686-ab971adec842-workload-certs\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4\" (UID: \"53f53373-f6f8-43db-a686-ab971adec842\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4"
Nov 01 03:28:43.661516 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.661494    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53f53373-f6f8-43db-a686-ab971adec842-istio-data\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4\" (UID: \"53f53373-f6f8-43db-a686-ab971adec842\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4"
Nov 01 03:28:43.661694 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.661672    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f53373-f6f8-43db-a686-ab971adec842-credential-socket\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4\" (UID: \"53f53373-f6f8-43db-a686-ab971adec842\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4"
Nov 01 03:28:43.661844 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.661821    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53f53373-f6f8-43db-a686-ab971adec842-workload-certs\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4\" (UID: \"53f53373-f6f8-43db-a686-ab971adec842\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4"
Nov 01 03:28:43.661914 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.661859    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53f53373-f6f8-43db-a686-ab971adec842-istiod-ca-cert\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4\" (UID: \"53f53373-f6f8-43db-a686-ab971adec842\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4"
Nov 01 03:28:43.663698 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.663668    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53f53373-f6f8-43db-a686-ab971adec842-istio-envoy\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4\" (UID: \"53f53373-f6f8-43db-a686-ab971adec842\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4"
Nov 01 03:28:43.663927 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.663909    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53f53373-f6f8-43db-a686-ab971adec842-istio-podinfo\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4\" (UID: \"53f53373-f6f8-43db-a686-ab971adec842\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4"
Nov 01 03:28:43.723675 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.723649    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m52bc\" (UniqueName: \"kubernetes.io/projected/53f53373-f6f8-43db-a686-ab971adec842-kube-api-access-m52bc\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4\" (UID: \"53f53373-f6f8-43db-a686-ab971adec842\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4"
Nov 01 03:28:43.724550 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.724517    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53f53373-f6f8-43db-a686-ab971adec842-istio-token\") pod \"e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4\" (UID: \"53f53373-f6f8-43db-a686-ab971adec842\") " pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4"
Nov 01 03:28:43.773424 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.773398    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4"
Nov 01 03:28:43.823690 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.823660    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj"]
Nov 01 03:28:43.829092 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.829056    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:28:43.829190 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.829141    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:28:43.829190 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.829180    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:28:43.880119 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.880092    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvgwp"]
Nov 01 03:28:43.896355 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.896333    4930 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 03:28:43.965606 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.965564    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27c1b5ea-qtg9h"]
Nov 01 03:28:43.974551 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:43.974522    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c1b5ea-qtg9h"
Nov 01 03:28:44.030497 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.030465    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27c1b5ea-qtg9h"]
Nov 01 03:28:44.068087 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.068052    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/864d195a-d8a1-41e3-bcdf-fd2acefa7f8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27c1b5ea-qtg9h\" (UID: \"864d195a-d8a1-41e3-bcdf-fd2acefa7f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c1b5ea-qtg9h"
Nov 01 03:28:44.068087 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.068090    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/864d195a-d8a1-41e3-bcdf-fd2acefa7f8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27c1b5ea-qtg9h\" (UID: \"864d195a-d8a1-41e3-bcdf-fd2acefa7f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c1b5ea-qtg9h"
Nov 01 03:28:44.068338 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.068280    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq4fg\" (UniqueName: \"kubernetes.io/projected/864d195a-d8a1-41e3-bcdf-fd2acefa7f8b-kube-api-access-jq4fg\") pod \"maas-api-revoke-sub-27c1b5ea-qtg9h\" (UID: \"864d195a-d8a1-41e3-bcdf-fd2acefa7f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c1b5ea-qtg9h"
Nov 01 03:28:44.135036 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.134925    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj" event={"ID":"e9a70ec2-21fd-40ae-8d18-025a67248fce","Type":"ContainerStarted","Data":"477b9b9d8d539bd2cf241b2ffd19624dc8615362041762cf71cdf74597b51bf9"}
Nov 01 03:28:44.135036 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.134992    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj" event={"ID":"e9a70ec2-21fd-40ae-8d18-025a67248fce","Type":"ContainerStarted","Data":"92374760c787bee288de1b1132abb43ba653dd3de0c90d033c5eb17232df8448"}
Nov 01 03:28:44.137111 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.137084    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvgwp" event={"ID":"c7161bfc-1b3c-4245-a0e7-c35a7fe69be1","Type":"ContainerStarted","Data":"f25bed095bd4c46595b895841f8fa4e871aeb930982e01f224cd848975af3efd"}
Nov 01 03:28:44.170611 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:28:44.169576    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53f53373_f6f8_43db_a686_ab971adec842.slice/crio-e58fd47f953d2bce3a5099092c73a2e880c03b43e379fadcd640ae25fcdaf5ac WatchSource:0}: Error finding container e58fd47f953d2bce3a5099092c73a2e880c03b43e379fadcd640ae25fcdaf5ac: Status 404 returned error can't find the container with id e58fd47f953d2bce3a5099092c73a2e880c03b43e379fadcd640ae25fcdaf5ac
Nov 01 03:28:44.170611 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.169745    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq4fg\" (UniqueName: \"kubernetes.io/projected/864d195a-d8a1-41e3-bcdf-fd2acefa7f8b-kube-api-access-jq4fg\") pod \"maas-api-revoke-sub-27c1b5ea-qtg9h\" (UID: \"864d195a-d8a1-41e3-bcdf-fd2acefa7f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c1b5ea-qtg9h"
Nov 01 03:28:44.170611 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.170015    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/864d195a-d8a1-41e3-bcdf-fd2acefa7f8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27c1b5ea-qtg9h\" (UID: \"864d195a-d8a1-41e3-bcdf-fd2acefa7f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c1b5ea-qtg9h"
Nov 01 03:28:44.170611 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.170058    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/864d195a-d8a1-41e3-bcdf-fd2acefa7f8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27c1b5ea-qtg9h\" (UID: \"864d195a-d8a1-41e3-bcdf-fd2acefa7f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c1b5ea-qtg9h"
Nov 01 03:28:44.172611 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.171076    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/864d195a-d8a1-41e3-bcdf-fd2acefa7f8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27c1b5ea-qtg9h\" (UID: \"864d195a-d8a1-41e3-bcdf-fd2acefa7f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c1b5ea-qtg9h"
Nov 01 03:28:44.172611 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.172444    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4"]
Nov 01 03:28:44.176355 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.176323    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:28:44.176469 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.176388    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:28:44.176469 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.176420    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:28:44.179599 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.179260    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/864d195a-d8a1-41e3-bcdf-fd2acefa7f8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27c1b5ea-qtg9h\" (UID: \"864d195a-d8a1-41e3-bcdf-fd2acefa7f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c1b5ea-qtg9h"
Nov 01 03:28:44.201002 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.200854    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:44.238148 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.238119    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq4fg\" (UniqueName: \"kubernetes.io/projected/864d195a-d8a1-41e3-bcdf-fd2acefa7f8b-kube-api-access-jq4fg\") pod \"maas-api-revoke-sub-27c1b5ea-qtg9h\" (UID: \"864d195a-d8a1-41e3-bcdf-fd2acefa7f8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c1b5ea-qtg9h"
Nov 01 03:28:44.274209 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.274129    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj" podStartSLOduration=2.274109127 podStartE2EDuration="2.274109127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:28:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:28:44.269283012 +0000 UTC m=+1328.792289602" watchObservedRunningTime="2026-11-01 03:28:44.274109127 +0000 UTC m=+1328.797115717"
Nov 01 03:28:44.290207 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.289991    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c1b5ea-qtg9h"
Nov 01 03:28:44.335284 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.334674    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-367a7f78-w4t5v"
Nov 01 03:28:44.381095 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.379695    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db76ac41-529c-4927-b1fc-b029877dd3fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db76ac41-529c-4927-b1fc-b029877dd3fc-maas-api-xks-ca\") pod \"db76ac41-529c-4927-b1fc-b029877dd3fc\" (UID: \"db76ac41-529c-4927-b1fc-b029877dd3fc\") "
Nov 01 03:28:44.381095 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.379902    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db76ac41-529c-4927-b1fc-b029877dd3fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db76ac41-529c-4927-b1fc-b029877dd3fc-maas-api-openshift-ca\") pod \"db76ac41-529c-4927-b1fc-b029877dd3fc\" (UID: \"db76ac41-529c-4927-b1fc-b029877dd3fc\") "
Nov 01 03:28:44.381095 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.379988    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db76ac41-529c-4927-b1fc-b029877dd3fc-kube-api-access-c4lxk\" (UniqueName: \"kubernetes.io/projected/db76ac41-529c-4927-b1fc-b029877dd3fc-kube-api-access-c4lxk\") pod \"db76ac41-529c-4927-b1fc-b029877dd3fc\" (UID: \"db76ac41-529c-4927-b1fc-b029877dd3fc\") "
Nov 01 03:28:44.381095 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.380454    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:44.381095 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.381061    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj"
Nov 01 03:28:44.381468 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.381231    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db76ac41-529c-4927-b1fc-b029877dd3fc-maas-api-openshift-ca" pod "db76ac41-529c-4927-b1fc-b029877dd3fc" (UID: "db76ac41-529c-4927-b1fc-b029877dd3fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:28:44.393421 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.392791    4930 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 01 03:28:44.393421 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.392829    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj" podUID="e9a70ec2-21fd-40ae-8d18-025a67248fce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 01 03:28:44.417721 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.417678    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db76ac41-529c-4927-b1fc-b029877dd3fc-maas-api-xks-ca" pod "db76ac41-529c-4927-b1fc-b029877dd3fc" (UID: "db76ac41-529c-4927-b1fc-b029877dd3fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:28:44.417721 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.417683    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db76ac41-529c-4927-b1fc-b029877dd3fc-kube-api-access-c4lxk" pod "db76ac41-529c-4927-b1fc-b029877dd3fc" (UID: "db76ac41-529c-4927-b1fc-b029877dd3fc"). InnerVolumeSpecName "kube-api-access-c4lxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:28:44.456622 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.456529    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07782b1-wjp8r"
Nov 01 03:28:44.483861 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.482037    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/540c82b7-8782-49b8-86a4-576e10e21357-kube-api-access-xwsld\" (UniqueName: \"kubernetes.io/projected/540c82b7-8782-49b8-86a4-576e10e21357-kube-api-access-xwsld\") pod \"540c82b7-8782-49b8-86a4-576e10e21357\" (UID: \"540c82b7-8782-49b8-86a4-576e10e21357\") "
Nov 01 03:28:44.483861 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.482403    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/540c82b7-8782-49b8-86a4-576e10e21357-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540c82b7-8782-49b8-86a4-576e10e21357-maas-api-openshift-ca\") pod \"540c82b7-8782-49b8-86a4-576e10e21357\" (UID: \"540c82b7-8782-49b8-86a4-576e10e21357\") "
Nov 01 03:28:44.483861 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.482480    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/540c82b7-8782-49b8-86a4-576e10e21357-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540c82b7-8782-49b8-86a4-576e10e21357-maas-api-xks-ca\") pod \"540c82b7-8782-49b8-86a4-576e10e21357\" (UID: \"540c82b7-8782-49b8-86a4-576e10e21357\") "
Nov 01 03:28:44.483861 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.483095    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db76ac41-529c-4927-b1fc-b029877dd3fc-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:44.483861 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.483118    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db76ac41-529c-4927-b1fc-b029877dd3fc-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:44.483861 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.483134    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4lxk\" (UniqueName: \"kubernetes.io/projected/db76ac41-529c-4927-b1fc-b029877dd3fc-kube-api-access-c4lxk\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:44.487376 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.487243    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/540c82b7-8782-49b8-86a4-576e10e21357-maas-api-openshift-ca" pod "540c82b7-8782-49b8-86a4-576e10e21357" (UID: "540c82b7-8782-49b8-86a4-576e10e21357"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:28:44.488398 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.488363    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/540c82b7-8782-49b8-86a4-576e10e21357-kube-api-access-xwsld" pod "540c82b7-8782-49b8-86a4-576e10e21357" (UID: "540c82b7-8782-49b8-86a4-576e10e21357"). InnerVolumeSpecName "kube-api-access-xwsld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:28:44.521516 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.520444    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/540c82b7-8782-49b8-86a4-576e10e21357-maas-api-xks-ca" pod "540c82b7-8782-49b8-86a4-576e10e21357" (UID: "540c82b7-8782-49b8-86a4-576e10e21357"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:28:44.595911 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.583089    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:44.601852 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.600865    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540c82b7-8782-49b8-86a4-576e10e21357-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:44.601852 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.600899    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540c82b7-8782-49b8-86a4-576e10e21357-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:44.601852 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.600916    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwsld\" (UniqueName: \"kubernetes.io/projected/540c82b7-8782-49b8-86a4-576e10e21357-kube-api-access-xwsld\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:44.719682 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:44.719647    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27c1b5ea-qtg9h"]
Nov 01 03:28:45.154197 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:45.154155    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-367a7f78-w4t5v"
Nov 01 03:28:45.154197 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:45.154165    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-367a7f78-w4t5v" event={"ID":"db76ac41-529c-4927-b1fc-b029877dd3fc","Type":"ContainerDied","Data":"34a41bbc02247c1a7175ecc32e7a3cabb6ef9f3bf018db2a26eba3d090b70de9"}
Nov 01 03:28:45.154417 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:45.154250    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34a41bbc02247c1a7175ecc32e7a3cabb6ef9f3bf018db2a26eba3d090b70de9"
Nov 01 03:28:45.156435 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:45.156404    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4" event={"ID":"53f53373-f6f8-43db-a686-ab971adec842","Type":"ContainerStarted","Data":"7fe2a79cdbde1663d62b3eaced58e2205eecf109b361f7a1313060c2f04a70dc"}
Nov 01 03:28:45.156607 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:45.156458    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4" event={"ID":"53f53373-f6f8-43db-a686-ab971adec842","Type":"ContainerStarted","Data":"e58fd47f953d2bce3a5099092c73a2e880c03b43e379fadcd640ae25fcdaf5ac"}
Nov 01 03:28:45.157972 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:45.157954    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c1b5ea-qtg9h" event={"ID":"864d195a-d8a1-41e3-bcdf-fd2acefa7f8b","Type":"ContainerStarted","Data":"686ed25ff460d0ea4fd03edbc55e4f45996cc6d7df6b3296bb36d147e4dd0532"}
Nov 01 03:28:45.158314 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:45.158295    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c1b5ea-qtg9h" event={"ID":"864d195a-d8a1-41e3-bcdf-fd2acefa7f8b","Type":"ContainerStarted","Data":"0d803e33e982e904bbeaee15ad4f230e4116f3e1501e7c27033991b8c92feaec"}
Nov 01 03:28:45.160371 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:45.160352    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07782b1-wjp8r"
Nov 01 03:28:45.168750 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:45.168722    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a07782b1-wjp8r" event={"ID":"540c82b7-8782-49b8-86a4-576e10e21357","Type":"ContainerDied","Data":"f4d4e199529c45632ed9a014cf723dadf6d9f6a8b8e47d0895477c74bc8a8874"}
Nov 01 03:28:45.168970 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:45.168951    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4d4e199529c45632ed9a014cf723dadf6d9f6a8b8e47d0895477c74bc8a8874"
Nov 01 03:28:45.322866 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:45.322723    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4" podStartSLOduration=2.322703668 podStartE2EDuration="2.322703668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:28:45.316930602 +0000 UTC m=+1329.839937183" watchObservedRunningTime="2026-11-01 03:28:45.322703668 +0000 UTC m=+1329.845710256"
Nov 01 03:28:45.379627 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:45.375936    4930 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 01 03:28:45.379627 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:45.375998    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj" podUID="e9a70ec2-21fd-40ae-8d18-025a67248fce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 01 03:28:45.401582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:45.400811    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d17bc02d-82rld"]
Nov 01 03:28:45.405577 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:45.401870    4930 cpu_manager.go:414] "containerMap: removing container" podUID="540c82b7-8782-49b8-86a4-576e10e21357" containerName="revoke-keys"
Nov 01 03:28:45.405577 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:45.401898    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="540c82b7-8782-49b8-86a4-576e10e21357" containerName="revoke-keys"
Nov 01 03:28:45.405577 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:45.401920    4930 cpu_manager.go:414] "containerMap: removing container" podUID="db76ac41-529c-4927-b1fc-b029877dd3fc" containerName="revoke-keys"
Nov 01 03:28:45.405577 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:45.401927    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db76ac41-529c-4927-b1fc-b029877dd3fc" containerName="revoke-keys"
Nov 01 03:28:45.405577 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:45.402098    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="540c82b7-8782-49b8-86a4-576e10e21357" containerName="revoke-keys"
Nov 01 03:28:45.405577 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:45.402115    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="db76ac41-529c-4927-b1fc-b029877dd3fc" containerName="revoke-keys"
Nov 01 03:28:45.425637 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:45.421861    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d17bc02d-82rld"
Nov 01 03:28:45.449650 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:45.449607    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d17bc02d-82rld"]
Nov 01 03:28:45.534791 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:45.534748    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac769fae-f7ba-48c1-b254-594c1ac5899e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d17bc02d-82rld\" (UID: \"ac769fae-f7ba-48c1-b254-594c1ac5899e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d17bc02d-82rld"
Nov 01 03:28:45.534999 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:45.534944    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac769fae-f7ba-48c1-b254-594c1ac5899e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d17bc02d-82rld\" (UID: \"ac769fae-f7ba-48c1-b254-594c1ac5899e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d17bc02d-82rld"
Nov 01 03:28:45.535148 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:45.535120    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvxpq\" (UniqueName: \"kubernetes.io/projected/ac769fae-f7ba-48c1-b254-594c1ac5899e-kube-api-access-kvxpq\") pod \"maas-api-revoke-sub-d17bc02d-82rld\" (UID: \"ac769fae-f7ba-48c1-b254-594c1ac5899e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d17bc02d-82rld"
Nov 01 03:28:45.618509 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:45.618442    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c1b5ea-qtg9h" podStartSLOduration=2.618424336 podStartE2EDuration="2.618424336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:28:45.583331283 +0000 UTC m=+1330.106337872" watchObservedRunningTime="2026-11-01 03:28:45.618424336 +0000 UTC m=+1330.141430925"
Nov 01 03:28:45.636202 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:45.636127    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvxpq\" (UniqueName: \"kubernetes.io/projected/ac769fae-f7ba-48c1-b254-594c1ac5899e-kube-api-access-kvxpq\") pod \"maas-api-revoke-sub-d17bc02d-82rld\" (UID: \"ac769fae-f7ba-48c1-b254-594c1ac5899e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d17bc02d-82rld"
Nov 01 03:28:45.636347 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:45.636298    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac769fae-f7ba-48c1-b254-594c1ac5899e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d17bc02d-82rld\" (UID: \"ac769fae-f7ba-48c1-b254-594c1ac5899e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d17bc02d-82rld"
Nov 01 03:28:45.636504 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:45.636485    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac769fae-f7ba-48c1-b254-594c1ac5899e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d17bc02d-82rld\" (UID: \"ac769fae-f7ba-48c1-b254-594c1ac5899e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d17bc02d-82rld"
Nov 01 03:28:45.637683 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:45.637653    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac769fae-f7ba-48c1-b254-594c1ac5899e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d17bc02d-82rld\" (UID: \"ac769fae-f7ba-48c1-b254-594c1ac5899e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d17bc02d-82rld"
Nov 01 03:28:45.639681 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:45.639028    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac769fae-f7ba-48c1-b254-594c1ac5899e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d17bc02d-82rld\" (UID: \"ac769fae-f7ba-48c1-b254-594c1ac5899e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d17bc02d-82rld"
Nov 01 03:28:45.704709 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:45.704673    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvxpq\" (UniqueName: \"kubernetes.io/projected/ac769fae-f7ba-48c1-b254-594c1ac5899e-kube-api-access-kvxpq\") pod \"maas-api-revoke-sub-d17bc02d-82rld\" (UID: \"ac769fae-f7ba-48c1-b254-594c1ac5899e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d17bc02d-82rld"
Nov 01 03:28:45.743378 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:45.743350    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d17bc02d-82rld"
Nov 01 03:28:45.802604 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:45.785622    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4"
Nov 01 03:28:45.802604 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:45.789690    4930 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 01 03:28:45.802604 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:45.789740    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4" podUID="53f53373-f6f8-43db-a686-ab971adec842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 01 03:28:45.911063 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:45.907512    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:46.056541 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:46.056454    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:46.221441 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:46.221273    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:46.225436 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:46.225226    4930 generic.go:356] "Generic (PLEG): container finished" podID="864d195a-d8a1-41e3-bcdf-fd2acefa7f8b" containerID="686ed25ff460d0ea4fd03edbc55e4f45996cc6d7df6b3296bb36d147e4dd0532" exitCode=0
Nov 01 03:28:46.225737 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:46.225714    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c1b5ea-qtg9h" event={"ID":"864d195a-d8a1-41e3-bcdf-fd2acefa7f8b","Type":"ContainerDied","Data":"686ed25ff460d0ea4fd03edbc55e4f45996cc6d7df6b3296bb36d147e4dd0532"}
Nov 01 03:28:46.226810 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:28:46.226787    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac769fae_f7ba_48c1_b254_594c1ac5899e.slice/crio-a8fc942a52ce71a16b65a071ab7265e197226f6f4c7d100796c3cfd0cc5b64d5 WatchSource:0}: Error finding container a8fc942a52ce71a16b65a071ab7265e197226f6f4c7d100796c3cfd0cc5b64d5: Status 404 returned error can't find the container with id a8fc942a52ce71a16b65a071ab7265e197226f6f4c7d100796c3cfd0cc5b64d5
Nov 01 03:28:46.227821 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:46.227797    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d17bc02d-82rld"]
Nov 01 03:28:46.377426 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:46.369491    4930 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 01 03:28:46.377426 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:46.369592    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj" podUID="e9a70ec2-21fd-40ae-8d18-025a67248fce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 01 03:28:46.645119 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:46.644978    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:46.774528 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:46.774493    4930 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 01 03:28:46.774801 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:46.774768    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4" podUID="53f53373-f6f8-43db-a686-ab971adec842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 01 03:28:46.883960 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:46.883870    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:47.059216 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:47.058998    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:47.247177 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:47.247045    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:47.252897 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:47.252859    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d17bc02d-82rld" event={"ID":"ac769fae-f7ba-48c1-b254-594c1ac5899e","Type":"ContainerStarted","Data":"d56817ff89bde777468460b44f45665345a8e6069b0a9044463fb9349ae985d5"}
Nov 01 03:28:47.253063 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:47.252929    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d17bc02d-82rld" event={"ID":"ac769fae-f7ba-48c1-b254-594c1ac5899e","Type":"ContainerStarted","Data":"a8fc942a52ce71a16b65a071ab7265e197226f6f4c7d100796c3cfd0cc5b64d5"}
Nov 01 03:28:47.401578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:47.381426    4930 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 01 03:28:47.401578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:47.381488    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj" podUID="e9a70ec2-21fd-40ae-8d18-025a67248fce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 01 03:28:47.826799 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:47.826762    4930 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 01 03:28:47.827386 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:47.827352    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4" podUID="53f53373-f6f8-43db-a686-ab971adec842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 01 03:28:47.843291 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:47.843229    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d17bc02d-82rld" podStartSLOduration=2.843208796 podStartE2EDuration="2.843208796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:28:47.394760687 +0000 UTC m=+1331.917767277" watchObservedRunningTime="2026-11-01 03:28:47.843208796 +0000 UTC m=+1332.366215388"
Nov 01 03:28:48.230094 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:48.230071    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c1b5ea-qtg9h"
Nov 01 03:28:48.313032 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:48.312941    4930 generic.go:356] "Generic (PLEG): container finished" podID="ac769fae-f7ba-48c1-b254-594c1ac5899e" containerID="d56817ff89bde777468460b44f45665345a8e6069b0a9044463fb9349ae985d5" exitCode=0
Nov 01 03:28:48.313515 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:48.313224    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d17bc02d-82rld" event={"ID":"ac769fae-f7ba-48c1-b254-594c1ac5899e","Type":"ContainerDied","Data":"d56817ff89bde777468460b44f45665345a8e6069b0a9044463fb9349ae985d5"}
Nov 01 03:28:48.325579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:48.323449    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c1b5ea-qtg9h" event={"ID":"864d195a-d8a1-41e3-bcdf-fd2acefa7f8b","Type":"ContainerDied","Data":"0d803e33e982e904bbeaee15ad4f230e4116f3e1501e7c27033991b8c92feaec"}
Nov 01 03:28:48.325579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:48.323474    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27c1b5ea-qtg9h"
Nov 01 03:28:48.325579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:48.323501    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d803e33e982e904bbeaee15ad4f230e4116f3e1501e7c27033991b8c92feaec"
Nov 01 03:28:48.352482 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:48.352427    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/864d195a-d8a1-41e3-bcdf-fd2acefa7f8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/864d195a-d8a1-41e3-bcdf-fd2acefa7f8b-maas-api-openshift-ca\") pod \"864d195a-d8a1-41e3-bcdf-fd2acefa7f8b\" (UID: \"864d195a-d8a1-41e3-bcdf-fd2acefa7f8b\") "
Nov 01 03:28:48.352482 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:48.352472    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/864d195a-d8a1-41e3-bcdf-fd2acefa7f8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/864d195a-d8a1-41e3-bcdf-fd2acefa7f8b-maas-api-xks-ca\") pod \"864d195a-d8a1-41e3-bcdf-fd2acefa7f8b\" (UID: \"864d195a-d8a1-41e3-bcdf-fd2acefa7f8b\") "
Nov 01 03:28:48.352677 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:48.352617    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/864d195a-d8a1-41e3-bcdf-fd2acefa7f8b-kube-api-access-jq4fg\" (UniqueName: \"kubernetes.io/projected/864d195a-d8a1-41e3-bcdf-fd2acefa7f8b-kube-api-access-jq4fg\") pod \"864d195a-d8a1-41e3-bcdf-fd2acefa7f8b\" (UID: \"864d195a-d8a1-41e3-bcdf-fd2acefa7f8b\") "
Nov 01 03:28:48.353104 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:48.353083    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/864d195a-d8a1-41e3-bcdf-fd2acefa7f8b-maas-api-openshift-ca" pod "864d195a-d8a1-41e3-bcdf-fd2acefa7f8b" (UID: "864d195a-d8a1-41e3-bcdf-fd2acefa7f8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:28:48.356739 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:48.356711    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/864d195a-d8a1-41e3-bcdf-fd2acefa7f8b-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:48.381587 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:48.362889    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/864d195a-d8a1-41e3-bcdf-fd2acefa7f8b-kube-api-access-jq4fg" pod "864d195a-d8a1-41e3-bcdf-fd2acefa7f8b" (UID: "864d195a-d8a1-41e3-bcdf-fd2acefa7f8b"). InnerVolumeSpecName "kube-api-access-jq4fg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:28:48.381587 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:48.365651    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/864d195a-d8a1-41e3-bcdf-fd2acefa7f8b-maas-api-xks-ca" pod "864d195a-d8a1-41e3-bcdf-fd2acefa7f8b" (UID: "864d195a-d8a1-41e3-bcdf-fd2acefa7f8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:28:48.403579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:48.392274    4930 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 01 03:28:48.403579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:48.392346    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj" podUID="e9a70ec2-21fd-40ae-8d18-025a67248fce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 01 03:28:48.462582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:48.459769    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/864d195a-d8a1-41e3-bcdf-fd2acefa7f8b-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:48.462582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:48.459809    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq4fg\" (UniqueName: \"kubernetes.io/projected/864d195a-d8a1-41e3-bcdf-fd2acefa7f8b-kube-api-access-jq4fg\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:48.604574 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:48.602131    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:48.784595 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:48.777675    4930 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 01 03:28:48.784595 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:48.777743    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4" podUID="53f53373-f6f8-43db-a686-ab971adec842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 01 03:28:49.234835 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:28:49.234791    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:28:49.433230 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:49.433196    4930 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 01 03:28:49.433421 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:49.433250    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj" podUID="e9a70ec2-21fd-40ae-8d18-025a67248fce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 01 03:28:49.785084 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:49.784478    4930 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 01 03:28:49.785084 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:49.784545    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4" podUID="53f53373-f6f8-43db-a686-ab971adec842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 01 03:28:49.834123 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:49.834005    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:49.843597 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:49.843055    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d17bc02d-82rld"
Nov 01 03:28:49.954652 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:49.954610    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac769fae-f7ba-48c1-b254-594c1ac5899e-kube-api-access-kvxpq\" (UniqueName: \"kubernetes.io/projected/ac769fae-f7ba-48c1-b254-594c1ac5899e-kube-api-access-kvxpq\") pod \"ac769fae-f7ba-48c1-b254-594c1ac5899e\" (UID: \"ac769fae-f7ba-48c1-b254-594c1ac5899e\") "
Nov 01 03:28:49.954849 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:49.954814    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac769fae-f7ba-48c1-b254-594c1ac5899e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac769fae-f7ba-48c1-b254-594c1ac5899e-maas-api-openshift-ca\") pod \"ac769fae-f7ba-48c1-b254-594c1ac5899e\" (UID: \"ac769fae-f7ba-48c1-b254-594c1ac5899e\") "
Nov 01 03:28:49.955809 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:49.954998    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac769fae-f7ba-48c1-b254-594c1ac5899e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac769fae-f7ba-48c1-b254-594c1ac5899e-maas-api-xks-ca\") pod \"ac769fae-f7ba-48c1-b254-594c1ac5899e\" (UID: \"ac769fae-f7ba-48c1-b254-594c1ac5899e\") "
Nov 01 03:28:49.955809 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:49.955419    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac769fae-f7ba-48c1-b254-594c1ac5899e-maas-api-openshift-ca" pod "ac769fae-f7ba-48c1-b254-594c1ac5899e" (UID: "ac769fae-f7ba-48c1-b254-594c1ac5899e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:28:49.955809 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:49.955767    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac769fae-f7ba-48c1-b254-594c1ac5899e-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:49.978193 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:49.977706    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac769fae-f7ba-48c1-b254-594c1ac5899e-maas-api-xks-ca" pod "ac769fae-f7ba-48c1-b254-594c1ac5899e" (UID: "ac769fae-f7ba-48c1-b254-594c1ac5899e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:28:50.145978 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:50.139501    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac769fae-f7ba-48c1-b254-594c1ac5899e-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:50.209642 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:50.208850    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac769fae-f7ba-48c1-b254-594c1ac5899e-kube-api-access-kvxpq" pod "ac769fae-f7ba-48c1-b254-594c1ac5899e" (UID: "ac769fae-f7ba-48c1-b254-594c1ac5899e"). InnerVolumeSpecName "kube-api-access-kvxpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:28:50.209642 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:50.209427    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:50.313191 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:50.313137    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvxpq\" (UniqueName: \"kubernetes.io/projected/ac769fae-f7ba-48c1-b254-594c1ac5899e-kube-api-access-kvxpq\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:50.379630 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:50.368621    4930 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 01 03:28:50.379630 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:50.368690    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj" podUID="e9a70ec2-21fd-40ae-8d18-025a67248fce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 01 03:28:50.379630 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:50.370998    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d17bc02d-82rld" event={"ID":"ac769fae-f7ba-48c1-b254-594c1ac5899e","Type":"ContainerDied","Data":"a8fc942a52ce71a16b65a071ab7265e197226f6f4c7d100796c3cfd0cc5b64d5"}
Nov 01 03:28:50.379630 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:50.371078    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8fc942a52ce71a16b65a071ab7265e197226f6f4c7d100796c3cfd0cc5b64d5"
Nov 01 03:28:50.379630 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:50.371171    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d17bc02d-82rld"
Nov 01 03:28:50.725644 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:50.721934    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7911e5a0-8699c789f-dspb9"]
Nov 01 03:28:50.765761 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:50.765716    4930 cpu_manager.go:414] "containerMap: removing container" podUID="ac769fae-f7ba-48c1-b254-594c1ac5899e" containerName="revoke-keys"
Nov 01 03:28:50.766109 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:50.766090    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac769fae-f7ba-48c1-b254-594c1ac5899e" containerName="revoke-keys"
Nov 01 03:28:50.766263 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:50.766251    4930 cpu_manager.go:414] "containerMap: removing container" podUID="864d195a-d8a1-41e3-bcdf-fd2acefa7f8b" containerName="revoke-keys"
Nov 01 03:28:50.766352 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:50.766340    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="864d195a-d8a1-41e3-bcdf-fd2acefa7f8b" containerName="revoke-keys"
Nov 01 03:28:50.766713 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:50.766690    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="864d195a-d8a1-41e3-bcdf-fd2acefa7f8b" containerName="revoke-keys"
Nov 01 03:28:50.772579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:50.766871    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac769fae-f7ba-48c1-b254-594c1ac5899e" containerName="revoke-keys"
Nov 01 03:28:50.847587 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:50.785019    4930 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 01 03:28:50.847587 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:50.827539    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4" podUID="53f53373-f6f8-43db-a686-ab971adec842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 01 03:28:50.883641 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:50.882729    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7911e5a0-8699c789f-dspb9"]
Nov 01 03:28:50.883641 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:50.882858    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7911e5a0-8699c789f-dspb9"
Nov 01 03:28:50.940289 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:50.940233    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-7911e5a0-dockercfg-rpbrl\""
Nov 01 03:28:50.940476 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:50.940415    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-7911e5a0\""
Nov 01 03:28:50.980189 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:50.980109    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d888m\" (UniqueName: \"kubernetes.io/projected/0a2203d6-efb2-416c-95b3-92638b3ef827-kube-api-access-d888m\") pod \"payload-pre-processing-e2e-ait-7911e5a0-8699c789f-dspb9\" (UID: \"0a2203d6-efb2-416c-95b3-92638b3ef827\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7911e5a0-8699c789f-dspb9"
Nov 01 03:28:50.980442 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:50.980416    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a2203d6-efb2-416c-95b3-92638b3ef827-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7911e5a0-8699c789f-dspb9\" (UID: \"0a2203d6-efb2-416c-95b3-92638b3ef827\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7911e5a0-8699c789f-dspb9"
Nov 01 03:28:51.044654 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:51.044612    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85680f45-cbjpd"]
Nov 01 03:28:51.050504 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:51.050461    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85680f45-cbjpd"
Nov 01 03:28:51.081827 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:51.081474    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a2203d6-efb2-416c-95b3-92638b3ef827-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7911e5a0-8699c789f-dspb9\" (UID: \"0a2203d6-efb2-416c-95b3-92638b3ef827\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7911e5a0-8699c789f-dspb9"
Nov 01 03:28:51.081827 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:51.081677    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d888m\" (UniqueName: \"kubernetes.io/projected/0a2203d6-efb2-416c-95b3-92638b3ef827-kube-api-access-d888m\") pod \"payload-pre-processing-e2e-ait-7911e5a0-8699c789f-dspb9\" (UID: \"0a2203d6-efb2-416c-95b3-92638b3ef827\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7911e5a0-8699c789f-dspb9"
Nov 01 03:28:51.082442 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:51.082317    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a2203d6-efb2-416c-95b3-92638b3ef827-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7911e5a0-8699c789f-dspb9\" (UID: \"0a2203d6-efb2-416c-95b3-92638b3ef827\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7911e5a0-8699c789f-dspb9"
Nov 01 03:28:51.105713 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:51.105080    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:28:51.105713 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:51.105146    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:28:51.105713 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:51.105246    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:28:51.105713 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:51.105287    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:28:51.112922 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:51.112896    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85680f45-cbjpd"]
Nov 01 03:28:51.145220 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:51.145166    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d888m\" (UniqueName: \"kubernetes.io/projected/0a2203d6-efb2-416c-95b3-92638b3ef827-kube-api-access-d888m\") pod \"payload-pre-processing-e2e-ait-7911e5a0-8699c789f-dspb9\" (UID: \"0a2203d6-efb2-416c-95b3-92638b3ef827\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7911e5a0-8699c789f-dspb9"
Nov 01 03:28:51.183601 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:51.183526    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d8e03bf-33e5-4142-9bfe-723785dce2a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85680f45-cbjpd\" (UID: \"3d8e03bf-33e5-4142-9bfe-723785dce2a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85680f45-cbjpd"
Nov 01 03:28:51.183758 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:51.183632    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d8e03bf-33e5-4142-9bfe-723785dce2a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85680f45-cbjpd\" (UID: \"3d8e03bf-33e5-4142-9bfe-723785dce2a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85680f45-cbjpd"
Nov 01 03:28:51.183758 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:51.183729    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt8zd\" (UniqueName: \"kubernetes.io/projected/3d8e03bf-33e5-4142-9bfe-723785dce2a0-kube-api-access-gt8zd\") pod \"maas-api-revoke-sub-85680f45-cbjpd\" (UID: \"3d8e03bf-33e5-4142-9bfe-723785dce2a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85680f45-cbjpd"
Nov 01 03:28:51.240660 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:51.239842    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7911e5a0-8699c789f-dspb9"
Nov 01 03:28:51.285318 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:51.285289    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d8e03bf-33e5-4142-9bfe-723785dce2a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85680f45-cbjpd\" (UID: \"3d8e03bf-33e5-4142-9bfe-723785dce2a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85680f45-cbjpd"
Nov 01 03:28:51.285462 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:51.285368    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d8e03bf-33e5-4142-9bfe-723785dce2a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85680f45-cbjpd\" (UID: \"3d8e03bf-33e5-4142-9bfe-723785dce2a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85680f45-cbjpd"
Nov 01 03:28:51.285528 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:51.285485    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt8zd\" (UniqueName: \"kubernetes.io/projected/3d8e03bf-33e5-4142-9bfe-723785dce2a0-kube-api-access-gt8zd\") pod \"maas-api-revoke-sub-85680f45-cbjpd\" (UID: \"3d8e03bf-33e5-4142-9bfe-723785dce2a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85680f45-cbjpd"
Nov 01 03:28:51.286545 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:51.286125    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d8e03bf-33e5-4142-9bfe-723785dce2a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85680f45-cbjpd\" (UID: \"3d8e03bf-33e5-4142-9bfe-723785dce2a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85680f45-cbjpd"
Nov 01 03:28:51.288621 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:51.288581    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d8e03bf-33e5-4142-9bfe-723785dce2a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85680f45-cbjpd\" (UID: \"3d8e03bf-33e5-4142-9bfe-723785dce2a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85680f45-cbjpd"
Nov 01 03:28:51.350593 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:51.349486    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt8zd\" (UniqueName: \"kubernetes.io/projected/3d8e03bf-33e5-4142-9bfe-723785dce2a0-kube-api-access-gt8zd\") pod \"maas-api-revoke-sub-85680f45-cbjpd\" (UID: \"3d8e03bf-33e5-4142-9bfe-723785dce2a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85680f45-cbjpd"
Nov 01 03:28:51.376502 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:51.369053    4930 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 01 03:28:51.376502 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:51.369109    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj" podUID="e9a70ec2-21fd-40ae-8d18-025a67248fce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 01 03:28:51.376502 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:51.372946    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85680f45-cbjpd"
Nov 01 03:28:51.398234 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:51.397590    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2408401a-m5s4p"]
Nov 01 03:28:51.411145 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:51.411107    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2408401a-m5s4p"
Nov 01 03:28:51.458582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:51.457412    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2408401a-m5s4p"]
Nov 01 03:28:51.503128 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:51.502704    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46b23170-f01f-4af1-b13b-fa55f359c43e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2408401a-m5s4p\" (UID: \"46b23170-f01f-4af1-b13b-fa55f359c43e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2408401a-m5s4p"
Nov 01 03:28:51.503128 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:51.502819    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46b23170-f01f-4af1-b13b-fa55f359c43e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2408401a-m5s4p\" (UID: \"46b23170-f01f-4af1-b13b-fa55f359c43e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2408401a-m5s4p"
Nov 01 03:28:51.503128 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:51.502934    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sg2w\" (UniqueName: \"kubernetes.io/projected/46b23170-f01f-4af1-b13b-fa55f359c43e-kube-api-access-6sg2w\") pod \"maas-api-revoke-sub-2408401a-m5s4p\" (UID: \"46b23170-f01f-4af1-b13b-fa55f359c43e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2408401a-m5s4p"
Nov 01 03:28:51.608583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:51.606717    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46b23170-f01f-4af1-b13b-fa55f359c43e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2408401a-m5s4p\" (UID: \"46b23170-f01f-4af1-b13b-fa55f359c43e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2408401a-m5s4p"
Nov 01 03:28:51.608583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:51.606856    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sg2w\" (UniqueName: \"kubernetes.io/projected/46b23170-f01f-4af1-b13b-fa55f359c43e-kube-api-access-6sg2w\") pod \"maas-api-revoke-sub-2408401a-m5s4p\" (UID: \"46b23170-f01f-4af1-b13b-fa55f359c43e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2408401a-m5s4p"
Nov 01 03:28:51.608583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:51.607143    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46b23170-f01f-4af1-b13b-fa55f359c43e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2408401a-m5s4p\" (UID: \"46b23170-f01f-4af1-b13b-fa55f359c43e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2408401a-m5s4p"
Nov 01 03:28:51.608583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:51.608056    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46b23170-f01f-4af1-b13b-fa55f359c43e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2408401a-m5s4p\" (UID: \"46b23170-f01f-4af1-b13b-fa55f359c43e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2408401a-m5s4p"
Nov 01 03:28:51.611730 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:51.611676    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46b23170-f01f-4af1-b13b-fa55f359c43e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2408401a-m5s4p\" (UID: \"46b23170-f01f-4af1-b13b-fa55f359c43e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2408401a-m5s4p"
Nov 01 03:28:51.670666 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:51.670590    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sg2w\" (UniqueName: \"kubernetes.io/projected/46b23170-f01f-4af1-b13b-fa55f359c43e-kube-api-access-6sg2w\") pod \"maas-api-revoke-sub-2408401a-m5s4p\" (UID: \"46b23170-f01f-4af1-b13b-fa55f359c43e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2408401a-m5s4p"
Nov 01 03:28:51.683896 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:51.683862    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7911e5a0-8699c789f-dspb9"]
Nov 01 03:28:51.695245 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:28:51.695210    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a2203d6_efb2_416c_95b3_92638b3ef827.slice/crio-cfcbdc87a87d69e3abb103a3b189c26bccf513ed45f77a04d0b2c7beef48abf8 WatchSource:0}: Error finding container cfcbdc87a87d69e3abb103a3b189c26bccf513ed45f77a04d0b2c7beef48abf8: Status 404 returned error can't find the container with id cfcbdc87a87d69e3abb103a3b189c26bccf513ed45f77a04d0b2c7beef48abf8
Nov 01 03:28:51.747925 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:51.747894    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2408401a-m5s4p"
Nov 01 03:28:51.774968 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:51.774938    4930 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Nov 01 03:28:51.775093 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:51.774987    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4" podUID="53f53373-f6f8-43db-a686-ab971adec842" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Nov 01 03:28:51.862273 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:51.862244    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85680f45-cbjpd"]
Nov 01 03:28:52.234329 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:28:52.233862    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46b23170_f01f_4af1_b13b_fa55f359c43e.slice/crio-6769a1560a465179f4826c8a4af822e4afa89ed102cf651d7a083c458daadbd6 WatchSource:0}: Error finding container 6769a1560a465179f4826c8a4af822e4afa89ed102cf651d7a083c458daadbd6: Status 404 returned error can't find the container with id 6769a1560a465179f4826c8a4af822e4afa89ed102cf651d7a083c458daadbd6
Nov 01 03:28:52.239818 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:52.239732    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2408401a-m5s4p"]
Nov 01 03:28:52.416731 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:52.413120    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj"
Nov 01 03:28:52.416731 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:52.414828    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj"
Nov 01 03:28:52.417261 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:52.417107    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj"
Nov 01 03:28:52.437198 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:52.436800    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7911e5a0-8699c789f-dspb9" event={"ID":"0a2203d6-efb2-416c-95b3-92638b3ef827","Type":"ContainerStarted","Data":"97e7690ec0f71228a53bc82f84b9b8bb291d7af5fadd051651f92972f4c65667"}
Nov 01 03:28:52.437198 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:52.436884    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7911e5a0-8699c789f-dspb9" event={"ID":"0a2203d6-efb2-416c-95b3-92638b3ef827","Type":"ContainerStarted","Data":"cfcbdc87a87d69e3abb103a3b189c26bccf513ed45f77a04d0b2c7beef48abf8"}
Nov 01 03:28:52.437198 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:52.436954    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7911e5a0-8699c789f-dspb9"
Nov 01 03:28:52.439323 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:52.439289    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2408401a-m5s4p" event={"ID":"46b23170-f01f-4af1-b13b-fa55f359c43e","Type":"ContainerStarted","Data":"6769a1560a465179f4826c8a4af822e4afa89ed102cf651d7a083c458daadbd6"}
Nov 01 03:28:52.442489 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:52.442458    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85680f45-cbjpd" event={"ID":"3d8e03bf-33e5-4142-9bfe-723785dce2a0","Type":"ContainerStarted","Data":"6b1a65ce0e89dc3548df9322c44ca33699c9510d5c02424326a17d4e26f5ccc3"}
Nov 01 03:28:52.442665 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:52.442511    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85680f45-cbjpd" event={"ID":"3d8e03bf-33e5-4142-9bfe-723785dce2a0","Type":"ContainerStarted","Data":"a0cbfdb17e3af37be2656dd40f4c78782ba8205c5b64f050c71a40e19837a9bb"}
Nov 01 03:28:52.690674 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:52.690604    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85680f45-cbjpd" podStartSLOduration=2.690575815 podStartE2EDuration="2.690575815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:28:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:28:52.688906283 +0000 UTC m=+1337.211912872" watchObservedRunningTime="2026-11-01 03:28:52.690575815 +0000 UTC m=+1337.213582401"
Nov 01 03:28:52.781193 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:52.778699    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4"
Nov 01 03:28:52.781193 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:52.780169    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4"
Nov 01 03:28:52.781193 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:52.781217    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4"
Nov 01 03:28:52.937974 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:52.937918    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-7911e5a0-8699c789f-dspb9" podStartSLOduration=2.937901964 podStartE2EDuration="2.937901964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:28:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:28:52.927223534 +0000 UTC m=+1337.450230123" watchObservedRunningTime="2026-11-01 03:28:52.937901964 +0000 UTC m=+1337.460908555"
Nov 01 03:28:53.456386 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:53.456351    4930 generic.go:356] "Generic (PLEG): container finished" podID="46b23170-f01f-4af1-b13b-fa55f359c43e" containerID="1b0576b8ca23994ea7e4544dedfd0ce9c3e3578b35d9606198628d10c12d7430" exitCode=0
Nov 01 03:28:53.456816 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:53.456440    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2408401a-m5s4p" event={"ID":"46b23170-f01f-4af1-b13b-fa55f359c43e","Type":"ContainerDied","Data":"1b0576b8ca23994ea7e4544dedfd0ce9c3e3578b35d9606198628d10c12d7430"}
Nov 01 03:28:53.458330 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:53.458303    4930 generic.go:356] "Generic (PLEG): container finished" podID="3d8e03bf-33e5-4142-9bfe-723785dce2a0" containerID="6b1a65ce0e89dc3548df9322c44ca33699c9510d5c02424326a17d4e26f5ccc3" exitCode=0
Nov 01 03:28:53.458477 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:53.458351    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85680f45-cbjpd" event={"ID":"3d8e03bf-33e5-4142-9bfe-723785dce2a0","Type":"ContainerDied","Data":"6b1a65ce0e89dc3548df9322c44ca33699c9510d5c02424326a17d4e26f5ccc3"}
Nov 01 03:28:53.639686 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:53.639650    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:28:53.639807 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:53.639714    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:28:53.844389 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:53.844351    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:54.059478 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:54.059374    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:54.228725 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:54.228683    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:54.384359 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:54.384297    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:54.473322 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:54.473290    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvgwp" event={"ID":"c7161bfc-1b3c-4245-a0e7-c35a7fe69be1","Type":"ContainerStarted","Data":"6c988adf0b9527638cb030cd350cd466a7e507ebf99dbd3c3ba94f0d76905966"}
Nov 01 03:28:54.473867 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:54.473478    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvgwp"
Nov 01 03:28:54.475292 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:54.475265    4930 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-bvgwp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:28:54.475465 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:54.475438    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvgwp" podUID="c7161bfc-1b3c-4245-a0e7-c35a7fe69be1" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:28:54.675055 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:54.671667    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvgwp" podStartSLOduration=1.930028032 podStartE2EDuration="11.671645462s" totalImagesPullingTime="9.74161743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:28:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:28:54.663140409 +0000 UTC m=+1339.186146997" watchObservedRunningTime="2026-11-01 03:28:54.671645462 +0000 UTC m=+1339.194652144"
Nov 01 03:28:54.702130 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:54.702099    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85680f45-cbjpd"
Nov 01 03:28:54.749872 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:54.749847    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2408401a-m5s4p"
Nov 01 03:28:54.791519 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:54.791481    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d8e03bf-33e5-4142-9bfe-723785dce2a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d8e03bf-33e5-4142-9bfe-723785dce2a0-maas-api-xks-ca\") pod \"3d8e03bf-33e5-4142-9bfe-723785dce2a0\" (UID: \"3d8e03bf-33e5-4142-9bfe-723785dce2a0\") "
Nov 01 03:28:54.791715 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:54.791626    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d8e03bf-33e5-4142-9bfe-723785dce2a0-kube-api-access-gt8zd\" (UniqueName: \"kubernetes.io/projected/3d8e03bf-33e5-4142-9bfe-723785dce2a0-kube-api-access-gt8zd\") pod \"3d8e03bf-33e5-4142-9bfe-723785dce2a0\" (UID: \"3d8e03bf-33e5-4142-9bfe-723785dce2a0\") "
Nov 01 03:28:54.791763 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:54.791733    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46b23170-f01f-4af1-b13b-fa55f359c43e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46b23170-f01f-4af1-b13b-fa55f359c43e-maas-api-xks-ca\") pod \"46b23170-f01f-4af1-b13b-fa55f359c43e\" (UID: \"46b23170-f01f-4af1-b13b-fa55f359c43e\") "
Nov 01 03:28:54.791907 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:54.791885    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46b23170-f01f-4af1-b13b-fa55f359c43e-kube-api-access-6sg2w\" (UniqueName: \"kubernetes.io/projected/46b23170-f01f-4af1-b13b-fa55f359c43e-kube-api-access-6sg2w\") pod \"46b23170-f01f-4af1-b13b-fa55f359c43e\" (UID: \"46b23170-f01f-4af1-b13b-fa55f359c43e\") "
Nov 01 03:28:54.791985 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:54.791963    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d8e03bf-33e5-4142-9bfe-723785dce2a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d8e03bf-33e5-4142-9bfe-723785dce2a0-maas-api-openshift-ca\") pod \"3d8e03bf-33e5-4142-9bfe-723785dce2a0\" (UID: \"3d8e03bf-33e5-4142-9bfe-723785dce2a0\") "
Nov 01 03:28:54.792048 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:54.792028    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46b23170-f01f-4af1-b13b-fa55f359c43e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46b23170-f01f-4af1-b13b-fa55f359c43e-maas-api-openshift-ca\") pod \"46b23170-f01f-4af1-b13b-fa55f359c43e\" (UID: \"46b23170-f01f-4af1-b13b-fa55f359c43e\") "
Nov 01 03:28:54.792765 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:54.792737    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d8e03bf-33e5-4142-9bfe-723785dce2a0-maas-api-openshift-ca" pod "3d8e03bf-33e5-4142-9bfe-723785dce2a0" (UID: "3d8e03bf-33e5-4142-9bfe-723785dce2a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:28:54.793014 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:54.792989    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46b23170-f01f-4af1-b13b-fa55f359c43e-maas-api-openshift-ca" pod "46b23170-f01f-4af1-b13b-fa55f359c43e" (UID: "46b23170-f01f-4af1-b13b-fa55f359c43e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:28:54.794298 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:54.794270    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d8e03bf-33e5-4142-9bfe-723785dce2a0-kube-api-access-gt8zd" pod "3d8e03bf-33e5-4142-9bfe-723785dce2a0" (UID: "3d8e03bf-33e5-4142-9bfe-723785dce2a0"). InnerVolumeSpecName "kube-api-access-gt8zd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:28:54.794448 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:54.794416    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d8e03bf-33e5-4142-9bfe-723785dce2a0-maas-api-xks-ca" pod "3d8e03bf-33e5-4142-9bfe-723785dce2a0" (UID: "3d8e03bf-33e5-4142-9bfe-723785dce2a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:28:54.794448 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:54.794441    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46b23170-f01f-4af1-b13b-fa55f359c43e-maas-api-xks-ca" pod "46b23170-f01f-4af1-b13b-fa55f359c43e" (UID: "46b23170-f01f-4af1-b13b-fa55f359c43e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:28:54.795165 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:54.795141    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b23170-f01f-4af1-b13b-fa55f359c43e-kube-api-access-6sg2w" pod "46b23170-f01f-4af1-b13b-fa55f359c43e" (UID: "46b23170-f01f-4af1-b13b-fa55f359c43e"). InnerVolumeSpecName "kube-api-access-6sg2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:28:54.895644 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:54.895609    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d8e03bf-33e5-4142-9bfe-723785dce2a0-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:54.895644 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:54.895642    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt8zd\" (UniqueName: \"kubernetes.io/projected/3d8e03bf-33e5-4142-9bfe-723785dce2a0-kube-api-access-gt8zd\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:54.895644 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:54.895653    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46b23170-f01f-4af1-b13b-fa55f359c43e-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:54.895948 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:54.895662    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sg2w\" (UniqueName: \"kubernetes.io/projected/46b23170-f01f-4af1-b13b-fa55f359c43e-kube-api-access-6sg2w\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:54.895948 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:54.895671    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d8e03bf-33e5-4142-9bfe-723785dce2a0-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:54.895948 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:54.895679    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46b23170-f01f-4af1-b13b-fa55f359c43e-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:28:55.483951 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:55.483918    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2408401a-m5s4p"
Nov 01 03:28:55.483951 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:55.483942    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2408401a-m5s4p" event={"ID":"46b23170-f01f-4af1-b13b-fa55f359c43e","Type":"ContainerDied","Data":"6769a1560a465179f4826c8a4af822e4afa89ed102cf651d7a083c458daadbd6"}
Nov 01 03:28:55.484454 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:55.483999    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6769a1560a465179f4826c8a4af822e4afa89ed102cf651d7a083c458daadbd6"
Nov 01 03:28:55.485526 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:55.485494    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85680f45-cbjpd" event={"ID":"3d8e03bf-33e5-4142-9bfe-723785dce2a0","Type":"ContainerDied","Data":"a0cbfdb17e3af37be2656dd40f4c78782ba8205c5b64f050c71a40e19837a9bb"}
Nov 01 03:28:55.485526 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:55.485520    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85680f45-cbjpd"
Nov 01 03:28:55.485711 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:55.485543    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0cbfdb17e3af37be2656dd40f4c78782ba8205c5b64f050c71a40e19837a9bb"
Nov 01 03:28:55.486397 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:55.486372    4930 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-bvgwp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:28:55.486476 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:55.486434    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvgwp" podUID="c7161bfc-1b3c-4245-a0e7-c35a7fe69be1" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:28:56.714159 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:56.714127    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:56.871621 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:56.871544    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:57.629640 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:57.629546    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:58.486493 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:58.486463    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvgwp"
Nov 01 03:28:59.206124 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:59.206086    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:59.422489 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:59.422452    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:59.567389 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:59.567301    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:28:59.976376 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:28:59.976326    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:29:00.178903 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:00.178647    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:29:00.778999 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:00.778958    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e6e7ea12-67857874ff-gmjm9"]
Nov 01 03:29:00.780512 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:00.780481    4930 cpu_manager.go:414] "containerMap: removing container" podUID="3d8e03bf-33e5-4142-9bfe-723785dce2a0" containerName="revoke-keys"
Nov 01 03:29:00.780706 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:00.780688    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d8e03bf-33e5-4142-9bfe-723785dce2a0" containerName="revoke-keys"
Nov 01 03:29:00.780881 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:00.780869    4930 cpu_manager.go:414] "containerMap: removing container" podUID="46b23170-f01f-4af1-b13b-fa55f359c43e" containerName="revoke-keys"
Nov 01 03:29:00.780956 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:00.780945    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46b23170-f01f-4af1-b13b-fa55f359c43e" containerName="revoke-keys"
Nov 01 03:29:00.781218 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:00.781203    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="46b23170-f01f-4af1-b13b-fa55f359c43e" containerName="revoke-keys"
Nov 01 03:29:00.781313 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:00.781302    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d8e03bf-33e5-4142-9bfe-723785dce2a0" containerName="revoke-keys"
Nov 01 03:29:00.792022 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:00.791998    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e6e7ea12-67857874ff-gmjm9"
Nov 01 03:29:00.797962 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:00.797935    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e6e7ea12-67857874ff-gmjm9"]
Nov 01 03:29:00.894142 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:00.894104    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e6e7ea12\""
Nov 01 03:29:00.894142 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:00.894116    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e6e7ea12-dockercfg-8j2gc\""
Nov 01 03:29:00.906894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:00.906854    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7529a025-fe09-4ba1-9268-a6670b123646-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e6e7ea12-67857874ff-gmjm9\" (UID: \"7529a025-fe09-4ba1-9268-a6670b123646\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e6e7ea12-67857874ff-gmjm9"
Nov 01 03:29:00.907097 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:00.906915    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plspb\" (UniqueName: \"kubernetes.io/projected/7529a025-fe09-4ba1-9268-a6670b123646-kube-api-access-plspb\") pod \"payload-pre-processing-e2e-ait-e6e7ea12-67857874ff-gmjm9\" (UID: \"7529a025-fe09-4ba1-9268-a6670b123646\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e6e7ea12-67857874ff-gmjm9"
Nov 01 03:29:01.008383 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:01.008342    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7529a025-fe09-4ba1-9268-a6670b123646-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e6e7ea12-67857874ff-gmjm9\" (UID: \"7529a025-fe09-4ba1-9268-a6670b123646\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e6e7ea12-67857874ff-gmjm9"
Nov 01 03:29:01.008623 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:01.008405    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plspb\" (UniqueName: \"kubernetes.io/projected/7529a025-fe09-4ba1-9268-a6670b123646-kube-api-access-plspb\") pod \"payload-pre-processing-e2e-ait-e6e7ea12-67857874ff-gmjm9\" (UID: \"7529a025-fe09-4ba1-9268-a6670b123646\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e6e7ea12-67857874ff-gmjm9"
Nov 01 03:29:01.009096 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:01.009075    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7529a025-fe09-4ba1-9268-a6670b123646-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e6e7ea12-67857874ff-gmjm9\" (UID: \"7529a025-fe09-4ba1-9268-a6670b123646\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e6e7ea12-67857874ff-gmjm9"
Nov 01 03:29:01.069690 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:01.069610    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plspb\" (UniqueName: \"kubernetes.io/projected/7529a025-fe09-4ba1-9268-a6670b123646-kube-api-access-plspb\") pod \"payload-pre-processing-e2e-ait-e6e7ea12-67857874ff-gmjm9\" (UID: \"7529a025-fe09-4ba1-9268-a6670b123646\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e6e7ea12-67857874ff-gmjm9"
Nov 01 03:29:01.120840 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:01.120811    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e6e7ea12-67857874ff-gmjm9"
Nov 01 03:29:01.517948 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:29:01.517917    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7529a025_fe09_4ba1_9268_a6670b123646.slice/crio-259118e0fd6daa77a14f9a553df3e6e43e5b920e36b331e9c8ae738913fa506d WatchSource:0}: Error finding container 259118e0fd6daa77a14f9a553df3e6e43e5b920e36b331e9c8ae738913fa506d: Status 404 returned error can't find the container with id 259118e0fd6daa77a14f9a553df3e6e43e5b920e36b331e9c8ae738913fa506d
Nov 01 03:29:01.521211 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:01.521186    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e6e7ea12-67857874ff-gmjm9"]
Nov 01 03:29:01.544862 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:01.544838    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e6e7ea12-67857874ff-gmjm9" event={"ID":"7529a025-fe09-4ba1-9268-a6670b123646","Type":"ContainerStarted","Data":"259118e0fd6daa77a14f9a553df3e6e43e5b920e36b331e9c8ae738913fa506d"}
Nov 01 03:29:02.556026 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:02.555985    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e6e7ea12-67857874ff-gmjm9" event={"ID":"7529a025-fe09-4ba1-9268-a6670b123646","Type":"ContainerStarted","Data":"7e3a91978eb0f701830b0e4d3c945cbecef7cd4e6b1bac3befc1084259b0f4cd"}
Nov 01 03:29:02.556463 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:02.556057    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e6e7ea12-67857874ff-gmjm9"
Nov 01 03:29:02.676110 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:02.676047    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e6e7ea12-67857874ff-gmjm9" podStartSLOduration=2.6760291389999997 podStartE2EDuration="2.676029139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:29:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:29:02.671775762 +0000 UTC m=+1347.194782347" watchObservedRunningTime="2026-11-01 03:29:02.676029139 +0000 UTC m=+1347.199035740"
Nov 01 03:29:03.461990 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:03.461961    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7911e5a0-8699c789f-dspb9"
Nov 01 03:29:04.536293 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:04.536255    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:29:04.830375 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:04.830290    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:29:04.977518 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:04.977483    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:29:05.129079 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:05.129047    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:29:05.478316 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:05.478236    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"]
Nov 01 03:29:05.484043 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:05.484018    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"
Nov 01 03:29:05.691631 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:05.664527    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-44379c05-openshift-default-dockercfg-b89qx\""
Nov 01 03:29:05.695581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:05.692490    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:29:05.738141 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:05.738057    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjq2p\" (UniqueName: \"kubernetes.io/projected/9165d842-245c-4270-a5ab-8789ab0a2e89-kube-api-access-sjq2p\") pod \"e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8\" (UID: \"9165d842-245c-4270-a5ab-8789ab0a2e89\") " pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"
Nov 01 03:29:05.738498 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:05.738415    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9165d842-245c-4270-a5ab-8789ab0a2e89-credential-socket\") pod \"e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8\" (UID: \"9165d842-245c-4270-a5ab-8789ab0a2e89\") " pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"
Nov 01 03:29:05.740172 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:05.738676    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9165d842-245c-4270-a5ab-8789ab0a2e89-istio-data\") pod \"e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8\" (UID: \"9165d842-245c-4270-a5ab-8789ab0a2e89\") " pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"
Nov 01 03:29:05.740172 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:05.738718    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9165d842-245c-4270-a5ab-8789ab0a2e89-workload-socket\") pod \"e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8\" (UID: \"9165d842-245c-4270-a5ab-8789ab0a2e89\") " pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"
Nov 01 03:29:05.740172 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:05.738762    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9165d842-245c-4270-a5ab-8789ab0a2e89-istio-token\") pod \"e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8\" (UID: \"9165d842-245c-4270-a5ab-8789ab0a2e89\") " pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"
Nov 01 03:29:05.740172 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:05.738821    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9165d842-245c-4270-a5ab-8789ab0a2e89-istio-podinfo\") pod \"e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8\" (UID: \"9165d842-245c-4270-a5ab-8789ab0a2e89\") " pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"
Nov 01 03:29:05.740172 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:05.738850    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9165d842-245c-4270-a5ab-8789ab0a2e89-istiod-ca-cert\") pod \"e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8\" (UID: \"9165d842-245c-4270-a5ab-8789ab0a2e89\") " pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"
Nov 01 03:29:05.740172 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:05.739049    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9165d842-245c-4270-a5ab-8789ab0a2e89-istio-envoy\") pod \"e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8\" (UID: \"9165d842-245c-4270-a5ab-8789ab0a2e89\") " pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"
Nov 01 03:29:05.740172 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:05.739139    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9165d842-245c-4270-a5ab-8789ab0a2e89-workload-certs\") pod \"e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8\" (UID: \"9165d842-245c-4270-a5ab-8789ab0a2e89\") " pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"
Nov 01 03:29:05.780913 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:05.780877    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"]
Nov 01 03:29:05.841033 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:05.841000    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9165d842-245c-4270-a5ab-8789ab0a2e89-istio-data\") pod \"e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8\" (UID: \"9165d842-245c-4270-a5ab-8789ab0a2e89\") " pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"
Nov 01 03:29:05.841227 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:05.841050    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9165d842-245c-4270-a5ab-8789ab0a2e89-workload-socket\") pod \"e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8\" (UID: \"9165d842-245c-4270-a5ab-8789ab0a2e89\") " pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"
Nov 01 03:29:05.841227 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:05.841092    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9165d842-245c-4270-a5ab-8789ab0a2e89-istio-token\") pod \"e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8\" (UID: \"9165d842-245c-4270-a5ab-8789ab0a2e89\") " pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"
Nov 01 03:29:05.841227 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:05.841139    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9165d842-245c-4270-a5ab-8789ab0a2e89-istio-podinfo\") pod \"e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8\" (UID: \"9165d842-245c-4270-a5ab-8789ab0a2e89\") " pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"
Nov 01 03:29:05.841227 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:05.841167    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9165d842-245c-4270-a5ab-8789ab0a2e89-istiod-ca-cert\") pod \"e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8\" (UID: \"9165d842-245c-4270-a5ab-8789ab0a2e89\") " pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"
Nov 01 03:29:05.841437 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:05.841316    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9165d842-245c-4270-a5ab-8789ab0a2e89-istio-envoy\") pod \"e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8\" (UID: \"9165d842-245c-4270-a5ab-8789ab0a2e89\") " pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"
Nov 01 03:29:05.841437 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:05.841385    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9165d842-245c-4270-a5ab-8789ab0a2e89-workload-certs\") pod \"e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8\" (UID: \"9165d842-245c-4270-a5ab-8789ab0a2e89\") " pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"
Nov 01 03:29:05.841548 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:05.841445    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9165d842-245c-4270-a5ab-8789ab0a2e89-istio-data\") pod \"e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8\" (UID: \"9165d842-245c-4270-a5ab-8789ab0a2e89\") " pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"
Nov 01 03:29:05.851356 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:05.841543    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjq2p\" (UniqueName: \"kubernetes.io/projected/9165d842-245c-4270-a5ab-8789ab0a2e89-kube-api-access-sjq2p\") pod \"e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8\" (UID: \"9165d842-245c-4270-a5ab-8789ab0a2e89\") " pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"
Nov 01 03:29:05.851356 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:05.841610    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9165d842-245c-4270-a5ab-8789ab0a2e89-workload-socket\") pod \"e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8\" (UID: \"9165d842-245c-4270-a5ab-8789ab0a2e89\") " pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"
Nov 01 03:29:05.851356 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:05.841720    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9165d842-245c-4270-a5ab-8789ab0a2e89-credential-socket\") pod \"e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8\" (UID: \"9165d842-245c-4270-a5ab-8789ab0a2e89\") " pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"
Nov 01 03:29:05.851356 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:05.842099    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9165d842-245c-4270-a5ab-8789ab0a2e89-credential-socket\") pod \"e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8\" (UID: \"9165d842-245c-4270-a5ab-8789ab0a2e89\") " pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"
Nov 01 03:29:05.851356 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:05.842202    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9165d842-245c-4270-a5ab-8789ab0a2e89-istiod-ca-cert\") pod \"e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8\" (UID: \"9165d842-245c-4270-a5ab-8789ab0a2e89\") " pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"
Nov 01 03:29:05.851356 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:05.842432    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9165d842-245c-4270-a5ab-8789ab0a2e89-workload-certs\") pod \"e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8\" (UID: \"9165d842-245c-4270-a5ab-8789ab0a2e89\") " pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"
Nov 01 03:29:05.851356 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:05.845154    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9165d842-245c-4270-a5ab-8789ab0a2e89-istio-podinfo\") pod \"e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8\" (UID: \"9165d842-245c-4270-a5ab-8789ab0a2e89\") " pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"
Nov 01 03:29:05.851356 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:05.847796    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9165d842-245c-4270-a5ab-8789ab0a2e89-istio-envoy\") pod \"e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8\" (UID: \"9165d842-245c-4270-a5ab-8789ab0a2e89\") " pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"
Nov 01 03:29:05.921339 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:05.921311    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9165d842-245c-4270-a5ab-8789ab0a2e89-istio-token\") pod \"e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8\" (UID: \"9165d842-245c-4270-a5ab-8789ab0a2e89\") " pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"
Nov 01 03:29:05.921835 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:05.921816    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjq2p\" (UniqueName: \"kubernetes.io/projected/9165d842-245c-4270-a5ab-8789ab0a2e89-kube-api-access-sjq2p\") pod \"e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8\" (UID: \"9165d842-245c-4270-a5ab-8789ab0a2e89\") " pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"
Nov 01 03:29:06.117809 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:06.116191    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"
Nov 01 03:29:06.119531 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:06.119494    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:29:06.298007 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:06.297968    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:29:06.445985 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:06.445894    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:29:06.579029 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:29:06.579000    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9165d842_245c_4270_a5ab_8789ab0a2e89.slice/crio-05683958c8e15ff743fc672041e2bf99a99a15cba82787962a5dd7a591d70f42 WatchSource:0}: Error finding container 05683958c8e15ff743fc672041e2bf99a99a15cba82787962a5dd7a591d70f42: Status 404 returned error can't find the container with id 05683958c8e15ff743fc672041e2bf99a99a15cba82787962a5dd7a591d70f42
Nov 01 03:29:06.581639 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:06.581608    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:29:06.581758 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:06.581688    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:29:06.581758 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:06.581734    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:29:06.584198 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:06.584169    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"]
Nov 01 03:29:06.792296 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:06.792199    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8" event={"ID":"9165d842-245c-4270-a5ab-8789ab0a2e89","Type":"ContainerStarted","Data":"f792968e49e2f52ea64875f94f3cb60fc433bea27522ddf2d426f5032482b857"}
Nov 01 03:29:06.792296 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:06.792264    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8" event={"ID":"9165d842-245c-4270-a5ab-8789ab0a2e89","Type":"ContainerStarted","Data":"05683958c8e15ff743fc672041e2bf99a99a15cba82787962a5dd7a591d70f42"}
Nov 01 03:29:06.917159 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:06.917104    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8" podStartSLOduration=1.917084716 podStartE2EDuration="1.917084716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:29:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:29:06.912226322 +0000 UTC m=+1351.435232910" watchObservedRunningTime="2026-11-01 03:29:06.917084716 +0000 UTC m=+1351.440091310"
Nov 01 03:29:07.119603 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:07.118680    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"
Nov 01 03:29:07.125090 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:07.125056    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"
Nov 01 03:29:07.894423 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:07.894393    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"
Nov 01 03:29:07.935595 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:07.932852    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"
Nov 01 03:29:08.780578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:08.770386    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3751a433-bz4tr"]
Nov 01 03:29:08.799580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:08.790129    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3751a433-bz4tr"
Nov 01 03:29:08.916581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:08.899923    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:29:08.916581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:08.908871    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:29:08.916581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:08.908960    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:29:08.947499 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:08.920928    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkxpg\" (UniqueName: \"kubernetes.io/projected/3b6239c6-f921-4896-b615-1206f64527bf-kube-api-access-nkxpg\") pod \"maas-api-revoke-sub-3751a433-bz4tr\" (UID: \"3b6239c6-f921-4896-b615-1206f64527bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3751a433-bz4tr"
Nov 01 03:29:08.947499 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:08.921126    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b6239c6-f921-4896-b615-1206f64527bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3751a433-bz4tr\" (UID: \"3b6239c6-f921-4896-b615-1206f64527bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3751a433-bz4tr"
Nov 01 03:29:08.947499 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:08.921294    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b6239c6-f921-4896-b615-1206f64527bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3751a433-bz4tr\" (UID: \"3b6239c6-f921-4896-b615-1206f64527bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3751a433-bz4tr"
Nov 01 03:29:08.947499 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:08.921712    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:29:09.035636 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:09.023053    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7" podUID="1ce834cc-d34d-46cc-a081-4d2cc261becb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:09.158584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:09.136264    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkxpg\" (UniqueName: \"kubernetes.io/projected/3b6239c6-f921-4896-b615-1206f64527bf-kube-api-access-nkxpg\") pod \"maas-api-revoke-sub-3751a433-bz4tr\" (UID: \"3b6239c6-f921-4896-b615-1206f64527bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3751a433-bz4tr"
Nov 01 03:29:09.158584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:09.136406    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b6239c6-f921-4896-b615-1206f64527bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3751a433-bz4tr\" (UID: \"3b6239c6-f921-4896-b615-1206f64527bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3751a433-bz4tr"
Nov 01 03:29:09.158584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:09.136531    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b6239c6-f921-4896-b615-1206f64527bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3751a433-bz4tr\" (UID: \"3b6239c6-f921-4896-b615-1206f64527bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3751a433-bz4tr"
Nov 01 03:29:09.158584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:09.137430    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b6239c6-f921-4896-b615-1206f64527bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3751a433-bz4tr\" (UID: \"3b6239c6-f921-4896-b615-1206f64527bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3751a433-bz4tr"
Nov 01 03:29:09.328633 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:09.327694    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n" podUID="d2511772-66f9-40a0-9166-87c4345485c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:09.352581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:09.348690    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3751a433-bz4tr"]
Nov 01 03:29:09.419581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:09.409409    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b6239c6-f921-4896-b615-1206f64527bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3751a433-bz4tr\" (UID: \"3b6239c6-f921-4896-b615-1206f64527bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3751a433-bz4tr"
Nov 01 03:29:09.419581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:09.409615    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm" podUID="75d142b3-7172-4b57-bc07-a2f404c4656e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": context deadline exceeded"
Nov 01 03:29:09.446580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:09.446230    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp" podUID="667240b0-64a7-45cd-84fe-9cd7e0d8a301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 01 03:29:09.493580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:09.493024    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf" podUID="40881383-61bb-4fc8-9c75-f8ffce4834c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:09.525578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:09.517449    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkxpg\" (UniqueName: \"kubernetes.io/projected/3b6239c6-f921-4896-b615-1206f64527bf-kube-api-access-nkxpg\") pod \"maas-api-revoke-sub-3751a433-bz4tr\" (UID: \"3b6239c6-f921-4896-b615-1206f64527bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3751a433-bz4tr"
Nov 01 03:29:09.551580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:09.531860    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3751a433-bz4tr"
Nov 01 03:29:09.551580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:09.538828    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" podUID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:09.563338 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:09.559015    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t" podUID="13930c1e-2200-4412-b8d3-ae3e4126a851" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:09.639583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:09.588784    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw" podUID="b6d688b8-b911-4a6a-a842-2b0fb7a75e83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:09.672581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:09.671732    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps" podUID="d658772c-b072-4513-a639-d19a94c78525" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:09.857735 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:09.857688    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8" podUID="88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:09.911764 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:09.911636    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj" podUID="b0e32cb3-ac18-4c1f-819c-f1139a88bfea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 01 03:29:10.025710 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:10.025645    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7" podUID="1ce834cc-d34d-46cc-a081-4d2cc261becb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Nov 01 03:29:10.027249 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:10.027219    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm" podUID="b2637bc6-3828-4b75-9783-5415519403a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 01 03:29:10.028303 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:10.028287    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"
Nov 01 03:29:10.057068 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:10.057012    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj" podUID="b0e32cb3-ac18-4c1f-819c-f1139a88bfea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded"
Nov 01 03:29:10.125825 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:10.125786    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:29:10.356490 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:10.346663    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n" podUID="d2511772-66f9-40a0-9166-87c4345485c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:10.356490 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:10.347425    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"
Nov 01 03:29:10.411977 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:10.411649    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm" podUID="75d142b3-7172-4b57-bc07-a2f404c4656e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": context deadline exceeded"
Nov 01 03:29:10.728634 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:10.724979    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps" podUID="d658772c-b072-4513-a639-d19a94c78525" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 01 03:29:10.728634 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:10.726182    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2" podUID="e2483a06-2f4b-4a1e-af4e-a63eb50d29bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": context deadline exceeded"
Nov 01 03:29:10.728634 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:10.727318    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t" podUID="13930c1e-2200-4412-b8d3-ae3e4126a851" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 01 03:29:10.775675 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:10.762922    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:29:10.943321 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:10.936421    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:29:10.949643 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:10.947750    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:29:10.980342 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:10.979587    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj" podUID="b0e32cb3-ac18-4c1f-819c-f1139a88bfea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 01 03:29:11.019580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:10.766995    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" podUID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded"
Nov 01 03:29:11.048580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:11.045335    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:29:11.089699 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:11.089650    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7" podUID="1ce834cc-d34d-46cc-a081-4d2cc261becb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:11.128778 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:11.126311    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm" podUID="b2637bc6-3828-4b75-9783-5415519403a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:11.365013 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:11.168450    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:29:11.365410 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:11.365375    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:29:11.369601 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:11.367343    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"
Nov 01 03:29:11.369601 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:11.197035    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8" podUID="88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Nov 01 03:29:11.394788 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:11.198060    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw" podUID="b6d688b8-b911-4a6a-a842-2b0fb7a75e83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 01 03:29:11.424047 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:11.198994    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n" podUID="d2511772-66f9-40a0-9166-87c4345485c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": context deadline exceeded"
Nov 01 03:29:11.424277 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:11.169332    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf" podUID="40881383-61bb-4fc8-9c75-f8ffce4834c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Nov 01 03:29:11.477577 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:11.244851    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:29:11.524582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:11.445632    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:29:11.524582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:11.518009    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:29:11.524582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:11.453634    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm" podUID="75d142b3-7172-4b57-bc07-a2f404c4656e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:11.524582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:11.514687    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n" podUID="d2511772-66f9-40a0-9166-87c4345485c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:11.524582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:11.518222    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:29:11.788951 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:11.787704    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t" podUID="13930c1e-2200-4412-b8d3-ae3e4126a851" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:11.790235 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:11.790199    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps" podUID="d658772c-b072-4513-a639-d19a94c78525" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 01 03:29:11.859578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:11.856937    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2" podUID="e2483a06-2f4b-4a1e-af4e-a63eb50d29bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": context deadline exceeded"
Nov 01 03:29:12.027634 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:12.023635    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj" podUID="b0e32cb3-ac18-4c1f-819c-f1139a88bfea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 01 03:29:12.052630 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:12.033270    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" podUID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:12.155686 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:12.138274    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"
Nov 01 03:29:12.155686 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:12.138345    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"
Nov 01 03:29:12.156991 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:12.156211    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp" podUID="667240b0-64a7-45cd-84fe-9cd7e0d8a301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:12.235997 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:12.235958    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7" podUID="1ce834cc-d34d-46cc-a081-4d2cc261becb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:12.422194 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:12.422161    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:29:12.441188 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:12.430162    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf" podUID="40881383-61bb-4fc8-9c75-f8ffce4834c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:12.441188 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:12.431274    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8" podUID="88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:12.459596 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:12.447934    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw" podUID="b6d688b8-b911-4a6a-a842-2b0fb7a75e83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:12.524085 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:12.523987    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm" podUID="75d142b3-7172-4b57-bc07-a2f404c4656e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:12.539914 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:12.539866    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n" podUID="d2511772-66f9-40a0-9166-87c4345485c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:12.597335 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:12.597307    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"
Nov 01 03:29:12.601815 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:12.601793    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"
Nov 01 03:29:12.622227 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:12.622189    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:29:12.630140 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:12.630118    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:29:12.642227 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:12.642089    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3751a433-bz4tr"]
Nov 01 03:29:12.642492 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:12.642474    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:29:12.648108 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:12.648069    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:29:12.649411 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:12.649388    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:29:12.662420 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:12.662400    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:29:12.662612 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:12.662591    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"
Nov 01 03:29:12.701848 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:12.701768    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:29:13.075686 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:13.075591    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3751a433-bz4tr" event={"ID":"3b6239c6-f921-4896-b615-1206f64527bf","Type":"ContainerStarted","Data":"f9d7b6248d0877439d7cba0ea4a76669b8ac01471d5383a49a02527c784f5186"}
Nov 01 03:29:13.075686 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:13.075659    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3751a433-bz4tr" event={"ID":"3b6239c6-f921-4896-b615-1206f64527bf","Type":"ContainerStarted","Data":"40183d2d00a286591e85f2b5d9664ccefbd59b6e15fdf0bda0ccfaeb47a05bc2"}
Nov 01 03:29:13.573825 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:13.573796    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e6e7ea12-67857874ff-gmjm9"
Nov 01 03:29:13.797313 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:13.797215    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:29:13.999762 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:13.999461    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:29:14.137893 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:14.137858    4930 generic.go:356] "Generic (PLEG): container finished" podID="3b6239c6-f921-4896-b615-1206f64527bf" containerID="f9d7b6248d0877439d7cba0ea4a76669b8ac01471d5383a49a02527c784f5186" exitCode=0
Nov 01 03:29:14.138626 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:14.138590    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3751a433-bz4tr" event={"ID":"3b6239c6-f921-4896-b615-1206f64527bf","Type":"ContainerDied","Data":"f9d7b6248d0877439d7cba0ea4a76669b8ac01471d5383a49a02527c784f5186"}
Nov 01 03:29:15.337885 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:15.337861    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3751a433-bz4tr"
Nov 01 03:29:15.366618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:15.366573    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b6239c6-f921-4896-b615-1206f64527bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b6239c6-f921-4896-b615-1206f64527bf-maas-api-xks-ca\") pod \"3b6239c6-f921-4896-b615-1206f64527bf\" (UID: \"3b6239c6-f921-4896-b615-1206f64527bf\") "
Nov 01 03:29:15.366767 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:15.366750    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b6239c6-f921-4896-b615-1206f64527bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b6239c6-f921-4896-b615-1206f64527bf-maas-api-openshift-ca\") pod \"3b6239c6-f921-4896-b615-1206f64527bf\" (UID: \"3b6239c6-f921-4896-b615-1206f64527bf\") "
Nov 01 03:29:15.367003 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:15.366987    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b6239c6-f921-4896-b615-1206f64527bf-kube-api-access-nkxpg\" (UniqueName: \"kubernetes.io/projected/3b6239c6-f921-4896-b615-1206f64527bf-kube-api-access-nkxpg\") pod \"3b6239c6-f921-4896-b615-1206f64527bf\" (UID: \"3b6239c6-f921-4896-b615-1206f64527bf\") "
Nov 01 03:29:15.370582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:15.369037    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b6239c6-f921-4896-b615-1206f64527bf-maas-api-openshift-ca" pod "3b6239c6-f921-4896-b615-1206f64527bf" (UID: "3b6239c6-f921-4896-b615-1206f64527bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:29:15.384949 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:15.384919    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b6239c6-f921-4896-b615-1206f64527bf-kube-api-access-nkxpg" pod "3b6239c6-f921-4896-b615-1206f64527bf" (UID: "3b6239c6-f921-4896-b615-1206f64527bf"). InnerVolumeSpecName "kube-api-access-nkxpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:29:15.385179 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:15.385156    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b6239c6-f921-4896-b615-1206f64527bf-maas-api-xks-ca" pod "3b6239c6-f921-4896-b615-1206f64527bf" (UID: "3b6239c6-f921-4896-b615-1206f64527bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:29:15.472645 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:15.470049    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b6239c6-f921-4896-b615-1206f64527bf-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:29:15.472645 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:15.470089    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b6239c6-f921-4896-b615-1206f64527bf-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:29:15.472645 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:15.470106    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkxpg\" (UniqueName: \"kubernetes.io/projected/3b6239c6-f921-4896-b615-1206f64527bf-kube-api-access-nkxpg\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:29:16.157756 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:16.157727    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3751a433-bz4tr"
Nov 01 03:29:16.157756 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:16.157742    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3751a433-bz4tr" event={"ID":"3b6239c6-f921-4896-b615-1206f64527bf","Type":"ContainerDied","Data":"40183d2d00a286591e85f2b5d9664ccefbd59b6e15fdf0bda0ccfaeb47a05bc2"}
Nov 01 03:29:16.157979 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:16.157801    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40183d2d00a286591e85f2b5d9664ccefbd59b6e15fdf0bda0ccfaeb47a05bc2"
Nov 01 03:29:16.542854 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:16.542779    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:29:19.123222 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:29:19.123184    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:29:19.468872 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:19.468772    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:29:21.342466 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:21.342429    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:29:24.044861 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:24.044824    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:29:25.410006 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:25.409958    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:29:26.094730 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:26.094695    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:29:26.282005 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:26.281961    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:29:26.430592 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:26.430478    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:29:28.120111 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:28.120073    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf"]
Nov 01 03:29:28.120812 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:28.120795    4930 cpu_manager.go:414] "containerMap: removing container" podUID="3b6239c6-f921-4896-b615-1206f64527bf" containerName="revoke-keys"
Nov 01 03:29:28.120812 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:28.120812    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b6239c6-f921-4896-b615-1206f64527bf" containerName="revoke-keys"
Nov 01 03:29:28.120949 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:28.120937    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b6239c6-f921-4896-b615-1206f64527bf" containerName="revoke-keys"
Nov 01 03:29:28.129238 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:28.129216    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf"
Nov 01 03:29:28.228939 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:28.228880    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf"]
Nov 01 03:29:28.251330 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:28.251286    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-istiod-ca-cert\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf\" (UID: \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf"
Nov 01 03:29:28.251633 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:28.251611    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-istio-envoy\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf\" (UID: \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf"
Nov 01 03:29:28.251725 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:28.251654    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-istio-podinfo\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf\" (UID: \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf"
Nov 01 03:29:28.251725 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:28.251714    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8h2j\" (UniqueName: \"kubernetes.io/projected/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-kube-api-access-f8h2j\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf\" (UID: \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf"
Nov 01 03:29:28.251898 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:28.251815    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-istio-data\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf\" (UID: \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf"
Nov 01 03:29:28.251898 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:28.251864    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-credential-socket\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf\" (UID: \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf"
Nov 01 03:29:28.252039 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:28.251954    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-workload-certs\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf\" (UID: \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf"
Nov 01 03:29:28.252039 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:28.252014    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-workload-socket\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf\" (UID: \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf"
Nov 01 03:29:28.252167 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:28.252149    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-istio-token\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf\" (UID: \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf"
Nov 01 03:29:28.353679 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:28.353644    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-istiod-ca-cert\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf\" (UID: \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf"
Nov 01 03:29:28.353857 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:28.353750    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-istio-envoy\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf\" (UID: \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf"
Nov 01 03:29:28.353857 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:28.353785    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-istio-podinfo\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf\" (UID: \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf"
Nov 01 03:29:28.353857 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:28.353837    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8h2j\" (UniqueName: \"kubernetes.io/projected/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-kube-api-access-f8h2j\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf\" (UID: \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf"
Nov 01 03:29:28.354021 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:28.353905    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-istio-data\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf\" (UID: \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf"
Nov 01 03:29:28.354021 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:28.353944    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-credential-socket\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf\" (UID: \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf"
Nov 01 03:29:28.354129 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:28.354028    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-workload-certs\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf\" (UID: \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf"
Nov 01 03:29:28.354129 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:28.354085    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-workload-socket\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf\" (UID: \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf"
Nov 01 03:29:28.354216 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:28.354164    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-istio-token\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf\" (UID: \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf"
Nov 01 03:29:28.354609 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:28.354580    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-istiod-ca-cert\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf\" (UID: \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf"
Nov 01 03:29:28.354937 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:28.354911    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-istio-data\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf\" (UID: \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf"
Nov 01 03:29:28.355189 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:28.355164    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-workload-certs\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf\" (UID: \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf"
Nov 01 03:29:28.355472 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:28.355448    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-workload-socket\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf\" (UID: \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf"
Nov 01 03:29:28.356585 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:28.355671    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-credential-socket\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf\" (UID: \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf"
Nov 01 03:29:28.362637 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:28.358220    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-istio-podinfo\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf\" (UID: \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf"
Nov 01 03:29:28.384646 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:28.382813    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-istio-envoy\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf\" (UID: \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf"
Nov 01 03:29:28.416794 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:28.416769    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8h2j\" (UniqueName: \"kubernetes.io/projected/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-kube-api-access-f8h2j\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf\" (UID: \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf"
Nov 01 03:29:28.416794 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:28.416783    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-istio-token\") pod \"e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf\" (UID: \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\") " pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf"
Nov 01 03:29:28.448385 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:28.448358    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf"
Nov 01 03:29:28.906795 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:28.906764    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf"]
Nov 01 03:29:28.965300 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:29:28.965233    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd5ef13a_5e5e_45c5_89e9_fdf4d6899067.slice/crio-828028ca9708d7473713337fee159a7e93856ce8d5bbcc87a70a71c054e91cac WatchSource:0}: Error finding container 828028ca9708d7473713337fee159a7e93856ce8d5bbcc87a70a71c054e91cac: Status 404 returned error can't find the container with id 828028ca9708d7473713337fee159a7e93856ce8d5bbcc87a70a71c054e91cac
Nov 01 03:29:28.984732 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:28.982669    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:29:28.984732 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:28.982758    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:29:28.984732 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:28.982805    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:29:29.288229 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:29.288141    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf" event={"ID":"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067","Type":"ContainerStarted","Data":"fbbd4cfcaf19fa12907a174dec4a95dbe4faa4dffba643a7c85423579ae31143"}
Nov 01 03:29:29.288229 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:29.288198    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf" event={"ID":"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067","Type":"ContainerStarted","Data":"828028ca9708d7473713337fee159a7e93856ce8d5bbcc87a70a71c054e91cac"}
Nov 01 03:29:29.413931 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:29.413878    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf" podStartSLOduration=1.413863697 podStartE2EDuration="1.413863697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:29:29.408102312 +0000 UTC m=+1373.931108902" watchObservedRunningTime="2026-11-01 03:29:29.413863697 +0000 UTC m=+1373.936870282"
Nov 01 03:29:29.448917 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:29.448885    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf"
Nov 01 03:29:30.450213 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:30.450184    4930 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 01 03:29:30.450640 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:30.450233    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf" podUID="fd5ef13a-5e5e-45c5-89e9-fdf4d6899067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 01 03:29:30.836590 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:30.836535    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:29:31.047025 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:31.046987    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:29:31.189237 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:31.189139    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:29:31.449118 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:31.449031    4930 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 01 03:29:31.449118 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:31.449086    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf" podUID="fd5ef13a-5e5e-45c5-89e9-fdf4d6899067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 01 03:29:32.449149 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:32.449117    4930 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 01 03:29:32.449654 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:32.449172    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf" podUID="fd5ef13a-5e5e-45c5-89e9-fdf4d6899067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 01 03:29:33.448895 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:33.448855    4930 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 01 03:29:33.449100 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:33.448907    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf" podUID="fd5ef13a-5e5e-45c5-89e9-fdf4d6899067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 01 03:29:33.938026 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:33.937989    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:29:34.449017 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:34.448979    4930 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 01 03:29:34.449210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:34.449033    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf" podUID="fd5ef13a-5e5e-45c5-89e9-fdf4d6899067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 01 03:29:35.448937 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:35.448901    4930 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Nov 01 03:29:35.449436 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:35.448974    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf" podUID="fd5ef13a-5e5e-45c5-89e9-fdf4d6899067" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Nov 01 03:29:36.131838 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:36.131807    4930 scope.go:122] "RemoveContainer" containerID="3abc7e643fba1b0d2a6c5daa16d67ea024db19e35cd0561072b3a5acccb3ddad"
Nov 01 03:29:36.148770 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:36.148747    4930 scope.go:122] "RemoveContainer" containerID="15cbfc6500a39e08475ae301f35233e47d7be9739a633417bfc9940e32be6d1a"
Nov 01 03:29:36.157342 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:36.157314    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:29:36.159667 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:36.159650    4930 scope.go:122] "RemoveContainer" containerID="0f32a582e190b21a77a35a407d25bf0d86c221776ee182fc8808c3423a13ed9b"
Nov 01 03:29:36.172446 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:36.172430    4930 scope.go:122] "RemoveContainer" containerID="72bfcfe9560db808d211c0fcfc775b7cc25ed1d3cd2777d909c4668294e1561a"
Nov 01 03:29:36.213819 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:36.213670    4930 scope.go:122] "RemoveContainer" containerID="fcdc66c8210b4f4612e8c0e01599464bcb45d1f599c5b44cdc41269df15fb66e"
Nov 01 03:29:36.229005 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:36.228974    4930 scope.go:122] "RemoveContainer" containerID="b0d1dae7166b333d8de5b9b2f1deebcb0e0daa167476945bb5bab796297ffe17"
Nov 01 03:29:36.244122 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:36.244082    4930 scope.go:122] "RemoveContainer" containerID="105bb53f1f5c178985a2ab52f829caa56ca33545699a3b97c4db8ef8ff3e80a5"
Nov 01 03:29:36.276939 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:36.276532    4930 scope.go:122] "RemoveContainer" containerID="13e611e45bc7a71f14948c2c0a1abb97d97093b7a02ce5e2fc4e2d56332fdf12"
Nov 01 03:29:36.295997 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:36.295975    4930 scope.go:122] "RemoveContainer" containerID="7525db34171186f94600ea1777d93416508732f6d925370498af6fa7f760af8d"
Nov 01 03:29:36.331812 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:36.331747    4930 scope.go:122] "RemoveContainer" containerID="ff4960e4b031b593cf7d2916a85c0b1486b8f95962d8756fb58568fe65aa89b6"
Nov 01 03:29:36.517510 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:36.517472    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf"
Nov 01 03:29:36.518916 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:36.518894    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf"
Nov 01 03:29:36.519959 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:36.519942    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf"
Nov 01 03:29:41.055997 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:41.055962    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:29:43.536041 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:43.529276    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:29:45.051593 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:45.047664    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj" podUID="b0e32cb3-ac18-4c1f-819c-f1139a88bfea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 01 03:29:45.087987 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:45.087941    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" podUID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:45.388419 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:45.387381    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp" podUID="667240b0-64a7-45cd-84fe-9cd7e0d8a301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:45.445499 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:45.443346    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8" podUID="88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:45.477698 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:45.477655    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw" podUID="b6d688b8-b911-4a6a-a842-2b0fb7a75e83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:45.489744 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:45.489659    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm" podUID="b2637bc6-3828-4b75-9783-5415519403a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:45.568265 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:45.566702    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm" podUID="75d142b3-7172-4b57-bc07-a2f404c4656e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:45.587871 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:45.586797    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n" podUID="d2511772-66f9-40a0-9166-87c4345485c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:45.661637 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:45.655362    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf" podUID="40881383-61bb-4fc8-9c75-f8ffce4834c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:45.879283 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:45.879237    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm" podUID="b2637bc6-3828-4b75-9783-5415519403a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:46.054674 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:46.051292    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj" podUID="b0e32cb3-ac18-4c1f-819c-f1139a88bfea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:46.054674 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:46.052027    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:29:46.162438 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:46.088799    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" podUID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:46.164518 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:46.164398    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:29:46.459604 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:46.436724    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp" podUID="667240b0-64a7-45cd-84fe-9cd7e0d8a301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:46.501579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:46.482837    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp"
Nov 01 03:29:46.636339 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:46.503944    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw" podUID="b6d688b8-b911-4a6a-a842-2b0fb7a75e83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:46.669751 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:46.576614    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8" podUID="88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Nov 01 03:29:46.681579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:46.678202    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:29:46.711750 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:46.681830    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:29:46.719233 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:46.576906    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm" podUID="75d142b3-7172-4b57-bc07-a2f404c4656e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:46.721874 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:46.721856    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:29:46.725716 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:46.613804    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n" podUID="d2511772-66f9-40a0-9166-87c4345485c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:46.935591 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:46.928040    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm" podUID="b2637bc6-3828-4b75-9783-5415519403a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:47.008265 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:46.980147    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"
Nov 01 03:29:47.032629 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:47.015832    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf" podUID="40881383-61bb-4fc8-9c75-f8ffce4834c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:47.032629 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:47.024019    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"
Nov 01 03:29:47.032629 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:47.028167    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:29:47.032629 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:47.028545    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:29:47.069579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:47.048111    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:29:47.192603 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:47.164484    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:29:47.192603 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:47.169314    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" podUID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:47.217583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:47.193820    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:29:47.263709 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:47.224668    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"
Nov 01 03:29:47.346571 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:47.342075    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:29:47.396984 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:47.396902    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:29:47.426390 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:47.426360    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp"
Nov 01 03:29:47.460844 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:47.460813    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:29:47.536532 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:47.529103    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"
Nov 01 03:29:47.548992 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:47.548965    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:29:47.589833 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:47.589796    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:29:49.129670 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:29:49.129628    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:29:52.459864 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:52.459823    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:29:56.070766 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:56.070723    4930 patch_prober.go:29] interesting pod/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 03:29:56.071533 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:56.071494    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f" podUID="0fd59ab7-fd50-4ce9-a290-7f018204eb66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:29:57.201361 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:57.201323    4930 generic.go:356] "Generic (PLEG): container finished" podID="0fd59ab7-fd50-4ce9-a290-7f018204eb66" containerID="fb093f377b7645fc576a77c9f9f21a1b0c51c1d7a97d12fc4c2a174636fa2df8" exitCode=0
Nov 01 03:29:57.201928 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:57.201403    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f" event={"ID":"0fd59ab7-fd50-4ce9-a290-7f018204eb66","Type":"ContainerDied","Data":"fb093f377b7645fc576a77c9f9f21a1b0c51c1d7a97d12fc4c2a174636fa2df8"}
Nov 01 03:29:57.202010 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:57.201979    4930 scope.go:122] "RemoveContainer" containerID="fb093f377b7645fc576a77c9f9f21a1b0c51c1d7a97d12fc4c2a174636fa2df8"
Nov 01 03:29:57.203889 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:57.203864    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:29:57.203978 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:57.203933    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:29:57.203978 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:57.203967    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:29:57.343004 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:57.342968    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:29:58.214463 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:58.214423    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f" event={"ID":"0fd59ab7-fd50-4ce9-a290-7f018204eb66","Type":"ContainerStarted","Data":"5430b25a0e6a45da064e414f6a4a164a847485e26ff22c074a83dac8ca4b2a76"}
Nov 01 03:29:58.529264 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:58.529173    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"
Nov 01 03:29:58.529596 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:58.529304    4930 patch_prober.go:29] interesting pod/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 01 03:29:58.529596 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:58.529350    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f" podUID="0fd59ab7-fd50-4ce9-a290-7f018204eb66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 01 03:29:59.529751 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:59.529717    4930 patch_prober.go:29] interesting pod/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 01 03:29:59.530195 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:29:59.529785    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f" podUID="0fd59ab7-fd50-4ce9-a290-7f018204eb66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 01 03:30:00.194648 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.194611    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e8c789-29891730-wqnkl"]
Nov 01 03:30:00.200348 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.200320    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e8c789-29891730-wqnkl"
Nov 01 03:30:00.207654 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.207630    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891730-m94pc"]
Nov 01 03:30:00.211966 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.211944    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891730-m94pc"
Nov 01 03:30:00.214741 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.214719    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891730-bzr6r"]
Nov 01 03:30:00.218267 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.218248    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891730-bzr6r"
Nov 01 03:30:00.225895 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.225869    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ded2d7-29891730-bmg7c"]
Nov 01 03:30:00.230291 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.230273    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ded2d7-29891730-bmg7c"
Nov 01 03:30:00.233697 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.233657    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891730-ts9lp"]
Nov 01 03:30:00.237832 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.237818    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891730-ts9lp"
Nov 01 03:30:00.241242 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.241220    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7911e5a0-29891730-wjggk"]
Nov 01 03:30:00.244452 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.244431    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7911e5a0-29891730-wjggk"
Nov 01 03:30:00.291590 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.288660    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:30:00.291590 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.288780    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:30:00.309902 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.308634    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891730-bzr6r"]
Nov 01 03:30:00.315901 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.315816    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e8c789-29891730-wqnkl"]
Nov 01 03:30:00.329591 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.326537    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c930267d-c979-4068-b469-17d331652e27-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e8c789-29891730-wqnkl\" (UID: \"c930267d-c979-4068-b469-17d331652e27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e8c789-29891730-wqnkl"
Nov 01 03:30:00.329591 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.326673    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dt49\" (UniqueName: \"kubernetes.io/projected/be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19-kube-api-access-7dt49\") pod \"maas-api-key-cleanup-29891730-ts9lp\" (UID: \"be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891730-ts9lp"
Nov 01 03:30:00.329591 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.326702    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9da05dfd-3a79-4c52-8ef7-f3b2c306673a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891730-bzr6r\" (UID: \"9da05dfd-3a79-4c52-8ef7-f3b2c306673a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891730-bzr6r"
Nov 01 03:30:00.329591 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.326758    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891730-ts9lp\" (UID: \"be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891730-ts9lp"
Nov 01 03:30:00.329591 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.326799    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c54k2\" (UniqueName: \"kubernetes.io/projected/c930267d-c979-4068-b469-17d331652e27-kube-api-access-c54k2\") pod \"maas-api-key-cleanup-e2e-worker-w4-e8c789-29891730-wqnkl\" (UID: \"c930267d-c979-4068-b469-17d331652e27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e8c789-29891730-wqnkl"
Nov 01 03:30:00.329591 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.326856    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zpjg\" (UniqueName: \"kubernetes.io/projected/9da05dfd-3a79-4c52-8ef7-f3b2c306673a-kube-api-access-4zpjg\") pod \"maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891730-bzr6r\" (UID: \"9da05dfd-3a79-4c52-8ef7-f3b2c306673a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891730-bzr6r"
Nov 01 03:30:00.329591 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.326954    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a21bb3b-ad09-4622-b497-b385d320bf6a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-7911e5a0-29891730-wjggk\" (UID: \"4a21bb3b-ad09-4622-b497-b385d320bf6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7911e5a0-29891730-wjggk"
Nov 01 03:30:00.329591 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.327042    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d610889c-d7df-4ed2-9636-1ec134c05f81-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ded2d7-29891730-bmg7c\" (UID: \"d610889c-d7df-4ed2-9636-1ec134c05f81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ded2d7-29891730-bmg7c"
Nov 01 03:30:00.329591 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.327118    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gzmc\" (UniqueName: \"kubernetes.io/projected/4a21bb3b-ad09-4622-b497-b385d320bf6a-kube-api-access-4gzmc\") pod \"maas-api-key-cleanup-e2e-ait-7911e5a0-29891730-wjggk\" (UID: \"4a21bb3b-ad09-4622-b497-b385d320bf6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7911e5a0-29891730-wjggk"
Nov 01 03:30:00.329591 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.327159    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mqw9\" (UniqueName: \"kubernetes.io/projected/d610889c-d7df-4ed2-9636-1ec134c05f81-kube-api-access-9mqw9\") pod \"maas-api-key-cleanup-e2e-worker-w0-ded2d7-29891730-bmg7c\" (UID: \"d610889c-d7df-4ed2-9636-1ec134c05f81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ded2d7-29891730-bmg7c"
Nov 01 03:30:00.329591 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.327289    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e8852d1-3681-4cc1-87f1-45f0ae2d1aec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d7f535-29891730-m94pc\" (UID: \"6e8852d1-3681-4cc1-87f1-45f0ae2d1aec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891730-m94pc"
Nov 01 03:30:00.329591 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.327376    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc47q\" (UniqueName: \"kubernetes.io/projected/6e8852d1-3681-4cc1-87f1-45f0ae2d1aec-kube-api-access-zc47q\") pod \"maas-api-key-cleanup-e2e-worker-w3-d7f535-29891730-m94pc\" (UID: \"6e8852d1-3681-4cc1-87f1-45f0ae2d1aec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891730-m94pc"
Nov 01 03:30:00.339671 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.338232    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:30:00.345818 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.345794    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891730-m94pc"]
Nov 01 03:30:00.355960 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.355935    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891730-ts9lp"]
Nov 01 03:30:00.407612 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.407254    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7911e5a0-29891730-wjggk"]
Nov 01 03:30:00.413048 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.413015    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ded2d7-29891730-bmg7c"]
Nov 01 03:30:00.429487 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.429449    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d610889c-d7df-4ed2-9636-1ec134c05f81-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ded2d7-29891730-bmg7c\" (UID: \"d610889c-d7df-4ed2-9636-1ec134c05f81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ded2d7-29891730-bmg7c"
Nov 01 03:30:00.429661 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.429511    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gzmc\" (UniqueName: \"kubernetes.io/projected/4a21bb3b-ad09-4622-b497-b385d320bf6a-kube-api-access-4gzmc\") pod \"maas-api-key-cleanup-e2e-ait-7911e5a0-29891730-wjggk\" (UID: \"4a21bb3b-ad09-4622-b497-b385d320bf6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7911e5a0-29891730-wjggk"
Nov 01 03:30:00.429661 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.429549    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mqw9\" (UniqueName: \"kubernetes.io/projected/d610889c-d7df-4ed2-9636-1ec134c05f81-kube-api-access-9mqw9\") pod \"maas-api-key-cleanup-e2e-worker-w0-ded2d7-29891730-bmg7c\" (UID: \"d610889c-d7df-4ed2-9636-1ec134c05f81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ded2d7-29891730-bmg7c"
Nov 01 03:30:00.429798 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.429686    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e8852d1-3681-4cc1-87f1-45f0ae2d1aec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d7f535-29891730-m94pc\" (UID: \"6e8852d1-3681-4cc1-87f1-45f0ae2d1aec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891730-m94pc"
Nov 01 03:30:00.429798 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.429767    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc47q\" (UniqueName: \"kubernetes.io/projected/6e8852d1-3681-4cc1-87f1-45f0ae2d1aec-kube-api-access-zc47q\") pod \"maas-api-key-cleanup-e2e-worker-w3-d7f535-29891730-m94pc\" (UID: \"6e8852d1-3681-4cc1-87f1-45f0ae2d1aec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891730-m94pc"
Nov 01 03:30:00.429910 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.429802    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c930267d-c979-4068-b469-17d331652e27-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e8c789-29891730-wqnkl\" (UID: \"c930267d-c979-4068-b469-17d331652e27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e8c789-29891730-wqnkl"
Nov 01 03:30:00.429910 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.429873    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dt49\" (UniqueName: \"kubernetes.io/projected/be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19-kube-api-access-7dt49\") pod \"maas-api-key-cleanup-29891730-ts9lp\" (UID: \"be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891730-ts9lp"
Nov 01 03:30:00.429910 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.429893    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9da05dfd-3a79-4c52-8ef7-f3b2c306673a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891730-bzr6r\" (UID: \"9da05dfd-3a79-4c52-8ef7-f3b2c306673a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891730-bzr6r"
Nov 01 03:30:00.430068 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.429932    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891730-ts9lp\" (UID: \"be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891730-ts9lp"
Nov 01 03:30:00.430068 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.429961    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c54k2\" (UniqueName: \"kubernetes.io/projected/c930267d-c979-4068-b469-17d331652e27-kube-api-access-c54k2\") pod \"maas-api-key-cleanup-e2e-worker-w4-e8c789-29891730-wqnkl\" (UID: \"c930267d-c979-4068-b469-17d331652e27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e8c789-29891730-wqnkl"
Nov 01 03:30:00.430068 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.430000    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zpjg\" (UniqueName: \"kubernetes.io/projected/9da05dfd-3a79-4c52-8ef7-f3b2c306673a-kube-api-access-4zpjg\") pod \"maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891730-bzr6r\" (UID: \"9da05dfd-3a79-4c52-8ef7-f3b2c306673a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891730-bzr6r"
Nov 01 03:30:00.430426 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.430404    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d610889c-d7df-4ed2-9636-1ec134c05f81-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ded2d7-29891730-bmg7c\" (UID: \"d610889c-d7df-4ed2-9636-1ec134c05f81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ded2d7-29891730-bmg7c"
Nov 01 03:30:00.430845 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.430804    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a21bb3b-ad09-4622-b497-b385d320bf6a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-7911e5a0-29891730-wjggk\" (UID: \"4a21bb3b-ad09-4622-b497-b385d320bf6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7911e5a0-29891730-wjggk"
Nov 01 03:30:00.430976 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.430950    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9da05dfd-3a79-4c52-8ef7-f3b2c306673a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891730-bzr6r\" (UID: \"9da05dfd-3a79-4c52-8ef7-f3b2c306673a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891730-bzr6r"
Nov 01 03:30:00.431025 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.431006    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891730-ts9lp\" (UID: \"be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891730-ts9lp"
Nov 01 03:30:00.431728 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.431700    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e8852d1-3681-4cc1-87f1-45f0ae2d1aec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d7f535-29891730-m94pc\" (UID: \"6e8852d1-3681-4cc1-87f1-45f0ae2d1aec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891730-m94pc"
Nov 01 03:30:00.431930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.431907    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a21bb3b-ad09-4622-b497-b385d320bf6a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-7911e5a0-29891730-wjggk\" (UID: \"4a21bb3b-ad09-4622-b497-b385d320bf6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7911e5a0-29891730-wjggk"
Nov 01 03:30:00.432323 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.432305    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c930267d-c979-4068-b469-17d331652e27-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-e8c789-29891730-wqnkl\" (UID: \"c930267d-c979-4068-b469-17d331652e27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e8c789-29891730-wqnkl"
Nov 01 03:30:00.500448 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.498606    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dt49\" (UniqueName: \"kubernetes.io/projected/be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19-kube-api-access-7dt49\") pod \"maas-api-key-cleanup-29891730-ts9lp\" (UID: \"be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891730-ts9lp"
Nov 01 03:30:00.500448 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.500009    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zpjg\" (UniqueName: \"kubernetes.io/projected/9da05dfd-3a79-4c52-8ef7-f3b2c306673a-kube-api-access-4zpjg\") pod \"maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891730-bzr6r\" (UID: \"9da05dfd-3a79-4c52-8ef7-f3b2c306673a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891730-bzr6r"
Nov 01 03:30:00.500448 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.500332    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gzmc\" (UniqueName: \"kubernetes.io/projected/4a21bb3b-ad09-4622-b497-b385d320bf6a-kube-api-access-4gzmc\") pod \"maas-api-key-cleanup-e2e-ait-7911e5a0-29891730-wjggk\" (UID: \"4a21bb3b-ad09-4622-b497-b385d320bf6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7911e5a0-29891730-wjggk"
Nov 01 03:30:00.500448 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.500340    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc47q\" (UniqueName: \"kubernetes.io/projected/6e8852d1-3681-4cc1-87f1-45f0ae2d1aec-kube-api-access-zc47q\") pod \"maas-api-key-cleanup-e2e-worker-w3-d7f535-29891730-m94pc\" (UID: \"6e8852d1-3681-4cc1-87f1-45f0ae2d1aec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891730-m94pc"
Nov 01 03:30:00.500816 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.500703    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c54k2\" (UniqueName: \"kubernetes.io/projected/c930267d-c979-4068-b469-17d331652e27-kube-api-access-c54k2\") pod \"maas-api-key-cleanup-e2e-worker-w4-e8c789-29891730-wqnkl\" (UID: \"c930267d-c979-4068-b469-17d331652e27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e8c789-29891730-wqnkl"
Nov 01 03:30:00.500993 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.500968    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mqw9\" (UniqueName: \"kubernetes.io/projected/d610889c-d7df-4ed2-9636-1ec134c05f81-kube-api-access-9mqw9\") pod \"maas-api-key-cleanup-e2e-worker-w0-ded2d7-29891730-bmg7c\" (UID: \"d610889c-d7df-4ed2-9636-1ec134c05f81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ded2d7-29891730-bmg7c"
Nov 01 03:30:00.523000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.522970    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e8c789-29891730-wqnkl"
Nov 01 03:30:00.530677 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.530312    4930 patch_prober.go:29] interesting pod/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 01 03:30:00.530677 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.530368    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f" podUID="0fd59ab7-fd50-4ce9-a290-7f018204eb66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 01 03:30:00.535818 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.535792    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891730-m94pc"
Nov 01 03:30:00.543192 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.543170    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891730-bzr6r"
Nov 01 03:30:00.556328 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.556294    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ded2d7-29891730-bmg7c"
Nov 01 03:30:00.603591 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.601762    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891730-ts9lp"
Nov 01 03:30:00.607599 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.606614    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7911e5a0-29891730-wjggk"
Nov 01 03:30:00.996631 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:30:00.995699    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc930267d_c979_4068_b469_17d331652e27.slice/crio-30d2fb7403751e2994692eafe5eadbb945cf10d9bfcab7cc95679b1cd39c8a2f WatchSource:0}: Error finding container 30d2fb7403751e2994692eafe5eadbb945cf10d9bfcab7cc95679b1cd39c8a2f: Status 404 returned error can't find the container with id 30d2fb7403751e2994692eafe5eadbb945cf10d9bfcab7cc95679b1cd39c8a2f
Nov 01 03:30:00.998716 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:00.998689    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e8c789-29891730-wqnkl"]
Nov 01 03:30:01.248564 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:01.248527    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e8c789-29891730-wqnkl" event={"ID":"c930267d-c979-4068-b469-17d331652e27","Type":"ContainerStarted","Data":"30d2fb7403751e2994692eafe5eadbb945cf10d9bfcab7cc95679b1cd39c8a2f"}
Nov 01 03:30:01.278773 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:30:01.267641    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e8852d1_3681_4cc1_87f1_45f0ae2d1aec.slice/crio-768accfeb7814228c2987a25a7ae832cf6dbd3d46531d6d9b1c66e80c6970832 WatchSource:0}: Error finding container 768accfeb7814228c2987a25a7ae832cf6dbd3d46531d6d9b1c66e80c6970832: Status 404 returned error can't find the container with id 768accfeb7814228c2987a25a7ae832cf6dbd3d46531d6d9b1c66e80c6970832
Nov 01 03:30:01.297282 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:01.297260    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891730-m94pc"]
Nov 01 03:30:01.304947 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:01.304922    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891730-bzr6r"]
Nov 01 03:30:01.406746 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:01.405011    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891730-ts9lp"]
Nov 01 03:30:01.439093 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:01.439065    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ded2d7-29891730-bmg7c"]
Nov 01 03:30:01.446456 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:01.446426    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7911e5a0-29891730-wjggk"]
Nov 01 03:30:01.529758 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:01.529728    4930 patch_prober.go:29] interesting pod/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 01 03:30:01.529875 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:01.529778    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f" podUID="0fd59ab7-fd50-4ce9-a290-7f018204eb66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 01 03:30:02.143345 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:02.143308    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:30:02.322274 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:02.322237    4930 generic.go:356] "Generic (PLEG): container finished" podID="c930267d-c979-4068-b469-17d331652e27" containerID="132c77aac5d24e8af8ce665063d71383bc97e3a0044a12474b7e249ea5c5a53c" exitCode=60
Nov 01 03:30:02.322551 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:02.322477    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e8c789-29891730-wqnkl" event={"ID":"c930267d-c979-4068-b469-17d331652e27","Type":"ContainerDied","Data":"132c77aac5d24e8af8ce665063d71383bc97e3a0044a12474b7e249ea5c5a53c"}
Nov 01 03:30:02.323098 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:02.322708    4930 scope.go:122] "RemoveContainer" containerID="132c77aac5d24e8af8ce665063d71383bc97e3a0044a12474b7e249ea5c5a53c"
Nov 01 03:30:02.327108 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:02.327090    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ded2d7-29891730-bmg7c" event={"ID":"d610889c-d7df-4ed2-9636-1ec134c05f81","Type":"ContainerStarted","Data":"a77def8e4af9de113bfcaa1639a39e8c3c157529d5a3a02cd08a329ad29b13d6"}
Nov 01 03:30:02.328178 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:02.328161    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ded2d7-29891730-bmg7c" event={"ID":"d610889c-d7df-4ed2-9636-1ec134c05f81","Type":"ContainerStarted","Data":"f2487744cbdbb2c883204af3d704019fcf75012d6fcb923e43e9e368e4a96511"}
Nov 01 03:30:02.329252 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:02.329235    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891730-bzr6r" event={"ID":"9da05dfd-3a79-4c52-8ef7-f3b2c306673a","Type":"ContainerStarted","Data":"f7845bc1851a6938b328a3d7ec7c75ec599793657e8cdd19d57369fb246012d6"}
Nov 01 03:30:02.329364 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:02.329353    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891730-bzr6r" event={"ID":"9da05dfd-3a79-4c52-8ef7-f3b2c306673a","Type":"ContainerStarted","Data":"60247faf01c3aa2252716f16911f7c09a123cdb4463379b511e068b1d8a50c80"}
Nov 01 03:30:02.333089 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:02.332450    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7911e5a0-29891730-wjggk" event={"ID":"4a21bb3b-ad09-4622-b497-b385d320bf6a","Type":"ContainerStarted","Data":"b8afecf119ad779af2c8d8313974356ffd1ab1dd8141fd98074bea3642d6a8dd"}
Nov 01 03:30:02.333089 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:02.332496    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7911e5a0-29891730-wjggk" event={"ID":"4a21bb3b-ad09-4622-b497-b385d320bf6a","Type":"ContainerStarted","Data":"e15b1e2f6661e418b054797dba2a6cab791b792688af45369b6579f217fe5234"}
Nov 01 03:30:02.334785 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:02.334768    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891730-ts9lp" event={"ID":"be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19","Type":"ContainerStarted","Data":"9e3dbd31ed802e2478020fb684103194868b41ce59e9adde01112601d1dff3ee"}
Nov 01 03:30:02.334913 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:02.334903    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891730-ts9lp" event={"ID":"be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19","Type":"ContainerStarted","Data":"bcf48861691580a045549b5ad7a69868cbf06371b69be970921df528e14c4d44"}
Nov 01 03:30:02.336873 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:02.336855    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891730-m94pc" event={"ID":"6e8852d1-3681-4cc1-87f1-45f0ae2d1aec","Type":"ContainerStarted","Data":"a39e5b4ba897b029983740b0ff20ccd677adfdae9cf9e8d8b7d255f2e0442b21"}
Nov 01 03:30:02.337065 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:02.337053    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891730-m94pc" event={"ID":"6e8852d1-3681-4cc1-87f1-45f0ae2d1aec","Type":"ContainerStarted","Data":"768accfeb7814228c2987a25a7ae832cf6dbd3d46531d6d9b1c66e80c6970832"}
Nov 01 03:30:02.529453 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:02.529418    4930 patch_prober.go:29] interesting pod/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 01 03:30:02.529635 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:02.529477    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f" podUID="0fd59ab7-fd50-4ce9-a290-7f018204eb66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 01 03:30:02.623352 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:02.623249    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891730-bzr6r" podStartSLOduration=2.623225845 podStartE2EDuration="2.623225845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:30:02.587929117 +0000 UTC m=+1407.110935708" watchObservedRunningTime="2026-11-01 03:30:02.623225845 +0000 UTC m=+1407.146232426"
Nov 01 03:30:02.713680 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:02.713627    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7911e5a0-29891730-wjggk" podStartSLOduration=2.713609998 podStartE2EDuration="2.713609998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:30:02.702993675 +0000 UTC m=+1407.226000267" watchObservedRunningTime="2026-11-01 03:30:02.713609998 +0000 UTC m=+1407.236616582"
Nov 01 03:30:02.820740 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:02.820691    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891730-ts9lp" podStartSLOduration=2.8206765320000002 podStartE2EDuration="2.820676532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:30:02.817475114 +0000 UTC m=+1407.340481702" watchObservedRunningTime="2026-11-01 03:30:02.820676532 +0000 UTC m=+1407.343683121"
Nov 01 03:30:02.945505 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:02.945446    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891730-m94pc" podStartSLOduration=2.945430907 podStartE2EDuration="2.945430907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:30:02.944530637 +0000 UTC m=+1407.467537231" watchObservedRunningTime="2026-11-01 03:30:02.945430907 +0000 UTC m=+1407.468437528"
Nov 01 03:30:03.096043 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:03.095956    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ded2d7-29891730-bmg7c" podStartSLOduration=3.095909353 podStartE2EDuration="3.095909353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:30:03.076950454 +0000 UTC m=+1407.599957043" watchObservedRunningTime="2026-11-01 03:30:03.095909353 +0000 UTC m=+1407.618915942"
Nov 01 03:30:03.349646 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:03.349612    4930 generic.go:356] "Generic (PLEG): container finished" podID="4a21bb3b-ad09-4622-b497-b385d320bf6a" containerID="b8afecf119ad779af2c8d8313974356ffd1ab1dd8141fd98074bea3642d6a8dd" exitCode=60
Nov 01 03:30:03.350137 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:03.349699    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7911e5a0-29891730-wjggk" event={"ID":"4a21bb3b-ad09-4622-b497-b385d320bf6a","Type":"ContainerDied","Data":"b8afecf119ad779af2c8d8313974356ffd1ab1dd8141fd98074bea3642d6a8dd"}
Nov 01 03:30:03.350137 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:03.349911    4930 scope.go:122] "RemoveContainer" containerID="b8afecf119ad779af2c8d8313974356ffd1ab1dd8141fd98074bea3642d6a8dd"
Nov 01 03:30:03.351612 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:03.351588    4930 generic.go:356] "Generic (PLEG): container finished" podID="be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19" containerID="9e3dbd31ed802e2478020fb684103194868b41ce59e9adde01112601d1dff3ee" exitCode=60
Nov 01 03:30:03.351692 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:03.351623    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891730-ts9lp" event={"ID":"be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19","Type":"ContainerDied","Data":"9e3dbd31ed802e2478020fb684103194868b41ce59e9adde01112601d1dff3ee"}
Nov 01 03:30:03.351898 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:03.351876    4930 scope.go:122] "RemoveContainer" containerID="9e3dbd31ed802e2478020fb684103194868b41ce59e9adde01112601d1dff3ee"
Nov 01 03:30:03.354708 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:03.354686    4930 generic.go:356] "Generic (PLEG): container finished" podID="6e8852d1-3681-4cc1-87f1-45f0ae2d1aec" containerID="a39e5b4ba897b029983740b0ff20ccd677adfdae9cf9e8d8b7d255f2e0442b21" exitCode=60
Nov 01 03:30:03.354816 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:03.354783    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891730-m94pc" event={"ID":"6e8852d1-3681-4cc1-87f1-45f0ae2d1aec","Type":"ContainerDied","Data":"a39e5b4ba897b029983740b0ff20ccd677adfdae9cf9e8d8b7d255f2e0442b21"}
Nov 01 03:30:03.355068 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:03.355050    4930 scope.go:122] "RemoveContainer" containerID="a39e5b4ba897b029983740b0ff20ccd677adfdae9cf9e8d8b7d255f2e0442b21"
Nov 01 03:30:03.358696 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:03.358546    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e8c789-29891730-wqnkl" event={"ID":"c930267d-c979-4068-b469-17d331652e27","Type":"ContainerStarted","Data":"ff34ce83206047f6b0cf3ba31646896ca9c84540a8cf8358943f92030cc06b87"}
Nov 01 03:30:03.400212 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:03.399108    4930 generic.go:356] "Generic (PLEG): container finished" podID="d610889c-d7df-4ed2-9636-1ec134c05f81" containerID="a77def8e4af9de113bfcaa1639a39e8c3c157529d5a3a02cd08a329ad29b13d6" exitCode=60
Nov 01 03:30:03.400212 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:03.399203    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ded2d7-29891730-bmg7c" event={"ID":"d610889c-d7df-4ed2-9636-1ec134c05f81","Type":"ContainerDied","Data":"a77def8e4af9de113bfcaa1639a39e8c3c157529d5a3a02cd08a329ad29b13d6"}
Nov 01 03:30:03.400212 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:03.399688    4930 scope.go:122] "RemoveContainer" containerID="a77def8e4af9de113bfcaa1639a39e8c3c157529d5a3a02cd08a329ad29b13d6"
Nov 01 03:30:03.403599 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:03.402836    4930 generic.go:356] "Generic (PLEG): container finished" podID="9da05dfd-3a79-4c52-8ef7-f3b2c306673a" containerID="f7845bc1851a6938b328a3d7ec7c75ec599793657e8cdd19d57369fb246012d6" exitCode=60
Nov 01 03:30:03.403599 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:03.402907    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891730-bzr6r" event={"ID":"9da05dfd-3a79-4c52-8ef7-f3b2c306673a","Type":"ContainerDied","Data":"f7845bc1851a6938b328a3d7ec7c75ec599793657e8cdd19d57369fb246012d6"}
Nov 01 03:30:03.403599 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:03.403225    4930 scope.go:122] "RemoveContainer" containerID="f7845bc1851a6938b328a3d7ec7c75ec599793657e8cdd19d57369fb246012d6"
Nov 01 03:30:03.530156 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:03.530093    4930 patch_prober.go:29] interesting pod/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body=
Nov 01 03:30:03.530156 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:03.530137    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f" podUID="0fd59ab7-fd50-4ce9-a290-7f018204eb66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused"
Nov 01 03:30:03.769794 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:03.769730    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e8c789-29891730-wqnkl" podStartSLOduration=3.76971118 podStartE2EDuration="3.76971118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:30:03.761809421 +0000 UTC m=+1408.284816021" watchObservedRunningTime="2026-11-01 03:30:03.76971118 +0000 UTC m=+1408.292717769"
Nov 01 03:30:04.414547 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:04.414510    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ded2d7-29891730-bmg7c" event={"ID":"d610889c-d7df-4ed2-9636-1ec134c05f81","Type":"ContainerStarted","Data":"b512ee46d0dad9679e450e4deec3d4d64e636cd6099e56490e13eef2cd8ae205"}
Nov 01 03:30:04.416822 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:04.416783    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891730-bzr6r" event={"ID":"9da05dfd-3a79-4c52-8ef7-f3b2c306673a","Type":"ContainerStarted","Data":"5b3b745ffb0fb425c437cd8f084700cf2858a6b7881ffa9c9025a4423ce91db6"}
Nov 01 03:30:04.419057 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:04.419035    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7911e5a0-29891730-wjggk" event={"ID":"4a21bb3b-ad09-4622-b497-b385d320bf6a","Type":"ContainerStarted","Data":"31d9e98628b64eada8b26f4fdff6daf8ec15833c84362ea39acaf681039f5865"}
Nov 01 03:30:04.421077 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:04.421054    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891730-ts9lp" event={"ID":"be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19","Type":"ContainerStarted","Data":"03cdfc444310303c9323a3386ec646c0f9e3c0318081d1b38df11ea97ba998af"}
Nov 01 03:30:04.423095 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:04.423076    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891730-m94pc" event={"ID":"6e8852d1-3681-4cc1-87f1-45f0ae2d1aec","Type":"ContainerStarted","Data":"e76fe87a29a52bded9de3f86f62af7078c2e4a1e0e695cb1a91d3ea1e6fd842e"}
Nov 01 03:30:04.424968 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:04.424942    4930 generic.go:356] "Generic (PLEG): container finished" podID="c930267d-c979-4068-b469-17d331652e27" containerID="ff34ce83206047f6b0cf3ba31646896ca9c84540a8cf8358943f92030cc06b87" exitCode=60
Nov 01 03:30:04.425057 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:04.425023    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e8c789-29891730-wqnkl" event={"ID":"c930267d-c979-4068-b469-17d331652e27","Type":"ContainerDied","Data":"ff34ce83206047f6b0cf3ba31646896ca9c84540a8cf8358943f92030cc06b87"}
Nov 01 03:30:04.425107 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:04.425076    4930 scope.go:122] "RemoveContainer" containerID="132c77aac5d24e8af8ce665063d71383bc97e3a0044a12474b7e249ea5c5a53c"
Nov 01 03:30:04.425187 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:04.425171    4930 scope.go:122] "RemoveContainer" containerID="ff34ce83206047f6b0cf3ba31646896ca9c84540a8cf8358943f92030cc06b87"
Nov 01 03:30:04.425479 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:30:04.425459    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e8c789-29891730-wqnkl_odh-ai-gateway-infra(c930267d-c979-4068-b469-17d331652e27)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e8c789-29891730-wqnkl" podUID="c930267d-c979-4068-b469-17d331652e27"
Nov 01 03:30:04.536550 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:04.536514    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"
Nov 01 03:30:04.537056 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:04.537028    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"
Nov 01 03:30:04.542993 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:04.542974    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"
Nov 01 03:30:05.438091 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:05.438049    4930 generic.go:356] "Generic (PLEG): container finished" podID="d610889c-d7df-4ed2-9636-1ec134c05f81" containerID="b512ee46d0dad9679e450e4deec3d4d64e636cd6099e56490e13eef2cd8ae205" exitCode=60
Nov 01 03:30:05.438654 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:05.438133    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ded2d7-29891730-bmg7c" event={"ID":"d610889c-d7df-4ed2-9636-1ec134c05f81","Type":"ContainerDied","Data":"b512ee46d0dad9679e450e4deec3d4d64e636cd6099e56490e13eef2cd8ae205"}
Nov 01 03:30:05.438654 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:05.438226    4930 scope.go:122] "RemoveContainer" containerID="a77def8e4af9de113bfcaa1639a39e8c3c157529d5a3a02cd08a329ad29b13d6"
Nov 01 03:30:05.438797 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:05.438673    4930 scope.go:122] "RemoveContainer" containerID="b512ee46d0dad9679e450e4deec3d4d64e636cd6099e56490e13eef2cd8ae205"
Nov 01 03:30:05.439128 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:30:05.439104    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ded2d7-29891730-bmg7c_odh-ai-gateway-infra(d610889c-d7df-4ed2-9636-1ec134c05f81)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ded2d7-29891730-bmg7c" podUID="d610889c-d7df-4ed2-9636-1ec134c05f81"
Nov 01 03:30:05.440337 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:05.440312    4930 generic.go:356] "Generic (PLEG): container finished" podID="9da05dfd-3a79-4c52-8ef7-f3b2c306673a" containerID="5b3b745ffb0fb425c437cd8f084700cf2858a6b7881ffa9c9025a4423ce91db6" exitCode=60
Nov 01 03:30:05.440422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:05.440390    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891730-bzr6r" event={"ID":"9da05dfd-3a79-4c52-8ef7-f3b2c306673a","Type":"ContainerDied","Data":"5b3b745ffb0fb425c437cd8f084700cf2858a6b7881ffa9c9025a4423ce91db6"}
Nov 01 03:30:05.440726 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:05.440711    4930 scope.go:122] "RemoveContainer" containerID="5b3b745ffb0fb425c437cd8f084700cf2858a6b7881ffa9c9025a4423ce91db6"
Nov 01 03:30:05.441080 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:30:05.441058    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891730-bzr6r_odh-ai-gateway-infra(9da05dfd-3a79-4c52-8ef7-f3b2c306673a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891730-bzr6r" podUID="9da05dfd-3a79-4c52-8ef7-f3b2c306673a"
Nov 01 03:30:05.442524 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:05.442501    4930 generic.go:356] "Generic (PLEG): container finished" podID="4a21bb3b-ad09-4622-b497-b385d320bf6a" containerID="31d9e98628b64eada8b26f4fdff6daf8ec15833c84362ea39acaf681039f5865" exitCode=60
Nov 01 03:30:05.442663 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:05.442644    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7911e5a0-29891730-wjggk" event={"ID":"4a21bb3b-ad09-4622-b497-b385d320bf6a","Type":"ContainerDied","Data":"31d9e98628b64eada8b26f4fdff6daf8ec15833c84362ea39acaf681039f5865"}
Nov 01 03:30:05.442854 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:05.442838    4930 scope.go:122] "RemoveContainer" containerID="31d9e98628b64eada8b26f4fdff6daf8ec15833c84362ea39acaf681039f5865"
Nov 01 03:30:05.443218 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:30:05.443190    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-7911e5a0-29891730-wjggk_odh-ai-gateway-infra(4a21bb3b-ad09-4622-b497-b385d320bf6a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7911e5a0-29891730-wjggk" podUID="4a21bb3b-ad09-4622-b497-b385d320bf6a"
Nov 01 03:30:05.444732 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:05.444710    4930 generic.go:356] "Generic (PLEG): container finished" podID="be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19" containerID="03cdfc444310303c9323a3386ec646c0f9e3c0318081d1b38df11ea97ba998af" exitCode=60
Nov 01 03:30:05.444818 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:05.444783    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891730-ts9lp" event={"ID":"be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19","Type":"ContainerDied","Data":"03cdfc444310303c9323a3386ec646c0f9e3c0318081d1b38df11ea97ba998af"}
Nov 01 03:30:05.445005 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:05.444988    4930 scope.go:122] "RemoveContainer" containerID="03cdfc444310303c9323a3386ec646c0f9e3c0318081d1b38df11ea97ba998af"
Nov 01 03:30:05.445313 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:30:05.445284    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891730-ts9lp_odh-ai-gateway-infra(be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891730-ts9lp" podUID="be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19"
Nov 01 03:30:05.446908 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:05.446886    4930 generic.go:356] "Generic (PLEG): container finished" podID="6e8852d1-3681-4cc1-87f1-45f0ae2d1aec" containerID="e76fe87a29a52bded9de3f86f62af7078c2e4a1e0e695cb1a91d3ea1e6fd842e" exitCode=60
Nov 01 03:30:05.447017 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:05.446968    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891730-m94pc" event={"ID":"6e8852d1-3681-4cc1-87f1-45f0ae2d1aec","Type":"ContainerDied","Data":"e76fe87a29a52bded9de3f86f62af7078c2e4a1e0e695cb1a91d3ea1e6fd842e"}
Nov 01 03:30:05.447156 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:05.447141    4930 scope.go:122] "RemoveContainer" containerID="e76fe87a29a52bded9de3f86f62af7078c2e4a1e0e695cb1a91d3ea1e6fd842e"
Nov 01 03:30:05.447425 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:30:05.447406    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d7f535-29891730-m94pc_odh-ai-gateway-infra(6e8852d1-3681-4cc1-87f1-45f0ae2d1aec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891730-m94pc" podUID="6e8852d1-3681-4cc1-87f1-45f0ae2d1aec"
Nov 01 03:30:05.449058 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:05.449040    4930 scope.go:122] "RemoveContainer" containerID="ff34ce83206047f6b0cf3ba31646896ca9c84540a8cf8358943f92030cc06b87"
Nov 01 03:30:05.449306 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:30:05.449282    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e8c789-29891730-wqnkl_odh-ai-gateway-infra(c930267d-c979-4068-b469-17d331652e27)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e8c789-29891730-wqnkl" podUID="c930267d-c979-4068-b469-17d331652e27"
Nov 01 03:30:05.453478 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:05.453454    4930 scope.go:122] "RemoveContainer" containerID="f7845bc1851a6938b328a3d7ec7c75ec599793657e8cdd19d57369fb246012d6"
Nov 01 03:30:05.466003 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:05.465985    4930 scope.go:122] "RemoveContainer" containerID="b8afecf119ad779af2c8d8313974356ffd1ab1dd8141fd98074bea3642d6a8dd"
Nov 01 03:30:05.480110 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:05.479131    4930 scope.go:122] "RemoveContainer" containerID="9e3dbd31ed802e2478020fb684103194868b41ce59e9adde01112601d1dff3ee"
Nov 01 03:30:05.491847 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:05.491827    4930 scope.go:122] "RemoveContainer" containerID="a39e5b4ba897b029983740b0ff20ccd677adfdae9cf9e8d8b7d255f2e0442b21"
Nov 01 03:30:06.469827 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:06.469788    4930 scope.go:122] "RemoveContainer" containerID="b512ee46d0dad9679e450e4deec3d4d64e636cd6099e56490e13eef2cd8ae205"
Nov 01 03:30:06.470467 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:30:06.470135    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ded2d7-29891730-bmg7c_odh-ai-gateway-infra(d610889c-d7df-4ed2-9636-1ec134c05f81)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ded2d7-29891730-bmg7c" podUID="d610889c-d7df-4ed2-9636-1ec134c05f81"
Nov 01 03:30:06.471837 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:06.471814    4930 scope.go:122] "RemoveContainer" containerID="5b3b745ffb0fb425c437cd8f084700cf2858a6b7881ffa9c9025a4423ce91db6"
Nov 01 03:30:06.472200 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:30:06.472176    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891730-bzr6r_odh-ai-gateway-infra(9da05dfd-3a79-4c52-8ef7-f3b2c306673a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891730-bzr6r" podUID="9da05dfd-3a79-4c52-8ef7-f3b2c306673a"
Nov 01 03:30:06.473584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:06.473547    4930 scope.go:122] "RemoveContainer" containerID="31d9e98628b64eada8b26f4fdff6daf8ec15833c84362ea39acaf681039f5865"
Nov 01 03:30:06.473837 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:30:06.473810    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-7911e5a0-29891730-wjggk_odh-ai-gateway-infra(4a21bb3b-ad09-4622-b497-b385d320bf6a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7911e5a0-29891730-wjggk" podUID="4a21bb3b-ad09-4622-b497-b385d320bf6a"
Nov 01 03:30:06.475159 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:06.475144    4930 scope.go:122] "RemoveContainer" containerID="03cdfc444310303c9323a3386ec646c0f9e3c0318081d1b38df11ea97ba998af"
Nov 01 03:30:06.475376 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:30:06.475353    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891730-ts9lp_odh-ai-gateway-infra(be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891730-ts9lp" podUID="be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19"
Nov 01 03:30:06.476780 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:06.476760    4930 scope.go:122] "RemoveContainer" containerID="e76fe87a29a52bded9de3f86f62af7078c2e4a1e0e695cb1a91d3ea1e6fd842e"
Nov 01 03:30:06.476994 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:30:06.476975    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d7f535-29891730-m94pc_odh-ai-gateway-infra(6e8852d1-3681-4cc1-87f1-45f0ae2d1aec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891730-m94pc" podUID="6e8852d1-3681-4cc1-87f1-45f0ae2d1aec"
Nov 01 03:30:07.095781 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:07.095750    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:30:07.286806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:07.286768    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:30:12.051395 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:12.051364    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:30:15.619930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:15.619899    4930 scope.go:122] "RemoveContainer" containerID="e76fe87a29a52bded9de3f86f62af7078c2e4a1e0e695cb1a91d3ea1e6fd842e"
Nov 01 03:30:16.574642 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:16.574593    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891730-m94pc" event={"ID":"6e8852d1-3681-4cc1-87f1-45f0ae2d1aec","Type":"ContainerStarted","Data":"43167d335a4a78e6c2837f879f2d218b553dce87de782df863bf2b7e7b5c7c00"}
Nov 01 03:30:16.613722 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:16.613686    4930 scope.go:122] "RemoveContainer" containerID="03cdfc444310303c9323a3386ec646c0f9e3c0318081d1b38df11ea97ba998af"
Nov 01 03:30:16.614087 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:16.614062    4930 scope.go:122] "RemoveContainer" containerID="ff34ce83206047f6b0cf3ba31646896ca9c84540a8cf8358943f92030cc06b87"
Nov 01 03:30:16.945829 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:16.945793    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:30:17.597750 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:17.597713    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891730-ts9lp" event={"ID":"be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19","Type":"ContainerStarted","Data":"285e7b6e6716c48097faf7cc025becff8e886f7f2936331ccb642d6b93cff818"}
Nov 01 03:30:17.599980 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:17.599954    4930 generic.go:356] "Generic (PLEG): container finished" podID="6e8852d1-3681-4cc1-87f1-45f0ae2d1aec" containerID="43167d335a4a78e6c2837f879f2d218b553dce87de782df863bf2b7e7b5c7c00" exitCode=60
Nov 01 03:30:17.600119 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:17.600027    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891730-m94pc" event={"ID":"6e8852d1-3681-4cc1-87f1-45f0ae2d1aec","Type":"ContainerDied","Data":"43167d335a4a78e6c2837f879f2d218b553dce87de782df863bf2b7e7b5c7c00"}
Nov 01 03:30:17.600119 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:17.600091    4930 scope.go:122] "RemoveContainer" containerID="e76fe87a29a52bded9de3f86f62af7078c2e4a1e0e695cb1a91d3ea1e6fd842e"
Nov 01 03:30:17.600453 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:17.600430    4930 scope.go:122] "RemoveContainer" containerID="43167d335a4a78e6c2837f879f2d218b553dce87de782df863bf2b7e7b5c7c00"
Nov 01 03:30:17.600739 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:30:17.600715    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d7f535-29891730-m94pc_odh-ai-gateway-infra(6e8852d1-3681-4cc1-87f1-45f0ae2d1aec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891730-m94pc" podUID="6e8852d1-3681-4cc1-87f1-45f0ae2d1aec"
Nov 01 03:30:17.602499 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:17.602481    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e8c789-29891730-wqnkl" event={"ID":"c930267d-c979-4068-b469-17d331652e27","Type":"ContainerStarted","Data":"9db8e5ffe3b7aa89fb5c50962a368c462f5330ea4885b1a03432782c31db5a4e"}
Nov 01 03:30:17.613656 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:17.613635    4930 scope.go:122] "RemoveContainer" containerID="b512ee46d0dad9679e450e4deec3d4d64e636cd6099e56490e13eef2cd8ae205"
Nov 01 03:30:17.613804 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:17.613785    4930 scope.go:122] "RemoveContainer" containerID="5b3b745ffb0fb425c437cd8f084700cf2858a6b7881ffa9c9025a4423ce91db6"
Nov 01 03:30:17.705159 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:17.705131    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891730-m94pc"]
Nov 01 03:30:18.613303 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:18.613278    4930 scope.go:122] "RemoveContainer" containerID="31d9e98628b64eada8b26f4fdff6daf8ec15833c84362ea39acaf681039f5865"
Nov 01 03:30:18.614512 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:18.614484    4930 generic.go:356] "Generic (PLEG): container finished" podID="be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19" containerID="285e7b6e6716c48097faf7cc025becff8e886f7f2936331ccb642d6b93cff818" exitCode=60
Nov 01 03:30:18.614684 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:18.614593    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891730-ts9lp" event={"ID":"be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19","Type":"ContainerDied","Data":"285e7b6e6716c48097faf7cc025becff8e886f7f2936331ccb642d6b93cff818"}
Nov 01 03:30:18.614684 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:18.614659    4930 scope.go:122] "RemoveContainer" containerID="03cdfc444310303c9323a3386ec646c0f9e3c0318081d1b38df11ea97ba998af"
Nov 01 03:30:18.615152 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:18.614998    4930 scope.go:122] "RemoveContainer" containerID="285e7b6e6716c48097faf7cc025becff8e886f7f2936331ccb642d6b93cff818"
Nov 01 03:30:18.615295 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:30:18.615272    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891730-ts9lp_odh-ai-gateway-infra(be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891730-ts9lp" podUID="be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19"
Nov 01 03:30:18.618999 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:18.618967    4930 generic.go:356] "Generic (PLEG): container finished" podID="c930267d-c979-4068-b469-17d331652e27" containerID="9db8e5ffe3b7aa89fb5c50962a368c462f5330ea4885b1a03432782c31db5a4e" exitCode=60
Nov 01 03:30:18.619110 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:18.619042    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e8c789-29891730-wqnkl" event={"ID":"c930267d-c979-4068-b469-17d331652e27","Type":"ContainerDied","Data":"9db8e5ffe3b7aa89fb5c50962a368c462f5330ea4885b1a03432782c31db5a4e"}
Nov 01 03:30:18.619409 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:18.619389    4930 scope.go:122] "RemoveContainer" containerID="9db8e5ffe3b7aa89fb5c50962a368c462f5330ea4885b1a03432782c31db5a4e"
Nov 01 03:30:18.619827 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:30:18.619687    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-e8c789-29891730-wqnkl_odh-ai-gateway-infra(c930267d-c979-4068-b469-17d331652e27)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e8c789-29891730-wqnkl" podUID="c930267d-c979-4068-b469-17d331652e27"
Nov 01 03:30:18.621409 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:18.621391    4930 generic.go:356] "Generic (PLEG): container finished" podID="d610889c-d7df-4ed2-9636-1ec134c05f81" containerID="e49438f86aa208cfdb8160ef4ed3b5190cef8067e9f804ec7505aed9efc0aae5" exitCode=60
Nov 01 03:30:18.621511 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:18.621454    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ded2d7-29891730-bmg7c" event={"ID":"d610889c-d7df-4ed2-9636-1ec134c05f81","Type":"ContainerDied","Data":"e49438f86aa208cfdb8160ef4ed3b5190cef8067e9f804ec7505aed9efc0aae5"}
Nov 01 03:30:18.621767 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:18.621749    4930 scope.go:122] "RemoveContainer" containerID="e49438f86aa208cfdb8160ef4ed3b5190cef8067e9f804ec7505aed9efc0aae5"
Nov 01 03:30:18.621994 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:30:18.621975    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ded2d7-29891730-bmg7c_odh-ai-gateway-infra(d610889c-d7df-4ed2-9636-1ec134c05f81)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ded2d7-29891730-bmg7c" podUID="d610889c-d7df-4ed2-9636-1ec134c05f81"
Nov 01 03:30:18.623612 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:18.623532    4930 generic.go:356] "Generic (PLEG): container finished" podID="9da05dfd-3a79-4c52-8ef7-f3b2c306673a" containerID="4fd7fdbf6d52a69a52c067ed9b3a6f2b131053710538aaffd7e6908dc11c9719" exitCode=60
Nov 01 03:30:18.623612 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:18.623585    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891730-bzr6r" event={"ID":"9da05dfd-3a79-4c52-8ef7-f3b2c306673a","Type":"ContainerDied","Data":"4fd7fdbf6d52a69a52c067ed9b3a6f2b131053710538aaffd7e6908dc11c9719"}
Nov 01 03:30:18.623853 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:18.623840    4930 scope.go:122] "RemoveContainer" containerID="4fd7fdbf6d52a69a52c067ed9b3a6f2b131053710538aaffd7e6908dc11c9719"
Nov 01 03:30:18.624038 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:30:18.624017    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891730-bzr6r_odh-ai-gateway-infra(9da05dfd-3a79-4c52-8ef7-f3b2c306673a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891730-bzr6r" podUID="9da05dfd-3a79-4c52-8ef7-f3b2c306673a"
Nov 01 03:30:18.639569 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:18.639309    4930 scope.go:122] "RemoveContainer" containerID="ff34ce83206047f6b0cf3ba31646896ca9c84540a8cf8358943f92030cc06b87"
Nov 01 03:30:18.730804 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:18.730772    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891730-ts9lp"]
Nov 01 03:30:18.737849 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:18.737827    4930 scope.go:122] "RemoveContainer" containerID="b512ee46d0dad9679e450e4deec3d4d64e636cd6099e56490e13eef2cd8ae205"
Nov 01 03:30:18.754196 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:18.754175    4930 scope.go:122] "RemoveContainer" containerID="5b3b745ffb0fb425c437cd8f084700cf2858a6b7881ffa9c9025a4423ce91db6"
Nov 01 03:30:18.803806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:18.800959    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891730-m94pc"
Nov 01 03:30:18.919596 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:18.919542    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e8852d1-3681-4cc1-87f1-45f0ae2d1aec-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e8852d1-3681-4cc1-87f1-45f0ae2d1aec-maas-api-service-ca\") pod \"6e8852d1-3681-4cc1-87f1-45f0ae2d1aec\" (UID: \"6e8852d1-3681-4cc1-87f1-45f0ae2d1aec\") "
Nov 01 03:30:18.919799 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:18.919779    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e8852d1-3681-4cc1-87f1-45f0ae2d1aec-kube-api-access-zc47q\" (UniqueName: \"kubernetes.io/projected/6e8852d1-3681-4cc1-87f1-45f0ae2d1aec-kube-api-access-zc47q\") pod \"6e8852d1-3681-4cc1-87f1-45f0ae2d1aec\" (UID: \"6e8852d1-3681-4cc1-87f1-45f0ae2d1aec\") "
Nov 01 03:30:18.919983 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:18.919955    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e8852d1-3681-4cc1-87f1-45f0ae2d1aec-maas-api-service-ca" pod "6e8852d1-3681-4cc1-87f1-45f0ae2d1aec" (UID: "6e8852d1-3681-4cc1-87f1-45f0ae2d1aec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:30:18.920412 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:18.920392    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e8852d1-3681-4cc1-87f1-45f0ae2d1aec-maas-api-service-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:30:18.922685 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:18.922655    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e8852d1-3681-4cc1-87f1-45f0ae2d1aec-kube-api-access-zc47q" pod "6e8852d1-3681-4cc1-87f1-45f0ae2d1aec" (UID: "6e8852d1-3681-4cc1-87f1-45f0ae2d1aec"). InnerVolumeSpecName "kube-api-access-zc47q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:30:18.957164 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:18.957136    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e8c789-29891730-wqnkl"]
Nov 01 03:30:19.021809 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:19.021779    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc47q\" (UniqueName: \"kubernetes.io/projected/6e8852d1-3681-4cc1-87f1-45f0ae2d1aec-kube-api-access-zc47q\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:30:19.124629 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:30:19.124593    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:30:19.649664 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:19.649635    4930 generic.go:356] "Generic (PLEG): container finished" podID="4a21bb3b-ad09-4622-b497-b385d320bf6a" containerID="a0be6f7471c88a8fb7a9b36be0db2f956bd3235bc43c106256fce493368daf79" exitCode=60
Nov 01 03:30:19.650065 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:19.649702    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7911e5a0-29891730-wjggk" event={"ID":"4a21bb3b-ad09-4622-b497-b385d320bf6a","Type":"ContainerDied","Data":"a0be6f7471c88a8fb7a9b36be0db2f956bd3235bc43c106256fce493368daf79"}
Nov 01 03:30:19.650065 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:19.649790    4930 scope.go:122] "RemoveContainer" containerID="31d9e98628b64eada8b26f4fdff6daf8ec15833c84362ea39acaf681039f5865"
Nov 01 03:30:19.650227 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:19.650207    4930 scope.go:122] "RemoveContainer" containerID="a0be6f7471c88a8fb7a9b36be0db2f956bd3235bc43c106256fce493368daf79"
Nov 01 03:30:19.650864 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:30:19.650537    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-7911e5a0-29891730-wjggk_odh-ai-gateway-infra(4a21bb3b-ad09-4622-b497-b385d320bf6a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7911e5a0-29891730-wjggk" podUID="4a21bb3b-ad09-4622-b497-b385d320bf6a"
Nov 01 03:30:19.653543 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:19.653520    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891730-m94pc" event={"ID":"6e8852d1-3681-4cc1-87f1-45f0ae2d1aec","Type":"ContainerDied","Data":"768accfeb7814228c2987a25a7ae832cf6dbd3d46531d6d9b1c66e80c6970832"}
Nov 01 03:30:19.653669 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:19.653527    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891730-m94pc"
Nov 01 03:30:19.664925 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:19.664902    4930 scope.go:122] "RemoveContainer" containerID="43167d335a4a78e6c2837f879f2d218b553dce87de782df863bf2b7e7b5c7c00"
Nov 01 03:30:19.899153 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:19.899130    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e8c789-29891730-wqnkl"
Nov 01 03:30:19.919211 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:19.919188    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891730-m94pc"]
Nov 01 03:30:19.927849 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:19.927828    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891730-ts9lp"
Nov 01 03:30:19.977488 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:19.977403    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891730-m94pc"]
Nov 01 03:30:19.997942 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:19.997910    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ded2d7-29891730-bmg7c"]
Nov 01 03:30:20.040091 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.039875    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c930267d-c979-4068-b469-17d331652e27-kube-api-access-c54k2\" (UniqueName: \"kubernetes.io/projected/c930267d-c979-4068-b469-17d331652e27-kube-api-access-c54k2\") pod \"c930267d-c979-4068-b469-17d331652e27\" (UID: \"c930267d-c979-4068-b469-17d331652e27\") "
Nov 01 03:30:20.040091 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.039998    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19-maas-api-service-ca\") pod \"be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19\" (UID: \"be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19\") "
Nov 01 03:30:20.040091 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.040047    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19-kube-api-access-7dt49\" (UniqueName: \"kubernetes.io/projected/be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19-kube-api-access-7dt49\") pod \"be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19\" (UID: \"be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19\") "
Nov 01 03:30:20.040379 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.040190    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c930267d-c979-4068-b469-17d331652e27-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c930267d-c979-4068-b469-17d331652e27-maas-api-service-ca\") pod \"c930267d-c979-4068-b469-17d331652e27\" (UID: \"c930267d-c979-4068-b469-17d331652e27\") "
Nov 01 03:30:20.040621 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.040477    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19-maas-api-service-ca" pod "be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19" (UID: "be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:30:20.041186 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.041160    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19-maas-api-service-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:30:20.041318 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.041291    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c930267d-c979-4068-b469-17d331652e27-maas-api-service-ca" pod "c930267d-c979-4068-b469-17d331652e27" (UID: "c930267d-c979-4068-b469-17d331652e27"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:30:20.043179 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.043155    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19-kube-api-access-7dt49" pod "be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19" (UID: "be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19"). InnerVolumeSpecName "kube-api-access-7dt49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:30:20.045231 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.045208    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c930267d-c979-4068-b469-17d331652e27-kube-api-access-c54k2" pod "c930267d-c979-4068-b469-17d331652e27" (UID: "c930267d-c979-4068-b469-17d331652e27"). InnerVolumeSpecName "kube-api-access-c54k2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:30:20.143823 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.143738    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c930267d-c979-4068-b469-17d331652e27-maas-api-service-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:30:20.143823 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.143777    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c54k2\" (UniqueName: \"kubernetes.io/projected/c930267d-c979-4068-b469-17d331652e27-kube-api-access-c54k2\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:30:20.143823 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.143791    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dt49\" (UniqueName: \"kubernetes.io/projected/be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19-kube-api-access-7dt49\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:30:20.210497 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.210465    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ded2d7-29891730-bmg7c"
Nov 01 03:30:20.233889 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.233857    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891730-bzr6r"]
Nov 01 03:30:20.350979 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.347716    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d610889c-d7df-4ed2-9636-1ec134c05f81-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d610889c-d7df-4ed2-9636-1ec134c05f81-maas-api-service-ca\") pod \"d610889c-d7df-4ed2-9636-1ec134c05f81\" (UID: \"d610889c-d7df-4ed2-9636-1ec134c05f81\") "
Nov 01 03:30:20.350979 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.347860    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d610889c-d7df-4ed2-9636-1ec134c05f81-kube-api-access-9mqw9\" (UniqueName: \"kubernetes.io/projected/d610889c-d7df-4ed2-9636-1ec134c05f81-kube-api-access-9mqw9\") pod \"d610889c-d7df-4ed2-9636-1ec134c05f81\" (UID: \"d610889c-d7df-4ed2-9636-1ec134c05f81\") "
Nov 01 03:30:20.350979 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.349471    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d610889c-d7df-4ed2-9636-1ec134c05f81-maas-api-service-ca" pod "d610889c-d7df-4ed2-9636-1ec134c05f81" (UID: "d610889c-d7df-4ed2-9636-1ec134c05f81"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:30:20.356044 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.355996    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d610889c-d7df-4ed2-9636-1ec134c05f81-kube-api-access-9mqw9" pod "d610889c-d7df-4ed2-9636-1ec134c05f81" (UID: "d610889c-d7df-4ed2-9636-1ec134c05f81"). InnerVolumeSpecName "kube-api-access-9mqw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:30:20.395273 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.395217    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891730-bzr6r"
Nov 01 03:30:20.449653 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.449620    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9da05dfd-3a79-4c52-8ef7-f3b2c306673a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9da05dfd-3a79-4c52-8ef7-f3b2c306673a-maas-api-service-ca\") pod \"9da05dfd-3a79-4c52-8ef7-f3b2c306673a\" (UID: \"9da05dfd-3a79-4c52-8ef7-f3b2c306673a\") "
Nov 01 03:30:20.449825 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.449759    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9da05dfd-3a79-4c52-8ef7-f3b2c306673a-kube-api-access-4zpjg\" (UniqueName: \"kubernetes.io/projected/9da05dfd-3a79-4c52-8ef7-f3b2c306673a-kube-api-access-4zpjg\") pod \"9da05dfd-3a79-4c52-8ef7-f3b2c306673a\" (UID: \"9da05dfd-3a79-4c52-8ef7-f3b2c306673a\") "
Nov 01 03:30:20.450081 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.450026    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9da05dfd-3a79-4c52-8ef7-f3b2c306673a-maas-api-service-ca" pod "9da05dfd-3a79-4c52-8ef7-f3b2c306673a" (UID: "9da05dfd-3a79-4c52-8ef7-f3b2c306673a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:30:20.450406 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.450388    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9da05dfd-3a79-4c52-8ef7-f3b2c306673a-maas-api-service-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:30:20.450491 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.450410    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d610889c-d7df-4ed2-9636-1ec134c05f81-maas-api-service-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:30:20.450491 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.450425    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mqw9\" (UniqueName: \"kubernetes.io/projected/d610889c-d7df-4ed2-9636-1ec134c05f81-kube-api-access-9mqw9\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:30:20.451989 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.451968    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9da05dfd-3a79-4c52-8ef7-f3b2c306673a-kube-api-access-4zpjg" pod "9da05dfd-3a79-4c52-8ef7-f3b2c306673a" (UID: "9da05dfd-3a79-4c52-8ef7-f3b2c306673a"). InnerVolumeSpecName "kube-api-access-4zpjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:30:20.543014 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.542906    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55"]
Nov 01 03:30:20.543987 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.543960    4930 cpu_manager.go:414] "containerMap: removing container" podUID="9da05dfd-3a79-4c52-8ef7-f3b2c306673a" containerName="cleanup"
Nov 01 03:30:20.543987 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.543987    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9da05dfd-3a79-4c52-8ef7-f3b2c306673a" containerName="cleanup"
Nov 01 03:30:20.544183 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.544005    4930 cpu_manager.go:414] "containerMap: removing container" podUID="be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19" containerName="cleanup"
Nov 01 03:30:20.544183 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.544014    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19" containerName="cleanup"
Nov 01 03:30:20.544183 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.544031    4930 cpu_manager.go:414] "containerMap: removing container" podUID="9da05dfd-3a79-4c52-8ef7-f3b2c306673a" containerName="cleanup"
Nov 01 03:30:20.544183 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.544040    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9da05dfd-3a79-4c52-8ef7-f3b2c306673a" containerName="cleanup"
Nov 01 03:30:20.544183 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.544062    4930 cpu_manager.go:414] "containerMap: removing container" podUID="c930267d-c979-4068-b469-17d331652e27" containerName="cleanup"
Nov 01 03:30:20.544183 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.544069    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c930267d-c979-4068-b469-17d331652e27" containerName="cleanup"
Nov 01 03:30:20.544183 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.544086    4930 cpu_manager.go:414] "containerMap: removing container" podUID="6e8852d1-3681-4cc1-87f1-45f0ae2d1aec" containerName="cleanup"
Nov 01 03:30:20.544183 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.544094    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e8852d1-3681-4cc1-87f1-45f0ae2d1aec" containerName="cleanup"
Nov 01 03:30:20.544183 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.544108    4930 cpu_manager.go:414] "containerMap: removing container" podUID="d610889c-d7df-4ed2-9636-1ec134c05f81" containerName="cleanup"
Nov 01 03:30:20.544183 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.544116    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d610889c-d7df-4ed2-9636-1ec134c05f81" containerName="cleanup"
Nov 01 03:30:20.544183 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.544129    4930 cpu_manager.go:414] "containerMap: removing container" podUID="9da05dfd-3a79-4c52-8ef7-f3b2c306673a" containerName="cleanup"
Nov 01 03:30:20.544183 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.544136    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9da05dfd-3a79-4c52-8ef7-f3b2c306673a" containerName="cleanup"
Nov 01 03:30:20.544183 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.544157    4930 cpu_manager.go:414] "containerMap: removing container" podUID="d610889c-d7df-4ed2-9636-1ec134c05f81" containerName="cleanup"
Nov 01 03:30:20.544183 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.544164    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d610889c-d7df-4ed2-9636-1ec134c05f81" containerName="cleanup"
Nov 01 03:30:20.544862 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.544196    4930 cpu_manager.go:414] "containerMap: removing container" podUID="c930267d-c979-4068-b469-17d331652e27" containerName="cleanup"
Nov 01 03:30:20.544862 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.544203    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c930267d-c979-4068-b469-17d331652e27" containerName="cleanup"
Nov 01 03:30:20.544862 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.544213    4930 cpu_manager.go:414] "containerMap: removing container" podUID="6e8852d1-3681-4cc1-87f1-45f0ae2d1aec" containerName="cleanup"
Nov 01 03:30:20.544862 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.544220    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e8852d1-3681-4cc1-87f1-45f0ae2d1aec" containerName="cleanup"
Nov 01 03:30:20.544862 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.544253    4930 cpu_manager.go:414] "containerMap: removing container" podUID="d610889c-d7df-4ed2-9636-1ec134c05f81" containerName="cleanup"
Nov 01 03:30:20.544862 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.544260    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d610889c-d7df-4ed2-9636-1ec134c05f81" containerName="cleanup"
Nov 01 03:30:20.544862 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.544272    4930 cpu_manager.go:414] "containerMap: removing container" podUID="be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19" containerName="cleanup"
Nov 01 03:30:20.544862 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.544280    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19" containerName="cleanup"
Nov 01 03:30:20.544862 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.544436    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="9da05dfd-3a79-4c52-8ef7-f3b2c306673a" containerName="cleanup"
Nov 01 03:30:20.544862 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.544449    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="c930267d-c979-4068-b469-17d331652e27" containerName="cleanup"
Nov 01 03:30:20.544862 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.544458    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e8852d1-3681-4cc1-87f1-45f0ae2d1aec" containerName="cleanup"
Nov 01 03:30:20.544862 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.544467    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="d610889c-d7df-4ed2-9636-1ec134c05f81" containerName="cleanup"
Nov 01 03:30:20.544862 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.544478    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="c930267d-c979-4068-b469-17d331652e27" containerName="cleanup"
Nov 01 03:30:20.544862 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.544490    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e8852d1-3681-4cc1-87f1-45f0ae2d1aec" containerName="cleanup"
Nov 01 03:30:20.544862 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.544502    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="c930267d-c979-4068-b469-17d331652e27" containerName="cleanup"
Nov 01 03:30:20.544862 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.544510    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e8852d1-3681-4cc1-87f1-45f0ae2d1aec" containerName="cleanup"
Nov 01 03:30:20.544862 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.544520    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19" containerName="cleanup"
Nov 01 03:30:20.544862 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.544530    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="d610889c-d7df-4ed2-9636-1ec134c05f81" containerName="cleanup"
Nov 01 03:30:20.544862 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.544540    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="9da05dfd-3a79-4c52-8ef7-f3b2c306673a" containerName="cleanup"
Nov 01 03:30:20.544862 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.544550    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19" containerName="cleanup"
Nov 01 03:30:20.548386 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.548361    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55"
Nov 01 03:30:20.552826 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.552800    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zpjg\" (UniqueName: \"kubernetes.io/projected/9da05dfd-3a79-4c52-8ef7-f3b2c306673a-kube-api-access-4zpjg\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:30:20.552962 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.552838    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7"]
Nov 01 03:30:20.553714 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.553690    4930 cpu_manager.go:414] "containerMap: removing container" podUID="6e8852d1-3681-4cc1-87f1-45f0ae2d1aec" containerName="cleanup"
Nov 01 03:30:20.553821 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.553714    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e8852d1-3681-4cc1-87f1-45f0ae2d1aec" containerName="cleanup"
Nov 01 03:30:20.553821 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.553758    4930 cpu_manager.go:414] "containerMap: removing container" podUID="be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19" containerName="cleanup"
Nov 01 03:30:20.553821 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.553766    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19" containerName="cleanup"
Nov 01 03:30:20.553821 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.553791    4930 cpu_manager.go:414] "containerMap: removing container" podUID="c930267d-c979-4068-b469-17d331652e27" containerName="cleanup"
Nov 01 03:30:20.553821 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.553798    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c930267d-c979-4068-b469-17d331652e27" containerName="cleanup"
Nov 01 03:30:20.554009 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.553979    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19" containerName="cleanup"
Nov 01 03:30:20.554009 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.553992    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="9da05dfd-3a79-4c52-8ef7-f3b2c306673a" containerName="cleanup"
Nov 01 03:30:20.554009 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.554007    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="d610889c-d7df-4ed2-9636-1ec134c05f81" containerName="cleanup"
Nov 01 03:30:20.557225 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.557204    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7"
Nov 01 03:30:20.603938 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.603798    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-v7nmz\""
Nov 01 03:30:20.606881 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.606852    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55"]
Nov 01 03:30:20.619699 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.619669    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7"]
Nov 01 03:30:20.654166 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.654069    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4322b03e-6c6d-435d-bf3b-9e60757a5845-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cchc7\" (UID: \"4322b03e-6c6d-435d-bf3b-9e60757a5845\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7"
Nov 01 03:30:20.654166 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.654102    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-swk55\" (UID: \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55"
Nov 01 03:30:20.654166 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.654143    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-swk55\" (UID: \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55"
Nov 01 03:30:20.654803 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.654305    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7x48\" (UniqueName: \"kubernetes.io/projected/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-kube-api-access-l7x48\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-swk55\" (UID: \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55"
Nov 01 03:30:20.654803 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.654370    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4322b03e-6c6d-435d-bf3b-9e60757a5845-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cchc7\" (UID: \"4322b03e-6c6d-435d-bf3b-9e60757a5845\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7"
Nov 01 03:30:20.654803 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.654430    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-swk55\" (UID: \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55"
Nov 01 03:30:20.654803 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.654499    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbfhx\" (UniqueName: \"kubernetes.io/projected/4322b03e-6c6d-435d-bf3b-9e60757a5845-kube-api-access-xbfhx\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cchc7\" (UID: \"4322b03e-6c6d-435d-bf3b-9e60757a5845\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7"
Nov 01 03:30:20.654803 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.654576    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-swk55\" (UID: \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55"
Nov 01 03:30:20.654803 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.654624    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-swk55\" (UID: \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55"
Nov 01 03:30:20.654803 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.654681    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-swk55\" (UID: \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55"
Nov 01 03:30:20.654803 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.654700    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4322b03e-6c6d-435d-bf3b-9e60757a5845-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cchc7\" (UID: \"4322b03e-6c6d-435d-bf3b-9e60757a5845\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7"
Nov 01 03:30:20.654803 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.654791    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4322b03e-6c6d-435d-bf3b-9e60757a5845-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cchc7\" (UID: \"4322b03e-6c6d-435d-bf3b-9e60757a5845\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7"
Nov 01 03:30:20.655287 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.654865    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4322b03e-6c6d-435d-bf3b-9e60757a5845-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cchc7\" (UID: \"4322b03e-6c6d-435d-bf3b-9e60757a5845\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7"
Nov 01 03:30:20.655287 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.654903    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-swk55\" (UID: \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55"
Nov 01 03:30:20.655287 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.654950    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4322b03e-6c6d-435d-bf3b-9e60757a5845-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cchc7\" (UID: \"4322b03e-6c6d-435d-bf3b-9e60757a5845\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7"
Nov 01 03:30:20.655287 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.654986    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-swk55\" (UID: \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55"
Nov 01 03:30:20.655287 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.655088    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4322b03e-6c6d-435d-bf3b-9e60757a5845-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cchc7\" (UID: \"4322b03e-6c6d-435d-bf3b-9e60757a5845\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7"
Nov 01 03:30:20.655287 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.655123    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4322b03e-6c6d-435d-bf3b-9e60757a5845-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cchc7\" (UID: \"4322b03e-6c6d-435d-bf3b-9e60757a5845\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7"
Nov 01 03:30:20.668185 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.668155    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891730-ts9lp" event={"ID":"be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19","Type":"ContainerDied","Data":"bcf48861691580a045549b5ad7a69868cbf06371b69be970921df528e14c4d44"}
Nov 01 03:30:20.668384 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.668229    4930 scope.go:122] "RemoveContainer" containerID="285e7b6e6716c48097faf7cc025becff8e886f7f2936331ccb642d6b93cff818"
Nov 01 03:30:20.668384 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.668164    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891730-ts9lp"
Nov 01 03:30:20.671204 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.671187    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e8c789-29891730-wqnkl"
Nov 01 03:30:20.671342 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.671236    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e8c789-29891730-wqnkl" event={"ID":"c930267d-c979-4068-b469-17d331652e27","Type":"ContainerDied","Data":"30d2fb7403751e2994692eafe5eadbb945cf10d9bfcab7cc95679b1cd39c8a2f"}
Nov 01 03:30:20.673036 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.673018    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ded2d7-29891730-bmg7c"
Nov 01 03:30:20.673036 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.673020    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ded2d7-29891730-bmg7c" event={"ID":"d610889c-d7df-4ed2-9636-1ec134c05f81","Type":"ContainerDied","Data":"f2487744cbdbb2c883204af3d704019fcf75012d6fcb923e43e9e368e4a96511"}
Nov 01 03:30:20.674845 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.674816    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891730-bzr6r"
Nov 01 03:30:20.674954 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.674837    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891730-bzr6r" event={"ID":"9da05dfd-3a79-4c52-8ef7-f3b2c306673a","Type":"ContainerDied","Data":"60247faf01c3aa2252716f16911f7c09a123cdb4463379b511e068b1d8a50c80"}
Nov 01 03:30:20.687399 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.687376    4930 scope.go:122] "RemoveContainer" containerID="9db8e5ffe3b7aa89fb5c50962a368c462f5330ea4885b1a03432782c31db5a4e"
Nov 01 03:30:20.700516 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.700493    4930 scope.go:122] "RemoveContainer" containerID="e49438f86aa208cfdb8160ef4ed3b5190cef8067e9f804ec7505aed9efc0aae5"
Nov 01 03:30:20.712009 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.711993    4930 scope.go:122] "RemoveContainer" containerID="4fd7fdbf6d52a69a52c067ed9b3a6f2b131053710538aaffd7e6908dc11c9719"
Nov 01 03:30:20.756875 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.756848    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-swk55\" (UID: \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55"
Nov 01 03:30:20.757011 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.756896    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbfhx\" (UniqueName: \"kubernetes.io/projected/4322b03e-6c6d-435d-bf3b-9e60757a5845-kube-api-access-xbfhx\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cchc7\" (UID: \"4322b03e-6c6d-435d-bf3b-9e60757a5845\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7"
Nov 01 03:30:20.757011 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.756955    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-swk55\" (UID: \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55"
Nov 01 03:30:20.757011 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.756992    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-swk55\" (UID: \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55"
Nov 01 03:30:20.757204 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.757087    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-swk55\" (UID: \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55"
Nov 01 03:30:20.757204 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.757122    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4322b03e-6c6d-435d-bf3b-9e60757a5845-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cchc7\" (UID: \"4322b03e-6c6d-435d-bf3b-9e60757a5845\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7"
Nov 01 03:30:20.757204 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.757161    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4322b03e-6c6d-435d-bf3b-9e60757a5845-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cchc7\" (UID: \"4322b03e-6c6d-435d-bf3b-9e60757a5845\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7"
Nov 01 03:30:20.757365 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.757256    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4322b03e-6c6d-435d-bf3b-9e60757a5845-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cchc7\" (UID: \"4322b03e-6c6d-435d-bf3b-9e60757a5845\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7"
Nov 01 03:30:20.757365 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.757255    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-swk55\" (UID: \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55"
Nov 01 03:30:20.757365 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.757297    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-swk55\" (UID: \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55"
Nov 01 03:30:20.757507 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.757369    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4322b03e-6c6d-435d-bf3b-9e60757a5845-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cchc7\" (UID: \"4322b03e-6c6d-435d-bf3b-9e60757a5845\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7"
Nov 01 03:30:20.757507 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.757429    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-swk55\" (UID: \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55"
Nov 01 03:30:20.757507 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.757448    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-swk55\" (UID: \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55"
Nov 01 03:30:20.757507 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.757438    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-swk55\" (UID: \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55"
Nov 01 03:30:20.757747 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.757511    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4322b03e-6c6d-435d-bf3b-9e60757a5845-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cchc7\" (UID: \"4322b03e-6c6d-435d-bf3b-9e60757a5845\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7"
Nov 01 03:30:20.757747 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.757539    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4322b03e-6c6d-435d-bf3b-9e60757a5845-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cchc7\" (UID: \"4322b03e-6c6d-435d-bf3b-9e60757a5845\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7"
Nov 01 03:30:20.757747 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.757733    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4322b03e-6c6d-435d-bf3b-9e60757a5845-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cchc7\" (UID: \"4322b03e-6c6d-435d-bf3b-9e60757a5845\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7"
Nov 01 03:30:20.757896 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.757793    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4322b03e-6c6d-435d-bf3b-9e60757a5845-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cchc7\" (UID: \"4322b03e-6c6d-435d-bf3b-9e60757a5845\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7"
Nov 01 03:30:20.757896 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.757821    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-swk55\" (UID: \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55"
Nov 01 03:30:20.757997 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.757963    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-swk55\" (UID: \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55"
Nov 01 03:30:20.758064 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.758003    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-swk55\" (UID: \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55"
Nov 01 03:30:20.758064 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.758034    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7x48\" (UniqueName: \"kubernetes.io/projected/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-kube-api-access-l7x48\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-swk55\" (UID: \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55"
Nov 01 03:30:20.758173 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.758089    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4322b03e-6c6d-435d-bf3b-9e60757a5845-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cchc7\" (UID: \"4322b03e-6c6d-435d-bf3b-9e60757a5845\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7"
Nov 01 03:30:20.758621 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.758334    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4322b03e-6c6d-435d-bf3b-9e60757a5845-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cchc7\" (UID: \"4322b03e-6c6d-435d-bf3b-9e60757a5845\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7"
Nov 01 03:30:20.758621 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.758348    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4322b03e-6c6d-435d-bf3b-9e60757a5845-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cchc7\" (UID: \"4322b03e-6c6d-435d-bf3b-9e60757a5845\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7"
Nov 01 03:30:20.758621 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.758591    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4322b03e-6c6d-435d-bf3b-9e60757a5845-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cchc7\" (UID: \"4322b03e-6c6d-435d-bf3b-9e60757a5845\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7"
Nov 01 03:30:20.758853 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.758784    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-swk55\" (UID: \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55"
Nov 01 03:30:20.758970 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.758946    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4322b03e-6c6d-435d-bf3b-9e60757a5845-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cchc7\" (UID: \"4322b03e-6c6d-435d-bf3b-9e60757a5845\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7"
Nov 01 03:30:20.760060 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.760035    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4322b03e-6c6d-435d-bf3b-9e60757a5845-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cchc7\" (UID: \"4322b03e-6c6d-435d-bf3b-9e60757a5845\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7"
Nov 01 03:30:20.760550 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.760524    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-swk55\" (UID: \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55"
Nov 01 03:30:20.760685 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.760666    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4322b03e-6c6d-435d-bf3b-9e60757a5845-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cchc7\" (UID: \"4322b03e-6c6d-435d-bf3b-9e60757a5845\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7"
Nov 01 03:30:20.760883 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.760863    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-swk55\" (UID: \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55"
Nov 01 03:30:20.780592 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.780565    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7911e5a0-29891730-wjggk"]
Nov 01 03:30:20.821197 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.821159    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbfhx\" (UniqueName: \"kubernetes.io/projected/4322b03e-6c6d-435d-bf3b-9e60757a5845-kube-api-access-xbfhx\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cchc7\" (UID: \"4322b03e-6c6d-435d-bf3b-9e60757a5845\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7"
Nov 01 03:30:20.822770 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.822709    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-swk55\" (UID: \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55"
Nov 01 03:30:20.823573 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.823527    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4322b03e-6c6d-435d-bf3b-9e60757a5845-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cchc7\" (UID: \"4322b03e-6c6d-435d-bf3b-9e60757a5845\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7"
Nov 01 03:30:20.824450 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.824425    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7x48\" (UniqueName: \"kubernetes.io/projected/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-kube-api-access-l7x48\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-swk55\" (UID: \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55"
Nov 01 03:30:20.865583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.865537    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55"
Nov 01 03:30:20.881179 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:20.880917    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7"
Nov 01 03:30:21.068820 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:21.068785    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7911e5a0-29891730-wjggk"
Nov 01 03:30:21.100909 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:21.100874    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891730-ts9lp"]
Nov 01 03:30:21.109612 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:21.109577    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891730-ts9lp"]
Nov 01 03:30:21.165319 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:21.165291    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a21bb3b-ad09-4622-b497-b385d320bf6a-kube-api-access-4gzmc\" (UniqueName: \"kubernetes.io/projected/4a21bb3b-ad09-4622-b497-b385d320bf6a-kube-api-access-4gzmc\") pod \"4a21bb3b-ad09-4622-b497-b385d320bf6a\" (UID: \"4a21bb3b-ad09-4622-b497-b385d320bf6a\") "
Nov 01 03:30:21.165496 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:21.165480    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a21bb3b-ad09-4622-b497-b385d320bf6a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a21bb3b-ad09-4622-b497-b385d320bf6a-maas-api-service-ca\") pod \"4a21bb3b-ad09-4622-b497-b385d320bf6a\" (UID: \"4a21bb3b-ad09-4622-b497-b385d320bf6a\") "
Nov 01 03:30:21.165868 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:21.165845    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a21bb3b-ad09-4622-b497-b385d320bf6a-maas-api-service-ca" pod "4a21bb3b-ad09-4622-b497-b385d320bf6a" (UID: "4a21bb3b-ad09-4622-b497-b385d320bf6a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:30:21.166202 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:21.166186    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a21bb3b-ad09-4622-b497-b385d320bf6a-maas-api-service-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:30:21.167440 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:21.167419    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a21bb3b-ad09-4622-b497-b385d320bf6a-kube-api-access-4gzmc" pod "4a21bb3b-ad09-4622-b497-b385d320bf6a" (UID: "4a21bb3b-ad09-4622-b497-b385d320bf6a"). InnerVolumeSpecName "kube-api-access-4gzmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:30:21.211095 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:21.211068    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ded2d7-29891730-bmg7c"]
Nov 01 03:30:21.218081 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:21.218056    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ded2d7-29891730-bmg7c"]
Nov 01 03:30:21.267267 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:21.267208    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gzmc\" (UniqueName: \"kubernetes.io/projected/4a21bb3b-ad09-4622-b497-b385d320bf6a-kube-api-access-4gzmc\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:30:21.326503 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:30:21.326477    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab72eb99_1ab5_4e1d_8057_82feb8b67a66.slice/crio-4c7ed8bf62de2685f0b0fe65efe9fcdd44949fd626454a927a123f79cc6640e3 WatchSource:0}: Error finding container 4c7ed8bf62de2685f0b0fe65efe9fcdd44949fd626454a927a123f79cc6640e3: Status 404 returned error can't find the container with id 4c7ed8bf62de2685f0b0fe65efe9fcdd44949fd626454a927a123f79cc6640e3
Nov 01 03:30:21.327457 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:21.327320    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55"]
Nov 01 03:30:21.329636 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:21.329600    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:30:21.329736 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:21.329669    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:30:21.329736 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:21.329698    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:30:21.333595 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:21.333532    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e8c789-29891730-wqnkl"]
Nov 01 03:30:21.338467 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:21.338442    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-e8c789-29891730-wqnkl"]
Nov 01 03:30:21.371320 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:30:21.371293    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4322b03e_6c6d_435d_bf3b_9e60757a5845.slice/crio-6d5f17dc9e8e0da3f9955a5d1d0304278625c79c678802ba7638e63c93e4c229 WatchSource:0}: Error finding container 6d5f17dc9e8e0da3f9955a5d1d0304278625c79c678802ba7638e63c93e4c229: Status 404 returned error can't find the container with id 6d5f17dc9e8e0da3f9955a5d1d0304278625c79c678802ba7638e63c93e4c229
Nov 01 03:30:21.373789 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:21.373757    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7"]
Nov 01 03:30:21.374083 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:21.374055    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:30:21.374172 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:21.374131    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:30:21.374238 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:21.374177    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:30:21.443868 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:21.443843    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891730-bzr6r"]
Nov 01 03:30:21.449611 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:21.449585    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891730-bzr6r"]
Nov 01 03:30:21.625623 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:21.625587    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e8852d1-3681-4cc1-87f1-45f0ae2d1aec" path="/var/lib/kubelet/pods/6e8852d1-3681-4cc1-87f1-45f0ae2d1aec/volumes"
Nov 01 03:30:21.626173 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:21.626152    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9da05dfd-3a79-4c52-8ef7-f3b2c306673a" path="/var/lib/kubelet/pods/9da05dfd-3a79-4c52-8ef7-f3b2c306673a/volumes"
Nov 01 03:30:21.626664 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:21.626644    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19" path="/var/lib/kubelet/pods/be1ce526-8ffe-44cc-a9d3-4cc2b85b0c19/volumes"
Nov 01 03:30:21.627427 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:21.627405    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c930267d-c979-4068-b469-17d331652e27" path="/var/lib/kubelet/pods/c930267d-c979-4068-b469-17d331652e27/volumes"
Nov 01 03:30:21.627865 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:21.627848    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d610889c-d7df-4ed2-9636-1ec134c05f81" path="/var/lib/kubelet/pods/d610889c-d7df-4ed2-9636-1ec134c05f81/volumes"
Nov 01 03:30:21.698662 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:21.698621    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7" event={"ID":"4322b03e-6c6d-435d-bf3b-9e60757a5845","Type":"ContainerStarted","Data":"7e24f94513610119af73c3a07ae06698e072024f88709e07f4fb9a0ec7545d53"}
Nov 01 03:30:21.699110 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:21.698687    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7" event={"ID":"4322b03e-6c6d-435d-bf3b-9e60757a5845","Type":"ContainerStarted","Data":"6d5f17dc9e8e0da3f9955a5d1d0304278625c79c678802ba7638e63c93e4c229"}
Nov 01 03:30:21.707408 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:21.705077    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55" event={"ID":"ab72eb99-1ab5-4e1d-8057-82feb8b67a66","Type":"ContainerStarted","Data":"2870bd8b8a4485aa9c8a05375b8c7c290694e3d562c326993d227d1436ee5cd2"}
Nov 01 03:30:21.707408 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:21.705131    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55" event={"ID":"ab72eb99-1ab5-4e1d-8057-82feb8b67a66","Type":"ContainerStarted","Data":"4c7ed8bf62de2685f0b0fe65efe9fcdd44949fd626454a927a123f79cc6640e3"}
Nov 01 03:30:21.708739 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:21.707548    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7911e5a0-29891730-wjggk" event={"ID":"4a21bb3b-ad09-4622-b497-b385d320bf6a","Type":"ContainerDied","Data":"e15b1e2f6661e418b054797dba2a6cab791b792688af45369b6579f217fe5234"}
Nov 01 03:30:21.708739 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:21.707621    4930 scope.go:122] "RemoveContainer" containerID="a0be6f7471c88a8fb7a9b36be0db2f956bd3235bc43c106256fce493368daf79"
Nov 01 03:30:21.708739 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:21.707750    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7911e5a0-29891730-wjggk"
Nov 01 03:30:21.826427 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:21.826333    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7" podStartSLOduration=1.826320081 podStartE2EDuration="1.826320081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:30:21.818529543 +0000 UTC m=+1426.341536149" watchObservedRunningTime="2026-11-01 03:30:21.826320081 +0000 UTC m=+1426.349326668"
Nov 01 03:30:21.866913 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:21.866831    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55"
Nov 01 03:30:21.887018 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:21.886931    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7"
Nov 01 03:30:21.939500 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:21.939460    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7911e5a0-29891730-wjggk"]
Nov 01 03:30:21.943378 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:21.943355    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-7911e5a0-29891730-wjggk"]
Nov 01 03:30:22.059758 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:22.059694    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55" podStartSLOduration=2.059673591 podStartE2EDuration="2.059673591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:30:22.056696827 +0000 UTC m=+1426.579703417" watchObservedRunningTime="2026-11-01 03:30:22.059673591 +0000 UTC m=+1426.582680183"
Nov 01 03:30:22.892100 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:22.892060    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-swk55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 01 03:30:22.892758 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:22.892722    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55" podUID="ab72eb99-1ab5-4e1d-8057-82feb8b67a66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 01 03:30:22.896805 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:22.896669    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-cchc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 01 03:30:22.896805 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:22.896739    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7" podUID="4322b03e-6c6d-435d-bf3b-9e60757a5845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 01 03:30:23.013392 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:23.012929    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:30:23.632073 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:23.632040    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a21bb3b-ad09-4622-b497-b385d320bf6a" path="/var/lib/kubelet/pods/4a21bb3b-ad09-4622-b497-b385d320bf6a/volumes"
Nov 01 03:30:23.877025 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:23.876983    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-swk55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 01 03:30:23.877295 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:23.877262    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55" podUID="ab72eb99-1ab5-4e1d-8057-82feb8b67a66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 01 03:30:23.881909 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:23.881880    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-cchc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 01 03:30:23.882172 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:23.882112    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7" podUID="4322b03e-6c6d-435d-bf3b-9e60757a5845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 01 03:30:24.871585 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:24.868939    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-swk55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 01 03:30:24.871585 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:24.869010    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55" podUID="ab72eb99-1ab5-4e1d-8057-82feb8b67a66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 01 03:30:24.884006 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:24.883975    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-cchc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 01 03:30:24.884259 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:24.884230    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7" podUID="4322b03e-6c6d-435d-bf3b-9e60757a5845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 01 03:30:25.871933 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:25.871895    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-swk55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 01 03:30:25.872502 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:25.872467    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55" podUID="ab72eb99-1ab5-4e1d-8057-82feb8b67a66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 01 03:30:25.883910 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:25.883883    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-cchc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 01 03:30:25.884093 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:25.884070    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7" podUID="4322b03e-6c6d-435d-bf3b-9e60757a5845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 01 03:30:26.538584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:26.537849    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:30:26.882583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:26.871957    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-swk55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 01 03:30:26.882583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:26.872021    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55" podUID="ab72eb99-1ab5-4e1d-8057-82feb8b67a66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 01 03:30:26.882583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:26.881909    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-cchc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 01 03:30:26.882583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:26.881981    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7" podUID="4322b03e-6c6d-435d-bf3b-9e60757a5845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 01 03:30:27.866420 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:27.866380    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-swk55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 01 03:30:27.866778 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:27.866462    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55" podUID="ab72eb99-1ab5-4e1d-8057-82feb8b67a66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 01 03:30:27.882200 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:27.882164    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-cchc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 01 03:30:27.882381 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:27.882228    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7" podUID="4322b03e-6c6d-435d-bf3b-9e60757a5845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 01 03:30:28.866435 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:28.866397    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-swk55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 01 03:30:28.866955 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:28.866475    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55" podUID="ab72eb99-1ab5-4e1d-8057-82feb8b67a66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 01 03:30:28.882099 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:28.882068    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-cchc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body=
Nov 01 03:30:28.882252 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:28.882131    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7" podUID="4322b03e-6c6d-435d-bf3b-9e60757a5845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused"
Nov 01 03:30:30.640577 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:30.637874    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55"
Nov 01 03:30:30.640577 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:30.639953    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55"
Nov 01 03:30:30.652912 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:30.652886    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55"
Nov 01 03:30:30.896641 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:30.885506    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-cchc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 03:30:30.896641 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:30.885662    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7" podUID="4322b03e-6c6d-435d-bf3b-9e60757a5845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:30:30.960227 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:30.960102    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7"
Nov 01 03:30:30.961886 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:30.961856    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7"
Nov 01 03:30:30.963101 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:30.963077    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7"
Nov 01 03:30:31.441394 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:31.441355    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:30:32.052082 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:32.052046    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18318018-p94kg"]
Nov 01 03:30:32.057699 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:32.057671    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18318018-p94kg"]
Nov 01 03:30:33.623428 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:33.623386    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14655c03-e09f-4917-b51a-fd2a5b01e119" path="/var/lib/kubelet/pods/14655c03-e09f-4917-b51a-fd2a5b01e119/volumes"
Nov 01 03:30:34.290687 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:34.290637    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:30:34.561319 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:34.561233    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5c42938-fphr7"]
Nov 01 03:30:34.561965 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:34.561946    4930 cpu_manager.go:414] "containerMap: removing container" podUID="4a21bb3b-ad09-4622-b497-b385d320bf6a" containerName="cleanup"
Nov 01 03:30:34.562019 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:34.561966    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a21bb3b-ad09-4622-b497-b385d320bf6a" containerName="cleanup"
Nov 01 03:30:34.562019 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:34.561987    4930 cpu_manager.go:414] "containerMap: removing container" podUID="4a21bb3b-ad09-4622-b497-b385d320bf6a" containerName="cleanup"
Nov 01 03:30:34.562019 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:34.561993    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a21bb3b-ad09-4622-b497-b385d320bf6a" containerName="cleanup"
Nov 01 03:30:34.562019 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:34.562012    4930 cpu_manager.go:414] "containerMap: removing container" podUID="4a21bb3b-ad09-4622-b497-b385d320bf6a" containerName="cleanup"
Nov 01 03:30:34.562019 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:34.562017    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a21bb3b-ad09-4622-b497-b385d320bf6a" containerName="cleanup"
Nov 01 03:30:34.562177 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:34.562125    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a21bb3b-ad09-4622-b497-b385d320bf6a" containerName="cleanup"
Nov 01 03:30:34.562177 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:34.562137    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a21bb3b-ad09-4622-b497-b385d320bf6a" containerName="cleanup"
Nov 01 03:30:34.567307 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:34.567287    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5c42938-fphr7"
Nov 01 03:30:34.625131 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:34.625098    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5c42938-fphr7"]
Nov 01 03:30:34.649117 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:34.649079    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52261ab5-ad7f-4dd4-82ad-e82e82abf9fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d5c42938-fphr7\" (UID: \"52261ab5-ad7f-4dd4-82ad-e82e82abf9fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5c42938-fphr7"
Nov 01 03:30:34.649286 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:34.649252    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52261ab5-ad7f-4dd4-82ad-e82e82abf9fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d5c42938-fphr7\" (UID: \"52261ab5-ad7f-4dd4-82ad-e82e82abf9fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5c42938-fphr7"
Nov 01 03:30:34.649358 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:34.649310    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9266\" (UniqueName: \"kubernetes.io/projected/52261ab5-ad7f-4dd4-82ad-e82e82abf9fc-kube-api-access-q9266\") pod \"maas-api-revoke-keys-e2e-ait-d5c42938-fphr7\" (UID: \"52261ab5-ad7f-4dd4-82ad-e82e82abf9fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5c42938-fphr7"
Nov 01 03:30:34.666440 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:34.666410    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:30:34.666881 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:34.666860    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:30:34.666995 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:34.666938    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:30:34.670116 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:34.670094    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:30:34.751099 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:34.751064    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9266\" (UniqueName: \"kubernetes.io/projected/52261ab5-ad7f-4dd4-82ad-e82e82abf9fc-kube-api-access-q9266\") pod \"maas-api-revoke-keys-e2e-ait-d5c42938-fphr7\" (UID: \"52261ab5-ad7f-4dd4-82ad-e82e82abf9fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5c42938-fphr7"
Nov 01 03:30:34.751327 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:34.751310    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52261ab5-ad7f-4dd4-82ad-e82e82abf9fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d5c42938-fphr7\" (UID: \"52261ab5-ad7f-4dd4-82ad-e82e82abf9fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5c42938-fphr7"
Nov 01 03:30:34.751432 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:34.751419    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52261ab5-ad7f-4dd4-82ad-e82e82abf9fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d5c42938-fphr7\" (UID: \"52261ab5-ad7f-4dd4-82ad-e82e82abf9fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5c42938-fphr7"
Nov 01 03:30:34.752050 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:34.752020    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52261ab5-ad7f-4dd4-82ad-e82e82abf9fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d5c42938-fphr7\" (UID: \"52261ab5-ad7f-4dd4-82ad-e82e82abf9fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5c42938-fphr7"
Nov 01 03:30:34.753831 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:34.753812    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52261ab5-ad7f-4dd4-82ad-e82e82abf9fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d5c42938-fphr7\" (UID: \"52261ab5-ad7f-4dd4-82ad-e82e82abf9fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5c42938-fphr7"
Nov 01 03:30:34.812143 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:34.812066    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9266\" (UniqueName: \"kubernetes.io/projected/52261ab5-ad7f-4dd4-82ad-e82e82abf9fc-kube-api-access-q9266\") pod \"maas-api-revoke-keys-e2e-ait-d5c42938-fphr7\" (UID: \"52261ab5-ad7f-4dd4-82ad-e82e82abf9fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5c42938-fphr7"
Nov 01 03:30:34.884066 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:34.884037    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5c42938-fphr7"
Nov 01 03:30:35.270028 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:35.269995    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5c42938-fphr7"]
Nov 01 03:30:35.682943 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:35.682914    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb"]
Nov 01 03:30:35.684113 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:35.684092    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a21bb3b-ad09-4622-b497-b385d320bf6a" containerName="cleanup"
Nov 01 03:30:35.689764 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:35.689716    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb"
Nov 01 03:30:35.781941 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:35.777304    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96a82d86-3895-4362-88fa-2834dac3af47-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zlfnb\" (UID: \"96a82d86-3895-4362-88fa-2834dac3af47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb"
Nov 01 03:30:35.781941 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:35.777438    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96a82d86-3895-4362-88fa-2834dac3af47-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zlfnb\" (UID: \"96a82d86-3895-4362-88fa-2834dac3af47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb"
Nov 01 03:30:35.781941 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:35.777538    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9mbv\" (UniqueName: \"kubernetes.io/projected/96a82d86-3895-4362-88fa-2834dac3af47-kube-api-access-q9mbv\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zlfnb\" (UID: \"96a82d86-3895-4362-88fa-2834dac3af47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb"
Nov 01 03:30:35.781941 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:35.777584    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a82d86-3895-4362-88fa-2834dac3af47-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zlfnb\" (UID: \"96a82d86-3895-4362-88fa-2834dac3af47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb"
Nov 01 03:30:35.781941 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:35.777697    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96a82d86-3895-4362-88fa-2834dac3af47-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zlfnb\" (UID: \"96a82d86-3895-4362-88fa-2834dac3af47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb"
Nov 01 03:30:35.781941 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:35.777803    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a82d86-3895-4362-88fa-2834dac3af47-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zlfnb\" (UID: \"96a82d86-3895-4362-88fa-2834dac3af47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb"
Nov 01 03:30:35.781941 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:35.777827    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96a82d86-3895-4362-88fa-2834dac3af47-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zlfnb\" (UID: \"96a82d86-3895-4362-88fa-2834dac3af47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb"
Nov 01 03:30:35.781941 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:35.777879    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96a82d86-3895-4362-88fa-2834dac3af47-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zlfnb\" (UID: \"96a82d86-3895-4362-88fa-2834dac3af47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb"
Nov 01 03:30:35.781941 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:35.777917    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96a82d86-3895-4362-88fa-2834dac3af47-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zlfnb\" (UID: \"96a82d86-3895-4362-88fa-2834dac3af47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb"
Nov 01 03:30:35.952690 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:35.880677    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9mbv\" (UniqueName: \"kubernetes.io/projected/96a82d86-3895-4362-88fa-2834dac3af47-kube-api-access-q9mbv\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zlfnb\" (UID: \"96a82d86-3895-4362-88fa-2834dac3af47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb"
Nov 01 03:30:36.028894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:36.011708    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a82d86-3895-4362-88fa-2834dac3af47-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zlfnb\" (UID: \"96a82d86-3895-4362-88fa-2834dac3af47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb"
Nov 01 03:30:36.028894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:36.011873    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96a82d86-3895-4362-88fa-2834dac3af47-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zlfnb\" (UID: \"96a82d86-3895-4362-88fa-2834dac3af47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb"
Nov 01 03:30:36.028894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:36.011982    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a82d86-3895-4362-88fa-2834dac3af47-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zlfnb\" (UID: \"96a82d86-3895-4362-88fa-2834dac3af47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb"
Nov 01 03:30:36.028894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:36.012008    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96a82d86-3895-4362-88fa-2834dac3af47-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zlfnb\" (UID: \"96a82d86-3895-4362-88fa-2834dac3af47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb"
Nov 01 03:30:36.028894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:36.012067    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96a82d86-3895-4362-88fa-2834dac3af47-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zlfnb\" (UID: \"96a82d86-3895-4362-88fa-2834dac3af47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb"
Nov 01 03:30:36.028894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:36.012107    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96a82d86-3895-4362-88fa-2834dac3af47-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zlfnb\" (UID: \"96a82d86-3895-4362-88fa-2834dac3af47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb"
Nov 01 03:30:36.028894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:36.012256    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96a82d86-3895-4362-88fa-2834dac3af47-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zlfnb\" (UID: \"96a82d86-3895-4362-88fa-2834dac3af47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb"
Nov 01 03:30:36.028894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:36.012384    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96a82d86-3895-4362-88fa-2834dac3af47-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zlfnb\" (UID: \"96a82d86-3895-4362-88fa-2834dac3af47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb"
Nov 01 03:30:36.028894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:36.013277    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96a82d86-3895-4362-88fa-2834dac3af47-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zlfnb\" (UID: \"96a82d86-3895-4362-88fa-2834dac3af47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb"
Nov 01 03:30:36.028894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:36.013598    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a82d86-3895-4362-88fa-2834dac3af47-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zlfnb\" (UID: \"96a82d86-3895-4362-88fa-2834dac3af47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb"
Nov 01 03:30:36.028894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:36.016991    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96a82d86-3895-4362-88fa-2834dac3af47-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zlfnb\" (UID: \"96a82d86-3895-4362-88fa-2834dac3af47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb"
Nov 01 03:30:36.028894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:36.017238    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a82d86-3895-4362-88fa-2834dac3af47-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zlfnb\" (UID: \"96a82d86-3895-4362-88fa-2834dac3af47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb"
Nov 01 03:30:36.028894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:36.017471    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96a82d86-3895-4362-88fa-2834dac3af47-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zlfnb\" (UID: \"96a82d86-3895-4362-88fa-2834dac3af47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb"
Nov 01 03:30:36.028894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:36.027778    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb"]
Nov 01 03:30:36.052496 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:36.051648    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5c42938-fphr7" event={"ID":"52261ab5-ad7f-4dd4-82ad-e82e82abf9fc","Type":"ContainerStarted","Data":"9dbe4d408cdc02558d9479299f6e8017cebdd65517530ea27830368a083b40fc"}
Nov 01 03:30:36.053980 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:36.053880    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96a82d86-3895-4362-88fa-2834dac3af47-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zlfnb\" (UID: \"96a82d86-3895-4362-88fa-2834dac3af47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb"
Nov 01 03:30:36.059495 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:36.059471    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9mbv\" (UniqueName: \"kubernetes.io/projected/96a82d86-3895-4362-88fa-2834dac3af47-kube-api-access-q9mbv\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zlfnb\" (UID: \"96a82d86-3895-4362-88fa-2834dac3af47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb"
Nov 01 03:30:36.063286 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:36.063170    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96a82d86-3895-4362-88fa-2834dac3af47-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zlfnb\" (UID: \"96a82d86-3895-4362-88fa-2834dac3af47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb"
Nov 01 03:30:36.080425 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:36.080394    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96a82d86-3895-4362-88fa-2834dac3af47-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-zlfnb\" (UID: \"96a82d86-3895-4362-88fa-2834dac3af47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb"
Nov 01 03:30:36.307094 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:36.307012    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb"
Nov 01 03:30:36.748256 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:36.748221    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:30:36.759453 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:36.759422    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb"]
Nov 01 03:30:36.792203 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:30:36.792127    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96a82d86_3895_4362_88fa_2834dac3af47.slice/crio-c3c62671bc212dfe22465dc901780e673256b2c5b84294d4af2ee1a4265f28a3 WatchSource:0}: Error finding container c3c62671bc212dfe22465dc901780e673256b2c5b84294d4af2ee1a4265f28a3: Status 404 returned error can't find the container with id c3c62671bc212dfe22465dc901780e673256b2c5b84294d4af2ee1a4265f28a3
Nov 01 03:30:36.795304 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:36.795269    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:30:36.795421 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:36.795351    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:30:36.795421 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:36.795383    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:30:37.086581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:37.083244    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb" event={"ID":"96a82d86-3895-4362-88fa-2834dac3af47","Type":"ContainerStarted","Data":"47d28e890544e3839de55e9c64b4e9e823739ffdcf565f0c4f8600060265064a"}
Nov 01 03:30:37.086581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:37.083318    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb" event={"ID":"96a82d86-3895-4362-88fa-2834dac3af47","Type":"ContainerStarted","Data":"c3c62671bc212dfe22465dc901780e673256b2c5b84294d4af2ee1a4265f28a3"}
Nov 01 03:30:37.086868 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:37.086799    4930 generic.go:356] "Generic (PLEG): container finished" podID="52261ab5-ad7f-4dd4-82ad-e82e82abf9fc" containerID="62cea5625a1a4358640038d0340fa72508f1d86edf0982aac8e821fc35c8b9f2" exitCode=0
Nov 01 03:30:37.086868 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:37.086840    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5c42938-fphr7" event={"ID":"52261ab5-ad7f-4dd4-82ad-e82e82abf9fc","Type":"ContainerDied","Data":"62cea5625a1a4358640038d0340fa72508f1d86edf0982aac8e821fc35c8b9f2"}
Nov 01 03:30:37.257636 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:37.256174    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb" podStartSLOduration=2.255957582 podStartE2EDuration="2.255957582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:30:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:30:37.227761583 +0000 UTC m=+1441.750768175" watchObservedRunningTime="2026-11-01 03:30:37.255957582 +0000 UTC m=+1441.778964172"
Nov 01 03:30:37.307414 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:37.307377    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb"
Nov 01 03:30:37.309232 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:37.309206    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 01 03:30:37.309380 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:37.309260    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb" podUID="96a82d86-3895-4362-88fa-2834dac3af47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 01 03:30:38.308895 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:38.307848    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 01 03:30:38.308895 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:38.307906    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb" podUID="96a82d86-3895-4362-88fa-2834dac3af47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 01 03:30:38.426241 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:38.426208    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5c42938-fphr7"
Nov 01 03:30:38.505042 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:38.504996    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52261ab5-ad7f-4dd4-82ad-e82e82abf9fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52261ab5-ad7f-4dd4-82ad-e82e82abf9fc-maas-api-xks-ca\") pod \"52261ab5-ad7f-4dd4-82ad-e82e82abf9fc\" (UID: \"52261ab5-ad7f-4dd4-82ad-e82e82abf9fc\") "
Nov 01 03:30:38.505253 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:38.505080    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52261ab5-ad7f-4dd4-82ad-e82e82abf9fc-kube-api-access-q9266\" (UniqueName: \"kubernetes.io/projected/52261ab5-ad7f-4dd4-82ad-e82e82abf9fc-kube-api-access-q9266\") pod \"52261ab5-ad7f-4dd4-82ad-e82e82abf9fc\" (UID: \"52261ab5-ad7f-4dd4-82ad-e82e82abf9fc\") "
Nov 01 03:30:38.505253 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:38.505105    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52261ab5-ad7f-4dd4-82ad-e82e82abf9fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52261ab5-ad7f-4dd4-82ad-e82e82abf9fc-maas-api-openshift-ca\") pod \"52261ab5-ad7f-4dd4-82ad-e82e82abf9fc\" (UID: \"52261ab5-ad7f-4dd4-82ad-e82e82abf9fc\") "
Nov 01 03:30:38.505783 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:38.505756    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52261ab5-ad7f-4dd4-82ad-e82e82abf9fc-maas-api-openshift-ca" pod "52261ab5-ad7f-4dd4-82ad-e82e82abf9fc" (UID: "52261ab5-ad7f-4dd4-82ad-e82e82abf9fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:30:38.507275 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:38.507252    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52261ab5-ad7f-4dd4-82ad-e82e82abf9fc-maas-api-xks-ca" pod "52261ab5-ad7f-4dd4-82ad-e82e82abf9fc" (UID: "52261ab5-ad7f-4dd4-82ad-e82e82abf9fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:30:38.507847 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:38.507827    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52261ab5-ad7f-4dd4-82ad-e82e82abf9fc-kube-api-access-q9266" pod "52261ab5-ad7f-4dd4-82ad-e82e82abf9fc" (UID: "52261ab5-ad7f-4dd4-82ad-e82e82abf9fc"). InnerVolumeSpecName "kube-api-access-q9266". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:30:38.607108 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:38.607073    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52261ab5-ad7f-4dd4-82ad-e82e82abf9fc-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:30:38.607108 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:38.607108    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9266\" (UniqueName: \"kubernetes.io/projected/52261ab5-ad7f-4dd4-82ad-e82e82abf9fc-kube-api-access-q9266\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:30:38.607311 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:38.607125    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52261ab5-ad7f-4dd4-82ad-e82e82abf9fc-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:30:39.212702 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:39.206381    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5c42938-fphr7"
Nov 01 03:30:39.212702 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:39.209006    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5c42938-fphr7" event={"ID":"52261ab5-ad7f-4dd4-82ad-e82e82abf9fc","Type":"ContainerDied","Data":"9dbe4d408cdc02558d9479299f6e8017cebdd65517530ea27830368a083b40fc"}
Nov 01 03:30:39.212702 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:39.209095    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9dbe4d408cdc02558d9479299f6e8017cebdd65517530ea27830368a083b40fc"
Nov 01 03:30:39.319591 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:39.318886    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 01 03:30:39.319591 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:39.318950    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb" podUID="96a82d86-3895-4362-88fa-2834dac3af47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 01 03:30:40.308014 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:40.307972    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 01 03:30:40.308207 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:40.308038    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb" podUID="96a82d86-3895-4362-88fa-2834dac3af47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 01 03:30:41.308483 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:41.308020    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 01 03:30:41.308483 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:41.308067    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb" podUID="96a82d86-3895-4362-88fa-2834dac3af47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 01 03:30:41.725364 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:41.724201    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7911e5a0-pqsfp"]
Nov 01 03:30:41.725364 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:41.725246    4930 cpu_manager.go:414] "containerMap: removing container" podUID="52261ab5-ad7f-4dd4-82ad-e82e82abf9fc" containerName="revoke-keys"
Nov 01 03:30:41.725364 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:41.725268    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52261ab5-ad7f-4dd4-82ad-e82e82abf9fc" containerName="revoke-keys"
Nov 01 03:30:41.725656 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:41.725497    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="52261ab5-ad7f-4dd4-82ad-e82e82abf9fc" containerName="revoke-keys"
Nov 01 03:30:41.730427 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:41.730405    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7911e5a0-pqsfp"
Nov 01 03:30:41.795835 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:41.795804    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7911e5a0-pqsfp"]
Nov 01 03:30:41.826985 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:41.826950    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:30:41.827169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:41.827021    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:30:41.827169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:41.827066    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:30:41.827902 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:41.827877    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnfpq\" (UniqueName: \"kubernetes.io/projected/25527c49-7998-4df3-bf54-afd67695bd61-kube-api-access-xnfpq\") pod \"maas-api-revoke-keys-e2e-ait-7911e5a0-pqsfp\" (UID: \"25527c49-7998-4df3-bf54-afd67695bd61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7911e5a0-pqsfp"
Nov 01 03:30:41.828094 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:41.828076    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25527c49-7998-4df3-bf54-afd67695bd61-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-7911e5a0-pqsfp\" (UID: \"25527c49-7998-4df3-bf54-afd67695bd61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7911e5a0-pqsfp"
Nov 01 03:30:41.828175 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:41.828142    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25527c49-7998-4df3-bf54-afd67695bd61-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-7911e5a0-pqsfp\" (UID: \"25527c49-7998-4df3-bf54-afd67695bd61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7911e5a0-pqsfp"
Nov 01 03:30:41.852720 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:41.848123    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:30:41.929313 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:41.929279    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnfpq\" (UniqueName: \"kubernetes.io/projected/25527c49-7998-4df3-bf54-afd67695bd61-kube-api-access-xnfpq\") pod \"maas-api-revoke-keys-e2e-ait-7911e5a0-pqsfp\" (UID: \"25527c49-7998-4df3-bf54-afd67695bd61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7911e5a0-pqsfp"
Nov 01 03:30:41.929582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:41.929546    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25527c49-7998-4df3-bf54-afd67695bd61-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-7911e5a0-pqsfp\" (UID: \"25527c49-7998-4df3-bf54-afd67695bd61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7911e5a0-pqsfp"
Nov 01 03:30:41.929660 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:41.929642    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25527c49-7998-4df3-bf54-afd67695bd61-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-7911e5a0-pqsfp\" (UID: \"25527c49-7998-4df3-bf54-afd67695bd61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7911e5a0-pqsfp"
Nov 01 03:30:41.930481 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:41.930462    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25527c49-7998-4df3-bf54-afd67695bd61-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-7911e5a0-pqsfp\" (UID: \"25527c49-7998-4df3-bf54-afd67695bd61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7911e5a0-pqsfp"
Nov 01 03:30:41.932606 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:41.932207    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25527c49-7998-4df3-bf54-afd67695bd61-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-7911e5a0-pqsfp\" (UID: \"25527c49-7998-4df3-bf54-afd67695bd61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7911e5a0-pqsfp"
Nov 01 03:30:41.993988 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:41.993921    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnfpq\" (UniqueName: \"kubernetes.io/projected/25527c49-7998-4df3-bf54-afd67695bd61-kube-api-access-xnfpq\") pod \"maas-api-revoke-keys-e2e-ait-7911e5a0-pqsfp\" (UID: \"25527c49-7998-4df3-bf54-afd67695bd61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7911e5a0-pqsfp"
Nov 01 03:30:42.076591 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:42.048690    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7911e5a0-pqsfp"
Nov 01 03:30:42.307745 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:42.307538    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 01 03:30:42.307745 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:42.307656    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb" podUID="96a82d86-3895-4362-88fa-2834dac3af47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 01 03:30:42.528758 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:42.528703    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7911e5a0-pqsfp"]
Nov 01 03:30:43.069274 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:43.069240    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:30:43.300389 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:43.300272    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7911e5a0-pqsfp" event={"ID":"25527c49-7998-4df3-bf54-afd67695bd61","Type":"ContainerStarted","Data":"9f53ee83f6113d88e02c7258ff41a9d3c7a0b71758de179a6774ce56d6d050cd"}
Nov 01 03:30:43.300747 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:43.300703    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7911e5a0-pqsfp" event={"ID":"25527c49-7998-4df3-bf54-afd67695bd61","Type":"ContainerStarted","Data":"d5b6e94eee3641d66b267dab33adfa01b90266e7e38e7a20f1686d2ff4daa293"}
Nov 01 03:30:43.307517 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:43.307484    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 01 03:30:43.307629 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:43.307601    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb" podUID="96a82d86-3895-4362-88fa-2834dac3af47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 01 03:30:43.426137 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:43.426065    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7911e5a0-pqsfp" podStartSLOduration=2.426046356 podStartE2EDuration="2.426046356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:30:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:30:43.41913416 +0000 UTC m=+1447.942140748" watchObservedRunningTime="2026-11-01 03:30:43.426046356 +0000 UTC m=+1447.949052947"
Nov 01 03:30:43.912041 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:43.912007    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:30:44.308350 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:44.308246    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 01 03:30:44.308533 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:44.308344    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb" podUID="96a82d86-3895-4362-88fa-2834dac3af47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 01 03:30:44.313018 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:44.312987    4930 generic.go:356] "Generic (PLEG): container finished" podID="25527c49-7998-4df3-bf54-afd67695bd61" containerID="9f53ee83f6113d88e02c7258ff41a9d3c7a0b71758de179a6774ce56d6d050cd" exitCode=0
Nov 01 03:30:44.313138 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:44.313081    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7911e5a0-pqsfp" event={"ID":"25527c49-7998-4df3-bf54-afd67695bd61","Type":"ContainerDied","Data":"9f53ee83f6113d88e02c7258ff41a9d3c7a0b71758de179a6774ce56d6d050cd"}
Nov 01 03:30:45.621761 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:45.621728    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7911e5a0-pqsfp"
Nov 01 03:30:45.751995 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:45.751957    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25527c49-7998-4df3-bf54-afd67695bd61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25527c49-7998-4df3-bf54-afd67695bd61-maas-api-xks-ca\") pod \"25527c49-7998-4df3-bf54-afd67695bd61\" (UID: \"25527c49-7998-4df3-bf54-afd67695bd61\") "
Nov 01 03:30:45.752193 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:45.752042    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25527c49-7998-4df3-bf54-afd67695bd61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25527c49-7998-4df3-bf54-afd67695bd61-maas-api-openshift-ca\") pod \"25527c49-7998-4df3-bf54-afd67695bd61\" (UID: \"25527c49-7998-4df3-bf54-afd67695bd61\") "
Nov 01 03:30:45.752275 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:45.752209    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25527c49-7998-4df3-bf54-afd67695bd61-kube-api-access-xnfpq\" (UniqueName: \"kubernetes.io/projected/25527c49-7998-4df3-bf54-afd67695bd61-kube-api-access-xnfpq\") pod \"25527c49-7998-4df3-bf54-afd67695bd61\" (UID: \"25527c49-7998-4df3-bf54-afd67695bd61\") "
Nov 01 03:30:45.752785 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:45.752756    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25527c49-7998-4df3-bf54-afd67695bd61-maas-api-openshift-ca" pod "25527c49-7998-4df3-bf54-afd67695bd61" (UID: "25527c49-7998-4df3-bf54-afd67695bd61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:30:45.753142 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:45.753114    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25527c49-7998-4df3-bf54-afd67695bd61-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:30:45.783131 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:45.783094    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25527c49-7998-4df3-bf54-afd67695bd61-maas-api-xks-ca" pod "25527c49-7998-4df3-bf54-afd67695bd61" (UID: "25527c49-7998-4df3-bf54-afd67695bd61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:30:45.786113 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:45.786082    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25527c49-7998-4df3-bf54-afd67695bd61-kube-api-access-xnfpq" pod "25527c49-7998-4df3-bf54-afd67695bd61" (UID: "25527c49-7998-4df3-bf54-afd67695bd61"). InnerVolumeSpecName "kube-api-access-xnfpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:30:45.855038 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:45.855001    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25527c49-7998-4df3-bf54-afd67695bd61-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:30:45.855038 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:45.855041    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnfpq\" (UniqueName: \"kubernetes.io/projected/25527c49-7998-4df3-bf54-afd67695bd61-kube-api-access-xnfpq\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:30:46.308579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:46.307925    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 03:30:46.308579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:46.307995    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb" podUID="96a82d86-3895-4362-88fa-2834dac3af47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:30:46.333929 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:46.333898    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb"
Nov 01 03:30:46.334529 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:46.334499    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb"
Nov 01 03:30:46.336993 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:46.336965    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb"
Nov 01 03:30:46.345697 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:46.345666    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7911e5a0-pqsfp" event={"ID":"25527c49-7998-4df3-bf54-afd67695bd61","Type":"ContainerDied","Data":"d5b6e94eee3641d66b267dab33adfa01b90266e7e38e7a20f1686d2ff4daa293"}
Nov 01 03:30:46.345839 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:46.345720    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5b6e94eee3641d66b267dab33adfa01b90266e7e38e7a20f1686d2ff4daa293"
Nov 01 03:30:46.345839 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:46.345788    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7911e5a0-pqsfp"
Nov 01 03:30:46.403415 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:46.403375    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:30:46.559401 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:46.558037    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:30:47.057846 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:47.057801    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99b75fdf-68cgr"]
Nov 01 03:30:47.064132 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:47.064104    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99b75fdf-68cgr"]
Nov 01 03:30:47.691585 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:47.675434    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d48777d1-2695-4350-9a34-07d3319086c3" path="/var/lib/kubelet/pods/d48777d1-2695-4350-9a34-07d3319086c3/volumes"
Nov 01 03:30:49.107433 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:49.107398    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:30:49.129991 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:30:49.129956    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:30:50.188819 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:50.188752    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2b25020-m2ql9"]
Nov 01 03:30:50.189534 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:50.189516    4930 cpu_manager.go:414] "containerMap: removing container" podUID="25527c49-7998-4df3-bf54-afd67695bd61" containerName="revoke-keys"
Nov 01 03:30:50.189592 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:50.189536    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25527c49-7998-4df3-bf54-afd67695bd61" containerName="revoke-keys"
Nov 01 03:30:50.189707 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:50.189696    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="25527c49-7998-4df3-bf54-afd67695bd61" containerName="revoke-keys"
Nov 01 03:30:50.193776 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:50.193756    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b25020-m2ql9"
Nov 01 03:30:50.253738 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:50.253703    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2b25020-m2ql9"]
Nov 01 03:30:50.293922 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:50.293885    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:30:50.293922 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:50.293898    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:30:50.293922 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:50.293921    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:30:50.295116 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:50.295095    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:30:50.329305 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:50.329261    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebdbcdf4-a27e-4735-8d69-39d0d0df4b29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2b25020-m2ql9\" (UID: \"ebdbcdf4-a27e-4735-8d69-39d0d0df4b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b25020-m2ql9"
Nov 01 03:30:50.329539 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:50.329515    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk4sl\" (UniqueName: \"kubernetes.io/projected/ebdbcdf4-a27e-4735-8d69-39d0d0df4b29-kube-api-access-hk4sl\") pod \"maas-api-revoke-sub-f2b25020-m2ql9\" (UID: \"ebdbcdf4-a27e-4735-8d69-39d0d0df4b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b25020-m2ql9"
Nov 01 03:30:50.329649 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:50.329623    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebdbcdf4-a27e-4735-8d69-39d0d0df4b29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2b25020-m2ql9\" (UID: \"ebdbcdf4-a27e-4735-8d69-39d0d0df4b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b25020-m2ql9"
Nov 01 03:30:50.431124 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:50.431091    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebdbcdf4-a27e-4735-8d69-39d0d0df4b29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2b25020-m2ql9\" (UID: \"ebdbcdf4-a27e-4735-8d69-39d0d0df4b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b25020-m2ql9"
Nov 01 03:30:50.431286 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:50.431258    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebdbcdf4-a27e-4735-8d69-39d0d0df4b29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2b25020-m2ql9\" (UID: \"ebdbcdf4-a27e-4735-8d69-39d0d0df4b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b25020-m2ql9"
Nov 01 03:30:50.431477 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:50.431457    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk4sl\" (UniqueName: \"kubernetes.io/projected/ebdbcdf4-a27e-4735-8d69-39d0d0df4b29-kube-api-access-hk4sl\") pod \"maas-api-revoke-sub-f2b25020-m2ql9\" (UID: \"ebdbcdf4-a27e-4735-8d69-39d0d0df4b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b25020-m2ql9"
Nov 01 03:30:50.431836 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:50.431814    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebdbcdf4-a27e-4735-8d69-39d0d0df4b29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2b25020-m2ql9\" (UID: \"ebdbcdf4-a27e-4735-8d69-39d0d0df4b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b25020-m2ql9"
Nov 01 03:30:50.433851 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:50.433828    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebdbcdf4-a27e-4735-8d69-39d0d0df4b29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2b25020-m2ql9\" (UID: \"ebdbcdf4-a27e-4735-8d69-39d0d0df4b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b25020-m2ql9"
Nov 01 03:30:50.492767 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:50.492688    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk4sl\" (UniqueName: \"kubernetes.io/projected/ebdbcdf4-a27e-4735-8d69-39d0d0df4b29-kube-api-access-hk4sl\") pod \"maas-api-revoke-sub-f2b25020-m2ql9\" (UID: \"ebdbcdf4-a27e-4735-8d69-39d0d0df4b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b25020-m2ql9"
Nov 01 03:30:50.509734 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:50.509706    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b25020-m2ql9"
Nov 01 03:30:50.898847 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:30:50.898819    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebdbcdf4_a27e_4735_8d69_39d0d0df4b29.slice/crio-743175c16c70f91aa0452869b439b5aabdb184b58c2f2d9d9bcc3c2710611f9b WatchSource:0}: Error finding container 743175c16c70f91aa0452869b439b5aabdb184b58c2f2d9d9bcc3c2710611f9b: Status 404 returned error can't find the container with id 743175c16c70f91aa0452869b439b5aabdb184b58c2f2d9d9bcc3c2710611f9b
Nov 01 03:30:50.901784 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:50.901755    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2b25020-m2ql9"]
Nov 01 03:30:51.395107 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:51.395067    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b25020-m2ql9" event={"ID":"ebdbcdf4-a27e-4735-8d69-39d0d0df4b29","Type":"ContainerStarted","Data":"b8a5a47d04cdc578425573fe9edeeee464ce95526463821910387637d62c24bb"}
Nov 01 03:30:51.395483 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:51.395133    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b25020-m2ql9" event={"ID":"ebdbcdf4-a27e-4735-8d69-39d0d0df4b29","Type":"ContainerStarted","Data":"743175c16c70f91aa0452869b439b5aabdb184b58c2f2d9d9bcc3c2710611f9b"}
Nov 01 03:30:51.517246 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:51.517149    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b25020-m2ql9" podStartSLOduration=1.517136758 podStartE2EDuration="1.517136758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:30:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:30:51.511322763 +0000 UTC m=+1456.034329354" watchObservedRunningTime="2026-11-01 03:30:51.517136758 +0000 UTC m=+1456.040143384"
Nov 01 03:30:52.408529 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:52.408491    4930 generic.go:356] "Generic (PLEG): container finished" podID="ebdbcdf4-a27e-4735-8d69-39d0d0df4b29" containerID="b8a5a47d04cdc578425573fe9edeeee464ce95526463821910387637d62c24bb" exitCode=0
Nov 01 03:30:52.409021 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:52.408590    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b25020-m2ql9" event={"ID":"ebdbcdf4-a27e-4735-8d69-39d0d0df4b29","Type":"ContainerDied","Data":"b8a5a47d04cdc578425573fe9edeeee464ce95526463821910387637d62c24bb"}
Nov 01 03:30:53.332334 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:53.332297    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7911e5a0-8699c789f-dspb9"]
Nov 01 03:30:53.576800 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:53.576644    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:30:53.807651 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:53.806179    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:30:53.977816 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:53.977789    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b25020-m2ql9"
Nov 01 03:30:54.127050 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:54.127018    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebdbcdf4-a27e-4735-8d69-39d0d0df4b29-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebdbcdf4-a27e-4735-8d69-39d0d0df4b29-maas-api-xks-ca\") pod \"ebdbcdf4-a27e-4735-8d69-39d0d0df4b29\" (UID: \"ebdbcdf4-a27e-4735-8d69-39d0d0df4b29\") "
Nov 01 03:30:54.127234 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:54.127124    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebdbcdf4-a27e-4735-8d69-39d0d0df4b29-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebdbcdf4-a27e-4735-8d69-39d0d0df4b29-maas-api-openshift-ca\") pod \"ebdbcdf4-a27e-4735-8d69-39d0d0df4b29\" (UID: \"ebdbcdf4-a27e-4735-8d69-39d0d0df4b29\") "
Nov 01 03:30:54.127387 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:54.127356    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebdbcdf4-a27e-4735-8d69-39d0d0df4b29-kube-api-access-hk4sl\" (UniqueName: \"kubernetes.io/projected/ebdbcdf4-a27e-4735-8d69-39d0d0df4b29-kube-api-access-hk4sl\") pod \"ebdbcdf4-a27e-4735-8d69-39d0d0df4b29\" (UID: \"ebdbcdf4-a27e-4735-8d69-39d0d0df4b29\") "
Nov 01 03:30:54.127990 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:54.127959    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebdbcdf4-a27e-4735-8d69-39d0d0df4b29-maas-api-openshift-ca" pod "ebdbcdf4-a27e-4735-8d69-39d0d0df4b29" (UID: "ebdbcdf4-a27e-4735-8d69-39d0d0df4b29"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:30:54.130712 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:54.130121    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebdbcdf4-a27e-4735-8d69-39d0d0df4b29-kube-api-access-hk4sl" pod "ebdbcdf4-a27e-4735-8d69-39d0d0df4b29" (UID: "ebdbcdf4-a27e-4735-8d69-39d0d0df4b29"). InnerVolumeSpecName "kube-api-access-hk4sl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:30:54.131118 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:54.131097    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebdbcdf4-a27e-4735-8d69-39d0d0df4b29-maas-api-xks-ca" pod "ebdbcdf4-a27e-4735-8d69-39d0d0df4b29" (UID: "ebdbcdf4-a27e-4735-8d69-39d0d0df4b29"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:30:54.191146 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:54.191071    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:30:54.229305 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:54.229276    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk4sl\" (UniqueName: \"kubernetes.io/projected/ebdbcdf4-a27e-4735-8d69-39d0d0df4b29-kube-api-access-hk4sl\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:30:54.229305 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:54.229305    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebdbcdf4-a27e-4735-8d69-39d0d0df4b29-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:30:54.229471 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:54.229317    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebdbcdf4-a27e-4735-8d69-39d0d0df4b29-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:30:54.391036 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:54.390976    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:30:54.435840 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:54.435816    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b25020-m2ql9"
Nov 01 03:30:54.436029 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:54.435808    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2b25020-m2ql9" event={"ID":"ebdbcdf4-a27e-4735-8d69-39d0d0df4b29","Type":"ContainerDied","Data":"743175c16c70f91aa0452869b439b5aabdb184b58c2f2d9d9bcc3c2710611f9b"}
Nov 01 03:30:54.436029 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:54.435943    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="743175c16c70f91aa0452869b439b5aabdb184b58c2f2d9d9bcc3c2710611f9b"
Nov 01 03:30:54.789342 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:54.789242    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:30:57.700671 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:57.700638    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6e7ea12-n4qjk"]
Nov 01 03:30:57.701286 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:57.701272    4930 cpu_manager.go:414] "containerMap: removing container" podUID="ebdbcdf4-a27e-4735-8d69-39d0d0df4b29" containerName="revoke-keys"
Nov 01 03:30:57.701286 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:57.701285    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebdbcdf4-a27e-4735-8d69-39d0d0df4b29" containerName="revoke-keys"
Nov 01 03:30:57.701437 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:57.701430    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebdbcdf4-a27e-4735-8d69-39d0d0df4b29" containerName="revoke-keys"
Nov 01 03:30:57.705186 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:57.705166    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6e7ea12-n4qjk"
Nov 01 03:30:57.766419 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:57.766386    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6e7ea12-n4qjk"]
Nov 01 03:30:57.794669 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:57.794514    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb7fa4d3-2f67-4dea-b20a-d8ffbceec029-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e6e7ea12-n4qjk\" (UID: \"cb7fa4d3-2f67-4dea-b20a-d8ffbceec029\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6e7ea12-n4qjk"
Nov 01 03:30:57.794669 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:57.794632    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb7fa4d3-2f67-4dea-b20a-d8ffbceec029-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e6e7ea12-n4qjk\" (UID: \"cb7fa4d3-2f67-4dea-b20a-d8ffbceec029\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6e7ea12-n4qjk"
Nov 01 03:30:57.794883 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:57.794778    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv6ds\" (UniqueName: \"kubernetes.io/projected/cb7fa4d3-2f67-4dea-b20a-d8ffbceec029-kube-api-access-vv6ds\") pod \"maas-api-revoke-keys-e2e-ait-e6e7ea12-n4qjk\" (UID: \"cb7fa4d3-2f67-4dea-b20a-d8ffbceec029\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6e7ea12-n4qjk"
Nov 01 03:30:57.805899 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:57.805867    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:30:57.805899 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:57.805878    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:30:57.806050 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:57.805934    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:30:57.807106 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:57.807089    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:30:57.896356 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:57.896323    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb7fa4d3-2f67-4dea-b20a-d8ffbceec029-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e6e7ea12-n4qjk\" (UID: \"cb7fa4d3-2f67-4dea-b20a-d8ffbceec029\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6e7ea12-n4qjk"
Nov 01 03:30:57.896492 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:57.896403    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb7fa4d3-2f67-4dea-b20a-d8ffbceec029-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e6e7ea12-n4qjk\" (UID: \"cb7fa4d3-2f67-4dea-b20a-d8ffbceec029\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6e7ea12-n4qjk"
Nov 01 03:30:57.896492 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:57.896467    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv6ds\" (UniqueName: \"kubernetes.io/projected/cb7fa4d3-2f67-4dea-b20a-d8ffbceec029-kube-api-access-vv6ds\") pod \"maas-api-revoke-keys-e2e-ait-e6e7ea12-n4qjk\" (UID: \"cb7fa4d3-2f67-4dea-b20a-d8ffbceec029\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6e7ea12-n4qjk"
Nov 01 03:30:57.897128 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:57.897107    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb7fa4d3-2f67-4dea-b20a-d8ffbceec029-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e6e7ea12-n4qjk\" (UID: \"cb7fa4d3-2f67-4dea-b20a-d8ffbceec029\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6e7ea12-n4qjk"
Nov 01 03:30:57.898695 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:57.898671    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb7fa4d3-2f67-4dea-b20a-d8ffbceec029-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e6e7ea12-n4qjk\" (UID: \"cb7fa4d3-2f67-4dea-b20a-d8ffbceec029\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6e7ea12-n4qjk"
Nov 01 03:30:57.958575 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:57.958501    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv6ds\" (UniqueName: \"kubernetes.io/projected/cb7fa4d3-2f67-4dea-b20a-d8ffbceec029-kube-api-access-vv6ds\") pod \"maas-api-revoke-keys-e2e-ait-e6e7ea12-n4qjk\" (UID: \"cb7fa4d3-2f67-4dea-b20a-d8ffbceec029\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6e7ea12-n4qjk"
Nov 01 03:30:58.022982 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:58.022956    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6e7ea12-n4qjk"
Nov 01 03:30:58.333413 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:58.333333    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-7911e5a0-8699c789f-dspb9" podUID="0a2203d6-efb2-416c-95b3-92638b3ef827" containerName="payload-pre-processing" containerID="cri-o://97e7690ec0f71228a53bc82f84b9b8bb291d7af5fadd051651f92972f4c65667" gracePeriod=25
Nov 01 03:30:58.405781 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:58.405753    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6e7ea12-n4qjk"]
Nov 01 03:30:58.433756 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:30:58.433731    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb7fa4d3_2f67_4dea_b20a_d8ffbceec029.slice/crio-383a4b8c767fcda6f43d83a651c00668f0f80bc3a46d465d641fd22696ed2eb1 WatchSource:0}: Error finding container 383a4b8c767fcda6f43d83a651c00668f0f80bc3a46d465d641fd22696ed2eb1: Status 404 returned error can't find the container with id 383a4b8c767fcda6f43d83a651c00668f0f80bc3a46d465d641fd22696ed2eb1
Nov 01 03:30:58.480269 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:58.480237    4930 generic.go:356] "Generic (PLEG): container finished" podID="0a2203d6-efb2-416c-95b3-92638b3ef827" containerID="97e7690ec0f71228a53bc82f84b9b8bb291d7af5fadd051651f92972f4c65667" exitCode=0
Nov 01 03:30:58.480401 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:58.480323    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7911e5a0-8699c789f-dspb9" event={"ID":"0a2203d6-efb2-416c-95b3-92638b3ef827","Type":"ContainerDied","Data":"97e7690ec0f71228a53bc82f84b9b8bb291d7af5fadd051651f92972f4c65667"}
Nov 01 03:30:58.481758 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:58.481734    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6e7ea12-n4qjk" event={"ID":"cb7fa4d3-2f67-4dea-b20a-d8ffbceec029","Type":"ContainerStarted","Data":"383a4b8c767fcda6f43d83a651c00668f0f80bc3a46d465d641fd22696ed2eb1"}
Nov 01 03:30:58.681932 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:58.681909    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7911e5a0-8699c789f-dspb9"
Nov 01 03:30:58.714343 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:58.714315    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a2203d6-efb2-416c-95b3-92638b3ef827-kube-api-access-d888m\" (UniqueName: \"kubernetes.io/projected/0a2203d6-efb2-416c-95b3-92638b3ef827-kube-api-access-d888m\") pod \"0a2203d6-efb2-416c-95b3-92638b3ef827\" (UID: \"0a2203d6-efb2-416c-95b3-92638b3ef827\") "
Nov 01 03:30:58.714778 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:58.714373    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a2203d6-efb2-416c-95b3-92638b3ef827-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a2203d6-efb2-416c-95b3-92638b3ef827-plugins-config-volume\") pod \"0a2203d6-efb2-416c-95b3-92638b3ef827\" (UID: \"0a2203d6-efb2-416c-95b3-92638b3ef827\") "
Nov 01 03:30:58.714778 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:58.714755    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a2203d6-efb2-416c-95b3-92638b3ef827-plugins-config-volume" pod "0a2203d6-efb2-416c-95b3-92638b3ef827" (UID: "0a2203d6-efb2-416c-95b3-92638b3ef827"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:30:58.715067 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:58.715046    4930 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a2203d6-efb2-416c-95b3-92638b3ef827-plugins-config-volume\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:30:58.716524 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:58.716500    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a2203d6-efb2-416c-95b3-92638b3ef827-kube-api-access-d888m" pod "0a2203d6-efb2-416c-95b3-92638b3ef827" (UID: "0a2203d6-efb2-416c-95b3-92638b3ef827"). InnerVolumeSpecName "kube-api-access-d888m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:30:58.816374 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:58.816339    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d888m\" (UniqueName: \"kubernetes.io/projected/0a2203d6-efb2-416c-95b3-92638b3ef827-kube-api-access-d888m\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:30:59.491448 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:59.491414    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7911e5a0-8699c789f-dspb9"
Nov 01 03:30:59.491624 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:59.491454    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7911e5a0-8699c789f-dspb9" event={"ID":"0a2203d6-efb2-416c-95b3-92638b3ef827","Type":"ContainerDied","Data":"cfcbdc87a87d69e3abb103a3b189c26bccf513ed45f77a04d0b2c7beef48abf8"}
Nov 01 03:30:59.491624 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:59.491518    4930 scope.go:122] "RemoveContainer" containerID="97e7690ec0f71228a53bc82f84b9b8bb291d7af5fadd051651f92972f4c65667"
Nov 01 03:30:59.493276 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:59.493250    4930 generic.go:356] "Generic (PLEG): container finished" podID="cb7fa4d3-2f67-4dea-b20a-d8ffbceec029" containerID="80efdc85614993a8b5aec7e8f614132212deb4db3b3454a5366283fb630b9e45" exitCode=0
Nov 01 03:30:59.493415 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:59.493334    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6e7ea12-n4qjk" event={"ID":"cb7fa4d3-2f67-4dea-b20a-d8ffbceec029","Type":"ContainerDied","Data":"80efdc85614993a8b5aec7e8f614132212deb4db3b3454a5366283fb630b9e45"}
Nov 01 03:30:59.774633 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:59.774484    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7911e5a0-8699c789f-dspb9"]
Nov 01 03:30:59.780267 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:30:59.780237    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7911e5a0-8699c789f-dspb9"]
Nov 01 03:31:00.382168 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:00.382130    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:00.816009 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:00.815981    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6e7ea12-n4qjk"
Nov 01 03:31:00.852097 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:00.852063    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb7fa4d3-2f67-4dea-b20a-d8ffbceec029-kube-api-access-vv6ds\" (UniqueName: \"kubernetes.io/projected/cb7fa4d3-2f67-4dea-b20a-d8ffbceec029-kube-api-access-vv6ds\") pod \"cb7fa4d3-2f67-4dea-b20a-d8ffbceec029\" (UID: \"cb7fa4d3-2f67-4dea-b20a-d8ffbceec029\") "
Nov 01 03:31:00.852322 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:00.852304    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb7fa4d3-2f67-4dea-b20a-d8ffbceec029-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb7fa4d3-2f67-4dea-b20a-d8ffbceec029-maas-api-xks-ca\") pod \"cb7fa4d3-2f67-4dea-b20a-d8ffbceec029\" (UID: \"cb7fa4d3-2f67-4dea-b20a-d8ffbceec029\") "
Nov 01 03:31:00.852378 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:00.852368    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb7fa4d3-2f67-4dea-b20a-d8ffbceec029-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb7fa4d3-2f67-4dea-b20a-d8ffbceec029-maas-api-openshift-ca\") pod \"cb7fa4d3-2f67-4dea-b20a-d8ffbceec029\" (UID: \"cb7fa4d3-2f67-4dea-b20a-d8ffbceec029\") "
Nov 01 03:31:00.852792 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:00.852762    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb7fa4d3-2f67-4dea-b20a-d8ffbceec029-maas-api-openshift-ca" pod "cb7fa4d3-2f67-4dea-b20a-d8ffbceec029" (UID: "cb7fa4d3-2f67-4dea-b20a-d8ffbceec029"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:31:00.853022 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:00.853002    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb7fa4d3-2f67-4dea-b20a-d8ffbceec029-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:31:00.854412 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:00.854381    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb7fa4d3-2f67-4dea-b20a-d8ffbceec029-kube-api-access-vv6ds" pod "cb7fa4d3-2f67-4dea-b20a-d8ffbceec029" (UID: "cb7fa4d3-2f67-4dea-b20a-d8ffbceec029"). InnerVolumeSpecName "kube-api-access-vv6ds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:31:00.854550 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:00.854527    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb7fa4d3-2f67-4dea-b20a-d8ffbceec029-maas-api-xks-ca" pod "cb7fa4d3-2f67-4dea-b20a-d8ffbceec029" (UID: "cb7fa4d3-2f67-4dea-b20a-d8ffbceec029"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:31:00.955010 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:00.954932    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb7fa4d3-2f67-4dea-b20a-d8ffbceec029-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:31:00.955010 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:00.954967    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv6ds\" (UniqueName: \"kubernetes.io/projected/cb7fa4d3-2f67-4dea-b20a-d8ffbceec029-kube-api-access-vv6ds\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:31:00.974795 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:00.974759    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:01.579022 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:01.578987    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6e7ea12-n4qjk"
Nov 01 03:31:01.579022 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:01.579017    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6e7ea12-n4qjk" event={"ID":"cb7fa4d3-2f67-4dea-b20a-d8ffbceec029","Type":"ContainerDied","Data":"383a4b8c767fcda6f43d83a651c00668f0f80bc3a46d465d641fd22696ed2eb1"}
Nov 01 03:31:01.579246 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:01.579068    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="383a4b8c767fcda6f43d83a651c00668f0f80bc3a46d465d641fd22696ed2eb1"
Nov 01 03:31:01.622194 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:01.622160    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a2203d6-efb2-416c-95b3-92638b3ef827" path="/var/lib/kubelet/pods/0a2203d6-efb2-416c-95b3-92638b3ef827/volumes"
Nov 01 03:31:05.541345 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:05.541307    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:07.821169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:07.821131    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:09.314117 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:09.314064    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e6e7ea12-67857874ff-gmjm9"]
Nov 01 03:31:09.554863 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:09.554822    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:09.714283 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:09.714248    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:10.581718 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:10.581678    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:11.036310 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:11.036273    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:14.055054 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:14.055019    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e202228-gh4tp"]
Nov 01 03:31:14.111727 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:14.111687    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e202228-gh4tp"]
Nov 01 03:31:14.314924 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:14.314810    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e6e7ea12-67857874ff-gmjm9" podUID="7529a025-fe09-4ba1-9268-a6670b123646" containerName="payload-pre-processing" containerID="cri-o://7e3a91978eb0f701830b0e4d3c945cbecef7cd4e6b1bac3befc1084259b0f4cd" gracePeriod=25
Nov 01 03:31:15.154578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:15.139354    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:15.154578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:15.139962    4930 generic.go:356] "Generic (PLEG): container finished" podID="7529a025-fe09-4ba1-9268-a6670b123646" containerID="7e3a91978eb0f701830b0e4d3c945cbecef7cd4e6b1bac3befc1084259b0f4cd" exitCode=0
Nov 01 03:31:15.154578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:15.140304    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e6e7ea12-67857874ff-gmjm9" event={"ID":"7529a025-fe09-4ba1-9268-a6670b123646","Type":"ContainerDied","Data":"7e3a91978eb0f701830b0e4d3c945cbecef7cd4e6b1bac3befc1084259b0f4cd"}
Nov 01 03:31:15.364311 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:15.364281    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e6e7ea12-67857874ff-gmjm9"
Nov 01 03:31:15.430855 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:15.430755    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:15.438121 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:15.438071    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7529a025-fe09-4ba1-9268-a6670b123646-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7529a025-fe09-4ba1-9268-a6670b123646-plugins-config-volume\") pod \"7529a025-fe09-4ba1-9268-a6670b123646\" (UID: \"7529a025-fe09-4ba1-9268-a6670b123646\") "
Nov 01 03:31:15.438443 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:15.438408    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7529a025-fe09-4ba1-9268-a6670b123646-kube-api-access-plspb\" (UniqueName: \"kubernetes.io/projected/7529a025-fe09-4ba1-9268-a6670b123646-kube-api-access-plspb\") pod \"7529a025-fe09-4ba1-9268-a6670b123646\" (UID: \"7529a025-fe09-4ba1-9268-a6670b123646\") "
Nov 01 03:31:15.439046 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:15.438490    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7529a025-fe09-4ba1-9268-a6670b123646-plugins-config-volume" pod "7529a025-fe09-4ba1-9268-a6670b123646" (UID: "7529a025-fe09-4ba1-9268-a6670b123646"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:31:15.439652 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:15.439631    4930 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7529a025-fe09-4ba1-9268-a6670b123646-plugins-config-volume\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:31:15.441414 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:15.441382    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7529a025-fe09-4ba1-9268-a6670b123646-kube-api-access-plspb" pod "7529a025-fe09-4ba1-9268-a6670b123646" (UID: "7529a025-fe09-4ba1-9268-a6670b123646"). InnerVolumeSpecName "kube-api-access-plspb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:31:15.549940 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:15.543114    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plspb\" (UniqueName: \"kubernetes.io/projected/7529a025-fe09-4ba1-9268-a6670b123646-kube-api-access-plspb\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:31:15.743757 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:15.743660    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eedbb6ba-9513-45df-804b-58a710c0f294" path="/var/lib/kubelet/pods/eedbb6ba-9513-45df-804b-58a710c0f294/volumes"
Nov 01 03:31:16.168002 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:16.167098    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e6e7ea12-67857874ff-gmjm9" event={"ID":"7529a025-fe09-4ba1-9268-a6670b123646","Type":"ContainerDied","Data":"259118e0fd6daa77a14f9a553df3e6e43e5b920e36b331e9c8ae738913fa506d"}
Nov 01 03:31:16.168002 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:16.167213    4930 scope.go:122] "RemoveContainer" containerID="7e3a91978eb0f701830b0e4d3c945cbecef7cd4e6b1bac3befc1084259b0f4cd"
Nov 01 03:31:16.168002 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:16.167474    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e6e7ea12-67857874ff-gmjm9"
Nov 01 03:31:16.363583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:16.361610    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e6e7ea12-67857874ff-gmjm9"]
Nov 01 03:31:16.434777 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:16.434618    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e6e7ea12-67857874ff-gmjm9"]
Nov 01 03:31:17.545578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:17.541652    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8" podUID="88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:31:17.594333 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:17.594294    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:17.611580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:17.607972    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp" podUID="667240b0-64a7-45cd-84fe-9cd7e0d8a301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:31:17.709735 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:17.709684    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2" podUID="e2483a06-2f4b-4a1e-af4e-a63eb50d29bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:31:17.823640 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:17.820675    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t" podUID="13930c1e-2200-4412-b8d3-ae3e4126a851" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:31:17.914579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:17.912181    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7529a025-fe09-4ba1-9268-a6670b123646" path="/var/lib/kubelet/pods/7529a025-fe09-4ba1-9268-a6670b123646/volumes"
Nov 01 03:31:17.965576 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:17.965214    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:17.992593 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:17.989708    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj" podUID="b0e32cb3-ac18-4c1f-819c-f1139a88bfea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:31:18.041591 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:18.040691    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8" podUID="88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:31:18.050578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:18.049638    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm" podUID="75d142b3-7172-4b57-bc07-a2f404c4656e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:31:18.174583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:18.173698    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf" podUID="40881383-61bb-4fc8-9c75-f8ffce4834c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:31:18.212582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:18.200724    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" podUID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded"
Nov 01 03:31:18.249262 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:18.249218    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7" podUID="1ce834cc-d34d-46cc-a081-4d2cc261becb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:31:18.357531 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:18.357485    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw" podUID="b6d688b8-b911-4a6a-a842-2b0fb7a75e83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:31:18.366119 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:18.366081    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps" podUID="d658772c-b072-4513-a639-d19a94c78525" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": context deadline exceeded"
Nov 01 03:31:18.467708 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:18.467604    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:18.545576 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:18.536790    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm" podUID="b2637bc6-3828-4b75-9783-5415519403a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:31:18.552571 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:18.547110    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp" podUID="667240b0-64a7-45cd-84fe-9cd7e0d8a301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:31:18.587791 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:18.585915    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8" podUID="88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:31:18.587791 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:18.586454    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:31:18.621056 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:18.621003    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:18.657712 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:18.657678    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:31:19.083674 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:19.080277    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:19.125960 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:31:19.125929    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:31:21.049238 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:21.049203    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:23.936738 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:23.936691    4930 patch_prober.go:29] interesting pod/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 03:31:23.937105 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:23.936750    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8" podUID="9165d842-245c-4270-a5ab-8789ab0a2e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:31:25.460497 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:25.460294    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:25.945610 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:25.945575    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:26.475458 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:26.475417    4930 generic.go:356] "Generic (PLEG): container finished" podID="9165d842-245c-4270-a5ab-8789ab0a2e89" containerID="f792968e49e2f52ea64875f94f3cb60fc433bea27522ddf2d426f5032482b857" exitCode=0
Nov 01 03:31:26.475880 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:26.475490    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8" event={"ID":"9165d842-245c-4270-a5ab-8789ab0a2e89","Type":"ContainerDied","Data":"f792968e49e2f52ea64875f94f3cb60fc433bea27522ddf2d426f5032482b857"}
Nov 01 03:31:26.476083 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:26.476065    4930 scope.go:122] "RemoveContainer" containerID="f792968e49e2f52ea64875f94f3cb60fc433bea27522ddf2d426f5032482b857"
Nov 01 03:31:26.477723 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:26.477698    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:31:26.477817 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:26.477765    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:31:26.477817 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:26.477796    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:31:27.510215 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:27.510181    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8" event={"ID":"9165d842-245c-4270-a5ab-8789ab0a2e89","Type":"ContainerStarted","Data":"37ea7590884339c5454463228bc2325ec7d25f2daabc0802ce5d94f1bc4b3d4e"}
Nov 01 03:31:28.122258 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:28.122219    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"
Nov 01 03:31:28.122566 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:28.122339    4930 patch_prober.go:29] interesting pod/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 01 03:31:28.122703 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:28.122658    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8" podUID="9165d842-245c-4270-a5ab-8789ab0a2e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 01 03:31:29.050958 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:29.050920    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fffcf7a5-qjs29"]
Nov 01 03:31:29.103900 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:29.103862    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fffcf7a5-qjs29"]
Nov 01 03:31:29.117383 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:29.117356    4930 patch_prober.go:29] interesting pod/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 01 03:31:29.117506 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:29.117403    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8" podUID="9165d842-245c-4270-a5ab-8789ab0a2e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 01 03:31:29.624037 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:29.623999    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8" path="/var/lib/kubelet/pods/f14dfb49-9ee5-4d7d-a32b-c9f8b6d6b2e8/volumes"
Nov 01 03:31:30.105419 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:30.105390    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:30.117182 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:30.117152    4930 patch_prober.go:29] interesting pod/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 01 03:31:30.117336 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:30.117226    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8" podUID="9165d842-245c-4270-a5ab-8789ab0a2e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 01 03:31:31.121026 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:31.120987    4930 patch_prober.go:29] interesting pod/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 01 03:31:31.121663 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:31.121631    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8" podUID="9165d842-245c-4270-a5ab-8789ab0a2e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 01 03:31:31.288042 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:31.288006    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:31.636797 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:31.633048    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:31.905232 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:31.905130    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:32.119986 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:32.119944    4930 patch_prober.go:29] interesting pod/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 01 03:31:32.120213 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:32.120019    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8" podUID="9165d842-245c-4270-a5ab-8789ab0a2e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 01 03:31:32.274585 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:32.274465    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5c42938-fphr7"]
Nov 01 03:31:32.385730 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:32.385669    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5c42938-fphr7"]
Nov 01 03:31:32.423549 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:32.408291    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:32.731602 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:32.728748    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"]
Nov 01 03:31:32.731602 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:32.729009    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f" podUID="0fd59ab7-fd50-4ce9-a290-7f018204eb66" containerName="istio-proxy" containerID="cri-o://5430b25a0e6a45da064e414f6a4a164a847485e26ff22c074a83dac8ca4b2a76" gracePeriod=30
Nov 01 03:31:33.040195 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:33.040087    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:33.117471 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:33.117436    4930 patch_prober.go:29] interesting pod/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 01 03:31:33.117682 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:33.117495    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8" podUID="9165d842-245c-4270-a5ab-8789ab0a2e89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 01 03:31:33.269241 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:33.269202    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:33.433727 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:33.433688    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:33.634821 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:33.634782    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52261ab5-ad7f-4dd4-82ad-e82e82abf9fc" path="/var/lib/kubelet/pods/52261ab5-ad7f-4dd4-82ad-e82e82abf9fc/volumes"
Nov 01 03:31:33.931494 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:33.931455    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:34.078774 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:34.078735    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:34.244629 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:34.233802    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7911e5a0-pqsfp"]
Nov 01 03:31:34.244629 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:34.241096    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:34.283578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:34.282915    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"
Nov 01 03:31:34.292345 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:34.292285    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"
Nov 01 03:31:34.300035 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:34.299996    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"
Nov 01 03:31:34.316596 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:34.315139    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7911e5a0-pqsfp"]
Nov 01 03:31:34.539246 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:34.539155    4930 patch_prober.go:29] interesting pod/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:31:34.539638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:34.539223    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f" podUID="0fd59ab7-fd50-4ce9-a290-7f018204eb66" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:31:34.702908 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:34.702875    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:34.909578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:34.902099    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:35.289261 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:35.289176    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:35.622108 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:35.622078    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25527c49-7998-4df3-bf54-afd67695bd61" path="/var/lib/kubelet/pods/25527c49-7998-4df3-bf54-afd67695bd61/volumes"
Nov 01 03:31:35.810197 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:35.810163    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"]
Nov 01 03:31:35.811016 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:35.810991    4930 cpu_manager.go:414] "containerMap: removing container" podUID="cb7fa4d3-2f67-4dea-b20a-d8ffbceec029" containerName="revoke-keys"
Nov 01 03:31:35.811016 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:35.811015    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb7fa4d3-2f67-4dea-b20a-d8ffbceec029" containerName="revoke-keys"
Nov 01 03:31:35.811203 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:35.811043    4930 cpu_manager.go:414] "containerMap: removing container" podUID="0a2203d6-efb2-416c-95b3-92638b3ef827" containerName="payload-pre-processing"
Nov 01 03:31:35.811203 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:35.811048    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a2203d6-efb2-416c-95b3-92638b3ef827" containerName="payload-pre-processing"
Nov 01 03:31:35.811203 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:35.811069    4930 cpu_manager.go:414] "containerMap: removing container" podUID="7529a025-fe09-4ba1-9268-a6670b123646" containerName="payload-pre-processing"
Nov 01 03:31:35.811203 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:35.811075    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7529a025-fe09-4ba1-9268-a6670b123646" containerName="payload-pre-processing"
Nov 01 03:31:35.811203 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:35.811177    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb7fa4d3-2f67-4dea-b20a-d8ffbceec029" containerName="revoke-keys"
Nov 01 03:31:35.811203 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:35.811190    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a2203d6-efb2-416c-95b3-92638b3ef827" containerName="payload-pre-processing"
Nov 01 03:31:35.811203 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:35.811198    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="7529a025-fe09-4ba1-9268-a6670b123646" containerName="payload-pre-processing"
Nov 01 03:31:35.815213 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:35.815194    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"
Nov 01 03:31:35.897286 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:35.897198    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:35.902144 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:35.900964    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"]
Nov 01 03:31:35.927957 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:35.927927    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-3513c2f1-openshift-default-dockercfg-w5cpd\""
Nov 01 03:31:35.936058 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:35.936027    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jh955_f29ea61e-2a78-4c11-9566-f52497e135ff/console-operator/2.log"
Nov 01 03:31:35.940171 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:35.940148    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jh955_f29ea61e-2a78-4c11-9566-f52497e135ff/console-operator/2.log"
Nov 01 03:31:35.958493 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:35.958456    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8deab213-f1c7-4883-9a64-72789a741faf-istio-data\") pod \"e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm\" (UID: \"8deab213-f1c7-4883-9a64-72789a741faf\") " pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"
Nov 01 03:31:35.958758 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:35.958731    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm5lb\" (UniqueName: \"kubernetes.io/projected/8deab213-f1c7-4883-9a64-72789a741faf-kube-api-access-wm5lb\") pod \"e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm\" (UID: \"8deab213-f1c7-4883-9a64-72789a741faf\") " pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"
Nov 01 03:31:35.958921 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:35.958904    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8deab213-f1c7-4883-9a64-72789a741faf-credential-socket\") pod \"e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm\" (UID: \"8deab213-f1c7-4883-9a64-72789a741faf\") " pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"
Nov 01 03:31:35.959100 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:35.959083    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8deab213-f1c7-4883-9a64-72789a741faf-workload-certs\") pod \"e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm\" (UID: \"8deab213-f1c7-4883-9a64-72789a741faf\") " pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"
Nov 01 03:31:35.959252 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:35.959236    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8deab213-f1c7-4883-9a64-72789a741faf-workload-socket\") pod \"e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm\" (UID: \"8deab213-f1c7-4883-9a64-72789a741faf\") " pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"
Nov 01 03:31:35.959396 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:35.959380    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8deab213-f1c7-4883-9a64-72789a741faf-istiod-ca-cert\") pod \"e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm\" (UID: \"8deab213-f1c7-4883-9a64-72789a741faf\") " pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"
Nov 01 03:31:35.959690 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:35.959670    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8deab213-f1c7-4883-9a64-72789a741faf-istio-envoy\") pod \"e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm\" (UID: \"8deab213-f1c7-4883-9a64-72789a741faf\") " pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"
Nov 01 03:31:35.959894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:35.959833    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8deab213-f1c7-4883-9a64-72789a741faf-istio-podinfo\") pod \"e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm\" (UID: \"8deab213-f1c7-4883-9a64-72789a741faf\") " pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"
Nov 01 03:31:35.960063 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:35.959899    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8deab213-f1c7-4883-9a64-72789a741faf-istio-token\") pod \"e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm\" (UID: \"8deab213-f1c7-4883-9a64-72789a741faf\") " pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"
Nov 01 03:31:36.061617 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:36.061575    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8deab213-f1c7-4883-9a64-72789a741faf-istio-data\") pod \"e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm\" (UID: \"8deab213-f1c7-4883-9a64-72789a741faf\") " pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"
Nov 01 03:31:36.069047 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:36.069017    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm5lb\" (UniqueName: \"kubernetes.io/projected/8deab213-f1c7-4883-9a64-72789a741faf-kube-api-access-wm5lb\") pod \"e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm\" (UID: \"8deab213-f1c7-4883-9a64-72789a741faf\") " pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"
Nov 01 03:31:36.072118 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:36.072091    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8deab213-f1c7-4883-9a64-72789a741faf-credential-socket\") pod \"e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm\" (UID: \"8deab213-f1c7-4883-9a64-72789a741faf\") " pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"
Nov 01 03:31:36.072783 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:36.072756    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8deab213-f1c7-4883-9a64-72789a741faf-workload-certs\") pod \"e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm\" (UID: \"8deab213-f1c7-4883-9a64-72789a741faf\") " pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"
Nov 01 03:31:36.072930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:36.072835    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8deab213-f1c7-4883-9a64-72789a741faf-workload-socket\") pod \"e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm\" (UID: \"8deab213-f1c7-4883-9a64-72789a741faf\") " pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"
Nov 01 03:31:36.072930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:36.072891    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8deab213-f1c7-4883-9a64-72789a741faf-istiod-ca-cert\") pod \"e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm\" (UID: \"8deab213-f1c7-4883-9a64-72789a741faf\") " pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"
Nov 01 03:31:36.073099 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:36.073079    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8deab213-f1c7-4883-9a64-72789a741faf-istio-envoy\") pod \"e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm\" (UID: \"8deab213-f1c7-4883-9a64-72789a741faf\") " pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"
Nov 01 03:31:36.073328 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:36.073216    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8deab213-f1c7-4883-9a64-72789a741faf-istio-podinfo\") pod \"e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm\" (UID: \"8deab213-f1c7-4883-9a64-72789a741faf\") " pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"
Nov 01 03:31:36.073328 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:36.073302    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8deab213-f1c7-4883-9a64-72789a741faf-istio-token\") pod \"e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm\" (UID: \"8deab213-f1c7-4883-9a64-72789a741faf\") " pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"
Nov 01 03:31:36.074213 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:36.068150    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8deab213-f1c7-4883-9a64-72789a741faf-istio-data\") pod \"e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm\" (UID: \"8deab213-f1c7-4883-9a64-72789a741faf\") " pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"
Nov 01 03:31:36.074213 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:36.074065    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8deab213-f1c7-4883-9a64-72789a741faf-workload-certs\") pod \"e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm\" (UID: \"8deab213-f1c7-4883-9a64-72789a741faf\") " pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"
Nov 01 03:31:36.074213 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:36.072604    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8deab213-f1c7-4883-9a64-72789a741faf-credential-socket\") pod \"e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm\" (UID: \"8deab213-f1c7-4883-9a64-72789a741faf\") " pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"
Nov 01 03:31:36.085410 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:36.076452    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8deab213-f1c7-4883-9a64-72789a741faf-istiod-ca-cert\") pod \"e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm\" (UID: \"8deab213-f1c7-4883-9a64-72789a741faf\") " pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"
Nov 01 03:31:36.085410 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:36.076639    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8deab213-f1c7-4883-9a64-72789a741faf-workload-socket\") pod \"e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm\" (UID: \"8deab213-f1c7-4883-9a64-72789a741faf\") " pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"
Nov 01 03:31:36.094448 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:36.094416    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8deab213-f1c7-4883-9a64-72789a741faf-istio-envoy\") pod \"e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm\" (UID: \"8deab213-f1c7-4883-9a64-72789a741faf\") " pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"
Nov 01 03:31:36.096762 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:36.096737    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8deab213-f1c7-4883-9a64-72789a741faf-istio-podinfo\") pod \"e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm\" (UID: \"8deab213-f1c7-4883-9a64-72789a741faf\") " pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"
Nov 01 03:31:36.144880 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:36.144787    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:36.152640 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:36.149961    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm5lb\" (UniqueName: \"kubernetes.io/projected/8deab213-f1c7-4883-9a64-72789a741faf-kube-api-access-wm5lb\") pod \"e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm\" (UID: \"8deab213-f1c7-4883-9a64-72789a741faf\") " pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"
Nov 01 03:31:36.152824 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:36.152720    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8deab213-f1c7-4883-9a64-72789a741faf-istio-token\") pod \"e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm\" (UID: \"8deab213-f1c7-4883-9a64-72789a741faf\") " pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"
Nov 01 03:31:36.430213 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:36.430115    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"
Nov 01 03:31:36.551099 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:36.551069    4930 scope.go:122] "RemoveContainer" containerID="75cbc6f855f445fa673d0120006dbc6bc57936825d128b1e989c73370c4b43e4"
Nov 01 03:31:36.872567 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:31:36.872522    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8deab213_f1c7_4883_9a64_72789a741faf.slice/crio-74c28ce75257b1a93aa9fd70b5776b7c49c6c989d51b4d438c7abe24c375ffec WatchSource:0}: Error finding container 74c28ce75257b1a93aa9fd70b5776b7c49c6c989d51b4d438c7abe24c375ffec: Status 404 returned error can't find the container with id 74c28ce75257b1a93aa9fd70b5776b7c49c6c989d51b4d438c7abe24c375ffec
Nov 01 03:31:36.874043 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:36.874017    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"]
Nov 01 03:31:36.875519 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:36.875482    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:31:36.875654 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:36.875550    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:31:36.875654 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:36.875633    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:31:37.015285 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:37.015249    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm" event={"ID":"8deab213-f1c7-4883-9a64-72789a741faf","Type":"ContainerStarted","Data":"7be529d70abaa4f033fb767519510b1c2e98a21cbf5352fe59460d5b1aaebcde"}
Nov 01 03:31:37.015436 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:37.015318    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm" event={"ID":"8deab213-f1c7-4883-9a64-72789a741faf","Type":"ContainerStarted","Data":"74c28ce75257b1a93aa9fd70b5776b7c49c6c989d51b4d438c7abe24c375ffec"}
Nov 01 03:31:37.144703 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:37.144602    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm" podStartSLOduration=2.144583716 podStartE2EDuration="2.144583716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:31:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:31:37.136044246 +0000 UTC m=+1501.659050835" watchObservedRunningTime="2026-11-01 03:31:37.144583716 +0000 UTC m=+1501.667590307"
Nov 01 03:31:37.431067 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:37.430983    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"
Nov 01 03:31:37.436309 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:37.436281    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"
Nov 01 03:31:37.896726 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:31:37.896695    4930 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fd59ab7_fd50_4ce9_a290_7f018204eb66.slice/crio-conmon-5430b25a0e6a45da064e414f6a4a164a847485e26ff22c074a83dac8ca4b2a76.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fd59ab7_fd50_4ce9_a290_7f018204eb66.slice/crio-5430b25a0e6a45da064e414f6a4a164a847485e26ff22c074a83dac8ca4b2a76.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 03:31:38.025658 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:38.025622    4930 generic.go:356] "Generic (PLEG): container finished" podID="0fd59ab7-fd50-4ce9-a290-7f018204eb66" containerID="5430b25a0e6a45da064e414f6a4a164a847485e26ff22c074a83dac8ca4b2a76" exitCode=0
Nov 01 03:31:38.025808 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:38.025690    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f" event={"ID":"0fd59ab7-fd50-4ce9-a290-7f018204eb66","Type":"ContainerDied","Data":"5430b25a0e6a45da064e414f6a4a164a847485e26ff22c074a83dac8ca4b2a76"}
Nov 01 03:31:38.025808 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:38.025744    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f" event={"ID":"0fd59ab7-fd50-4ce9-a290-7f018204eb66","Type":"ContainerDied","Data":"3cf6635dca64f1c4bd30b0f2bb9f9b976bddc1a8044399ac43de6486eb51d30d"}
Nov 01 03:31:38.025808 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:38.025762    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cf6635dca64f1c4bd30b0f2bb9f9b976bddc1a8044399ac43de6486eb51d30d"
Nov 01 03:31:38.025808 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:38.025775    4930 scope.go:122] "RemoveContainer" containerID="fb093f377b7645fc576a77c9f9f21a1b0c51c1d7a97d12fc4c2a174636fa2df8"
Nov 01 03:31:38.026232 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:38.026210    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"
Nov 01 03:31:38.027389 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:38.027370    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"
Nov 01 03:31:38.045301 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:31:38.045267    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb093f377b7645fc576a77c9f9f21a1b0c51c1d7a97d12fc4c2a174636fa2df8\": container with ID starting with fb093f377b7645fc576a77c9f9f21a1b0c51c1d7a97d12fc4c2a174636fa2df8 not found: ID does not exist" containerID="fb093f377b7645fc576a77c9f9f21a1b0c51c1d7a97d12fc4c2a174636fa2df8"
Nov 01 03:31:38.047285 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:38.047263    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"
Nov 01 03:31:38.214760 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:38.214728    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fd59ab7-fd50-4ce9-a290-7f018204eb66-kube-api-access-247st\" (UniqueName: \"kubernetes.io/projected/0fd59ab7-fd50-4ce9-a290-7f018204eb66-kube-api-access-247st\") pod \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\" (UID: \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\") "
Nov 01 03:31:38.214960 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:38.214865    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0fd59ab7-fd50-4ce9-a290-7f018204eb66-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fd59ab7-fd50-4ce9-a290-7f018204eb66-istio-podinfo\") pod \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\" (UID: \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\") "
Nov 01 03:31:38.214960 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:38.214919    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fd59ab7-fd50-4ce9-a290-7f018204eb66-istio-token\" (UniqueName: \"kubernetes.io/projected/0fd59ab7-fd50-4ce9-a290-7f018204eb66-istio-token\") pod \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\" (UID: \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\") "
Nov 01 03:31:38.215084 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:38.214980    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fd59ab7-fd50-4ce9-a290-7f018204eb66-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fd59ab7-fd50-4ce9-a290-7f018204eb66-workload-certs\") pod \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\" (UID: \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\") "
Nov 01 03:31:38.215084 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:38.215046    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fd59ab7-fd50-4ce9-a290-7f018204eb66-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fd59ab7-fd50-4ce9-a290-7f018204eb66-istio-envoy\") pod \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\" (UID: \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\") "
Nov 01 03:31:38.215194 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:38.215080    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fd59ab7-fd50-4ce9-a290-7f018204eb66-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fd59ab7-fd50-4ce9-a290-7f018204eb66-istiod-ca-cert\") pod \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\" (UID: \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\") "
Nov 01 03:31:38.215194 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:38.215136    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fd59ab7-fd50-4ce9-a290-7f018204eb66-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd59ab7-fd50-4ce9-a290-7f018204eb66-workload-socket\") pod \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\" (UID: \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\") "
Nov 01 03:31:38.215194 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:38.215184    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fd59ab7-fd50-4ce9-a290-7f018204eb66-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fd59ab7-fd50-4ce9-a290-7f018204eb66-istio-data\") pod \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\" (UID: \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\") "
Nov 01 03:31:38.215421 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:38.215236    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fd59ab7-fd50-4ce9-a290-7f018204eb66-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd59ab7-fd50-4ce9-a290-7f018204eb66-credential-socket\") pod \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\" (UID: \"0fd59ab7-fd50-4ce9-a290-7f018204eb66\") "
Nov 01 03:31:38.216322 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:38.216292    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd59ab7-fd50-4ce9-a290-7f018204eb66-credential-socket" pod "0fd59ab7-fd50-4ce9-a290-7f018204eb66" (UID: "0fd59ab7-fd50-4ce9-a290-7f018204eb66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:31:38.217357 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:38.217326    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0fd59ab7-fd50-4ce9-a290-7f018204eb66-istio-podinfo" pod "0fd59ab7-fd50-4ce9-a290-7f018204eb66" (UID: "0fd59ab7-fd50-4ce9-a290-7f018204eb66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:31:38.217606 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:38.217577    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fd59ab7-fd50-4ce9-a290-7f018204eb66-kube-api-access-247st" pod "0fd59ab7-fd50-4ce9-a290-7f018204eb66" (UID: "0fd59ab7-fd50-4ce9-a290-7f018204eb66"). InnerVolumeSpecName "kube-api-access-247st". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:31:38.217706 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:38.217685    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fd59ab7-fd50-4ce9-a290-7f018204eb66-istiod-ca-cert" pod "0fd59ab7-fd50-4ce9-a290-7f018204eb66" (UID: "0fd59ab7-fd50-4ce9-a290-7f018204eb66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:31:38.217842 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:38.217819    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd59ab7-fd50-4ce9-a290-7f018204eb66-workload-certs" pod "0fd59ab7-fd50-4ce9-a290-7f018204eb66" (UID: "0fd59ab7-fd50-4ce9-a290-7f018204eb66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:31:38.218053 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:38.218029    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd59ab7-fd50-4ce9-a290-7f018204eb66-workload-socket" pod "0fd59ab7-fd50-4ce9-a290-7f018204eb66" (UID: "0fd59ab7-fd50-4ce9-a290-7f018204eb66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:31:38.218328 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:38.218216    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd59ab7-fd50-4ce9-a290-7f018204eb66-istio-data" pod "0fd59ab7-fd50-4ce9-a290-7f018204eb66" (UID: "0fd59ab7-fd50-4ce9-a290-7f018204eb66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:31:38.228734 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:38.228709    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd59ab7-fd50-4ce9-a290-7f018204eb66-istio-envoy" pod "0fd59ab7-fd50-4ce9-a290-7f018204eb66" (UID: "0fd59ab7-fd50-4ce9-a290-7f018204eb66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:31:38.229053 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:38.229025    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fd59ab7-fd50-4ce9-a290-7f018204eb66-istio-token" pod "0fd59ab7-fd50-4ce9-a290-7f018204eb66" (UID: "0fd59ab7-fd50-4ce9-a290-7f018204eb66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:31:38.317335 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:38.317290    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-247st\" (UniqueName: \"kubernetes.io/projected/0fd59ab7-fd50-4ce9-a290-7f018204eb66-kube-api-access-247st\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:31:38.317335 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:38.317332    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fd59ab7-fd50-4ce9-a290-7f018204eb66-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:31:38.317335 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:38.317343    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fd59ab7-fd50-4ce9-a290-7f018204eb66-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:31:38.317335 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:38.317352    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fd59ab7-fd50-4ce9-a290-7f018204eb66-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:31:38.317775 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:38.317361    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fd59ab7-fd50-4ce9-a290-7f018204eb66-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:31:38.317775 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:38.317372    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fd59ab7-fd50-4ce9-a290-7f018204eb66-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:31:38.317775 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:38.317387    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd59ab7-fd50-4ce9-a290-7f018204eb66-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:31:38.317775 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:38.317395    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fd59ab7-fd50-4ce9-a290-7f018204eb66-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:31:38.317775 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:38.317405    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd59ab7-fd50-4ce9-a290-7f018204eb66-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:31:38.405479 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:38.405429    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:38.609877 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:38.609841    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:38.788859 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:38.788812    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:39.036504 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:39.036432    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"
Nov 01 03:31:39.250920 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:39.250890    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"]
Nov 01 03:31:39.300212 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:39.300139    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d5c42938-openshift-default-55f4446845-pwf8f"]
Nov 01 03:31:39.622653 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:39.622623    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fd59ab7-fd50-4ce9-a290-7f018204eb66" path="/var/lib/kubelet/pods/0fd59ab7-fd50-4ce9-a290-7f018204eb66/volumes"
Nov 01 03:31:40.668630 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:40.668591    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:40.871215 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:40.871179    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:41.206588 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:41.199847    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:43.349516 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:43.349481    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:44.238921 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:44.238883    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:44.946427 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:44.946394    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:45.223602 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:45.223500    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:45.368898 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:45.368855    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:45.452703 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:45.452671    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6e7ea12-n4qjk"]
Nov 01 03:31:45.501748 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:45.501664    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e6e7ea12-n4qjk"]
Nov 01 03:31:45.624411 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:45.624379    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb7fa4d3-2f67-4dea-b20a-d8ffbceec029" path="/var/lib/kubelet/pods/cb7fa4d3-2f67-4dea-b20a-d8ffbceec029/volumes"
Nov 01 03:31:46.214496 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:46.214463    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:46.401576 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:46.395030    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:47.079188 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:47.079152    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:47.898035 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:47.897994    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:48.102954 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:48.102916    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:49.129345 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:31:49.129301    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:31:50.446163 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:50.446124    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:31:52.057407 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:52.057376    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60f05eb9-x2qlg"]
Nov 01 03:31:52.106773 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:52.106742    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60f05eb9-x2qlg"]
Nov 01 03:31:53.623796 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:53.623758    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9979f531-593e-431a-a614-2b5dcad44446" path="/var/lib/kubelet/pods/9979f531-593e-431a-a614-2b5dcad44446/volumes"
Nov 01 03:31:55.245363 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:31:55.245324    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:32:00.042506 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:00.042464    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:32:04.833349 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:04.833297    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:32:09.939486 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:09.939435    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:32:11.053221 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:11.053189    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fdc5677-fs7dr"]
Nov 01 03:32:11.103195 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:11.103160    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fdc5677-fs7dr"]
Nov 01 03:32:11.621716 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:11.621680    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="604d1816-9db7-471b-8ad7-7f11c819937d" path="/var/lib/kubelet/pods/604d1816-9db7-471b-8ad7-7f11c819937d/volumes"
Nov 01 03:32:14.745483 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:14.745442    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:32:16.200524 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:16.200482    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:32:19.123119 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:32:19.123083    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:32:19.487780 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:19.487673    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:32:22.310808 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:22.310767    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:32:22.544944 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:22.544882    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:32:24.053260 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:24.053223    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8358d178-lj8hq"]
Nov 01 03:32:24.105432 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:24.105403    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8358d178-lj8hq"]
Nov 01 03:32:24.121703 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:24.121656    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:32:24.317591 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:24.316507    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:32:24.496177 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:24.496139    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:32:25.621337 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:25.621307    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8b1a43c-397a-43d7-8c23-4a0638f1ac18" path="/var/lib/kubelet/pods/c8b1a43c-397a-43d7-8c23-4a0638f1ac18/volumes"
Nov 01 03:32:28.056171 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:28.056138    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40ab1571-h4jqz"]
Nov 01 03:32:28.105604 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:28.105531    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40ab1571-h4jqz"]
Nov 01 03:32:28.701900 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:28.701819    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:32:28.892784 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:28.892744    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:32:29.038100 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:29.038012    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:32:29.622241 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:29.622205    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7b6d3eb-e0cf-40b9-bfa2-e543626883fa" path="/var/lib/kubelet/pods/e7b6d3eb-e0cf-40b9-bfa2-e543626883fa/volumes"
Nov 01 03:32:30.053779 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:30.053694    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d87394e-pgwwl"]
Nov 01 03:32:30.102123 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:30.102095    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d87394e-pgwwl"]
Nov 01 03:32:31.123974 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:31.123935    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:32:31.410634 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:31.408205    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:32:31.622576 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:31.599200    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:32:31.696213 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:31.696132    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5211cbab-e13b-4e4e-899d-3195a8c8ce23" path="/var/lib/kubelet/pods/5211cbab-e13b-4e4e-899d-3195a8c8ce23/volumes"
Nov 01 03:32:33.747749 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:33.747718    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:32:35.084573 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:35.084523    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:32:36.609848 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:36.609814    4930 scope.go:122] "RemoveContainer" containerID="3d286c6c758c527acdba11ced454de710d99b0b9d9a7b726b937875c5bd683b5"
Nov 01 03:32:36.621375 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:36.621354    4930 scope.go:122] "RemoveContainer" containerID="e0e1d464c7abb6214395efa5bf6573b3d8ca32c8cbf6e090b7d3a557585cf000"
Nov 01 03:32:36.631774 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:36.631750    4930 scope.go:122] "RemoveContainer" containerID="63346684a0712acb2706b4cfa16013fff396dda94465ed609e094b2e1cf9bd4a"
Nov 01 03:32:38.006430 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:38.006389    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:32:38.200976 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:38.200920    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:32:38.506067 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:38.506007    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:32:38.648089 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:38.648054    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:32:39.187583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:39.166285    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:32:39.337782 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:39.337279    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:32:40.133150 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:40.133094    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bd986dd-fk4kj"]
Nov 01 03:32:40.188834 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:40.188800    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9134a935-tb9nz"]
Nov 01 03:32:40.197922 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:40.197872    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bd986dd-fk4kj"]
Nov 01 03:32:40.206263 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:40.206224    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9134a935-tb9nz"]
Nov 01 03:32:41.627165 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:41.627128    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dc62e14-76de-4a5a-bb5c-0d79b591be39" path="/var/lib/kubelet/pods/2dc62e14-76de-4a5a-bb5c-0d79b591be39/volumes"
Nov 01 03:32:41.628126 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:41.628100    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82e946ea-cee9-4205-bace-d5132fc6b754" path="/var/lib/kubelet/pods/82e946ea-cee9-4205-bace-d5132fc6b754/volumes"
Nov 01 03:32:43.354843 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:43.354809    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:32:43.597194 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:43.597156    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:32:43.759326 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:43.759242    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:32:43.909600 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:43.909550    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:32:44.055112 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:44.055013    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faa90817-9lvwc"]
Nov 01 03:32:44.098416 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:44.098382    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faa90817-9lvwc"]
Nov 01 03:32:44.572494 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:44.572460    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:32:45.623689 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:45.623658    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19139bc8-a17f-4093-b596-4ac8225e260c" path="/var/lib/kubelet/pods/19139bc8-a17f-4093-b596-4ac8225e260c/volumes"
Nov 01 03:32:46.621146 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:46.621110    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:32:46.821706 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:46.821517    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:32:48.860551 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:48.860512    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:32:49.123441 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:32:49.123351    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:32:49.928963 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:49.928932    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-44379c05-vwm69"]
Nov 01 03:32:49.929632 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:49.929617    4930 cpu_manager.go:414] "containerMap: removing container" podUID="0fd59ab7-fd50-4ce9-a290-7f018204eb66" containerName="istio-proxy"
Nov 01 03:32:49.929689 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:49.929632    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fd59ab7-fd50-4ce9-a290-7f018204eb66" containerName="istio-proxy"
Nov 01 03:32:49.929689 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:49.929645    4930 cpu_manager.go:414] "containerMap: removing container" podUID="0fd59ab7-fd50-4ce9-a290-7f018204eb66" containerName="istio-proxy"
Nov 01 03:32:49.929689 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:49.929650    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fd59ab7-fd50-4ce9-a290-7f018204eb66" containerName="istio-proxy"
Nov 01 03:32:49.929791 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:49.929766    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fd59ab7-fd50-4ce9-a290-7f018204eb66" containerName="istio-proxy"
Nov 01 03:32:49.929791 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:49.929776    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fd59ab7-fd50-4ce9-a290-7f018204eb66" containerName="istio-proxy"
Nov 01 03:32:49.935218 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:49.935197    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-44379c05-vwm69"
Nov 01 03:32:50.016545 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:50.016505    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-44379c05-vwm69"]
Nov 01 03:32:50.035499 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:50.035466    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:32:50.035658 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:50.035512    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:32:50.035658 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:50.035532    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:32:50.035658 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:50.035512    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:32:50.039278 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:50.039251    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b68fed2-7cce-4c2b-8476-9205ef4cf5e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-44379c05-vwm69\" (UID: \"6b68fed2-7cce-4c2b-8476-9205ef4cf5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-44379c05-vwm69"
Nov 01 03:32:50.039457 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:50.039438    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk5jm\" (UniqueName: \"kubernetes.io/projected/6b68fed2-7cce-4c2b-8476-9205ef4cf5e4-kube-api-access-gk5jm\") pod \"maas-api-revoke-keys-e2e-mt-44379c05-vwm69\" (UID: \"6b68fed2-7cce-4c2b-8476-9205ef4cf5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-44379c05-vwm69"
Nov 01 03:32:50.039512 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:50.039495    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b68fed2-7cce-4c2b-8476-9205ef4cf5e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-44379c05-vwm69\" (UID: \"6b68fed2-7cce-4c2b-8476-9205ef4cf5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-44379c05-vwm69"
Nov 01 03:32:50.140777 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:50.140742    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b68fed2-7cce-4c2b-8476-9205ef4cf5e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-44379c05-vwm69\" (UID: \"6b68fed2-7cce-4c2b-8476-9205ef4cf5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-44379c05-vwm69"
Nov 01 03:32:50.140916 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:50.140893    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk5jm\" (UniqueName: \"kubernetes.io/projected/6b68fed2-7cce-4c2b-8476-9205ef4cf5e4-kube-api-access-gk5jm\") pod \"maas-api-revoke-keys-e2e-mt-44379c05-vwm69\" (UID: \"6b68fed2-7cce-4c2b-8476-9205ef4cf5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-44379c05-vwm69"
Nov 01 03:32:50.141003 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:50.140954    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b68fed2-7cce-4c2b-8476-9205ef4cf5e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-44379c05-vwm69\" (UID: \"6b68fed2-7cce-4c2b-8476-9205ef4cf5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-44379c05-vwm69"
Nov 01 03:32:50.141446 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:50.141414    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b68fed2-7cce-4c2b-8476-9205ef4cf5e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-44379c05-vwm69\" (UID: \"6b68fed2-7cce-4c2b-8476-9205ef4cf5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-44379c05-vwm69"
Nov 01 03:32:50.143364 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:50.143341    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b68fed2-7cce-4c2b-8476-9205ef4cf5e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-44379c05-vwm69\" (UID: \"6b68fed2-7cce-4c2b-8476-9205ef4cf5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-44379c05-vwm69"
Nov 01 03:32:50.202714 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:50.202642    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk5jm\" (UniqueName: \"kubernetes.io/projected/6b68fed2-7cce-4c2b-8476-9205ef4cf5e4-kube-api-access-gk5jm\") pod \"maas-api-revoke-keys-e2e-mt-44379c05-vwm69\" (UID: \"6b68fed2-7cce-4c2b-8476-9205ef4cf5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-44379c05-vwm69"
Nov 01 03:32:50.249526 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:50.249498    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-44379c05-vwm69"
Nov 01 03:32:50.640059 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:32:50.640023    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b68fed2_7cce_4c2b_8476_9205ef4cf5e4.slice/crio-a12559710aec9f37dd18f3aa0abfebf1ab975286027bd8e9492e1ac5475d651b WatchSource:0}: Error finding container a12559710aec9f37dd18f3aa0abfebf1ab975286027bd8e9492e1ac5475d651b: Status 404 returned error can't find the container with id a12559710aec9f37dd18f3aa0abfebf1ab975286027bd8e9492e1ac5475d651b
Nov 01 03:32:50.640910 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:50.640886    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-44379c05-vwm69"]
Nov 01 03:32:51.220619 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:51.220550    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-44379c05-vwm69" event={"ID":"6b68fed2-7cce-4c2b-8476-9205ef4cf5e4","Type":"ContainerStarted","Data":"25b6b1b773359bf82ee3bd9966bd600a22f9cf59454b59f548ec34f1c0afcb61"}
Nov 01 03:32:51.221062 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:51.220647    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-44379c05-vwm69" event={"ID":"6b68fed2-7cce-4c2b-8476-9205ef4cf5e4","Type":"ContainerStarted","Data":"a12559710aec9f37dd18f3aa0abfebf1ab975286027bd8e9492e1ac5475d651b"}
Nov 01 03:32:51.341799 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:51.341742    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-44379c05-vwm69" podStartSLOduration=2.34172823 podStartE2EDuration="2.34172823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:32:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:32:51.33780427 +0000 UTC m=+1575.860810863" watchObservedRunningTime="2026-11-01 03:32:51.34172823 +0000 UTC m=+1575.864734843"
Nov 01 03:32:52.230931 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:52.230895    4930 generic.go:356] "Generic (PLEG): container finished" podID="6b68fed2-7cce-4c2b-8476-9205ef4cf5e4" containerID="25b6b1b773359bf82ee3bd9966bd600a22f9cf59454b59f548ec34f1c0afcb61" exitCode=0
Nov 01 03:32:52.231330 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:52.230976    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-44379c05-vwm69" event={"ID":"6b68fed2-7cce-4c2b-8476-9205ef4cf5e4","Type":"ContainerDied","Data":"25b6b1b773359bf82ee3bd9966bd600a22f9cf59454b59f548ec34f1c0afcb61"}
Nov 01 03:32:53.429491 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:53.429463    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-44379c05-vwm69"
Nov 01 03:32:53.500482 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:53.500412    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b68fed2-7cce-4c2b-8476-9205ef4cf5e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b68fed2-7cce-4c2b-8476-9205ef4cf5e4-maas-api-openshift-ca\") pod \"6b68fed2-7cce-4c2b-8476-9205ef4cf5e4\" (UID: \"6b68fed2-7cce-4c2b-8476-9205ef4cf5e4\") "
Nov 01 03:32:53.500482 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:53.500463    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b68fed2-7cce-4c2b-8476-9205ef4cf5e4-kube-api-access-gk5jm\" (UniqueName: \"kubernetes.io/projected/6b68fed2-7cce-4c2b-8476-9205ef4cf5e4-kube-api-access-gk5jm\") pod \"6b68fed2-7cce-4c2b-8476-9205ef4cf5e4\" (UID: \"6b68fed2-7cce-4c2b-8476-9205ef4cf5e4\") "
Nov 01 03:32:53.500744 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:53.500490    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b68fed2-7cce-4c2b-8476-9205ef4cf5e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b68fed2-7cce-4c2b-8476-9205ef4cf5e4-maas-api-xks-ca\") pod \"6b68fed2-7cce-4c2b-8476-9205ef4cf5e4\" (UID: \"6b68fed2-7cce-4c2b-8476-9205ef4cf5e4\") "
Nov 01 03:32:53.500810 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:53.500772    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b68fed2-7cce-4c2b-8476-9205ef4cf5e4-maas-api-openshift-ca" pod "6b68fed2-7cce-4c2b-8476-9205ef4cf5e4" (UID: "6b68fed2-7cce-4c2b-8476-9205ef4cf5e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:32:53.501319 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:53.501293    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b68fed2-7cce-4c2b-8476-9205ef4cf5e4-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:32:53.502815 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:53.502791    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b68fed2-7cce-4c2b-8476-9205ef4cf5e4-kube-api-access-gk5jm" pod "6b68fed2-7cce-4c2b-8476-9205ef4cf5e4" (UID: "6b68fed2-7cce-4c2b-8476-9205ef4cf5e4"). InnerVolumeSpecName "kube-api-access-gk5jm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:32:53.502941 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:53.502869    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b68fed2-7cce-4c2b-8476-9205ef4cf5e4-maas-api-xks-ca" pod "6b68fed2-7cce-4c2b-8476-9205ef4cf5e4" (UID: "6b68fed2-7cce-4c2b-8476-9205ef4cf5e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:32:53.603319 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:53.603279    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk5jm\" (UniqueName: \"kubernetes.io/projected/6b68fed2-7cce-4c2b-8476-9205ef4cf5e4-kube-api-access-gk5jm\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:32:53.603577 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:53.603534    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b68fed2-7cce-4c2b-8476-9205ef4cf5e4-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:32:54.052486 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:54.052439    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c979941-n9zgq"]
Nov 01 03:32:54.062127 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:54.062096    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c979941-n9zgq"]
Nov 01 03:32:54.147939 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:54.147902    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:32:54.251669 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:54.251637    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-44379c05-vwm69"
Nov 01 03:32:54.251901 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:54.251876    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-44379c05-vwm69" event={"ID":"6b68fed2-7cce-4c2b-8476-9205ef4cf5e4","Type":"ContainerDied","Data":"a12559710aec9f37dd18f3aa0abfebf1ab975286027bd8e9492e1ac5475d651b"}
Nov 01 03:32:54.251956 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:54.251927    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a12559710aec9f37dd18f3aa0abfebf1ab975286027bd8e9492e1ac5475d651b"
Nov 01 03:32:55.623090 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:55.623059    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f046a7c-23d0-4995-88a1-2046b50666e6" path="/var/lib/kubelet/pods/0f046a7c-23d0-4995-88a1-2046b50666e6/volumes"
Nov 01 03:32:59.323505 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:32:59.323471    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:33:00.050056 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:00.050013    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa25001-m46sl"]
Nov 01 03:33:00.098894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:00.098861    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afa25001-m46sl"]
Nov 01 03:33:01.635125 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:01.635088    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f68d831-6076-4bf4-806d-742446d2dfbf" path="/var/lib/kubelet/pods/4f68d831-6076-4bf4-806d-742446d2dfbf/volumes"
Nov 01 03:33:01.764728 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:01.764690    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:33:01.956233 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:01.956145    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:33:04.061856 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:04.061824    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-124f550e-hrtsq"]
Nov 01 03:33:04.110188 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:04.110160    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-124f550e-hrtsq"]
Nov 01 03:33:04.479968 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:04.479928    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:33:05.049004 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:05.048969    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ed506bf-t2d2h"]
Nov 01 03:33:05.098483 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:05.098451    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ed506bf-t2d2h"]
Nov 01 03:33:05.623490 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:05.623457    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17c52d6e-b715-45f7-9cab-31b383dbce20" path="/var/lib/kubelet/pods/17c52d6e-b715-45f7-9cab-31b383dbce20/volumes"
Nov 01 03:33:05.623985 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:05.623970    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30aa6d42-2084-4a5c-a1c0-34037d291406" path="/var/lib/kubelet/pods/30aa6d42-2084-4a5c-a1c0-34037d291406/volumes"
Nov 01 03:33:09.336611 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:09.336549    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:33:13.051171 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:13.051127    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-305021d0-6nlmc"]
Nov 01 03:33:13.100251 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:13.100217    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-305021d0-6nlmc"]
Nov 01 03:33:13.622127 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:13.622090    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2242792-3f71-4878-9b72-4f2bac2c3f76" path="/var/lib/kubelet/pods/d2242792-3f71-4878-9b72-4f2bac2c3f76/volumes"
Nov 01 03:33:14.054895 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:14.054817    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8f22c0d-827rx"]
Nov 01 03:33:14.078869 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:14.078835    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-184a059e-jv2ln"]
Nov 01 03:33:14.085224 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:14.085197    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:33:14.126957 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:14.126925    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8f22c0d-827rx"]
Nov 01 03:33:14.166712 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:14.166676    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-184a059e-jv2ln"]
Nov 01 03:33:15.623777 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:15.623741    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bfb43fb-7e73-4be2-80f2-e4361bfa096a" path="/var/lib/kubelet/pods/6bfb43fb-7e73-4be2-80f2-e4361bfa096a/volumes"
Nov 01 03:33:15.624283 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:15.624267    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc079b72-10cb-4437-afa0-74902f4ef55c" path="/var/lib/kubelet/pods/dc079b72-10cb-4437-afa0-74902f4ef55c/volumes"
Nov 01 03:33:16.059388 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:16.059309    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-210e6572-lxcdd"]
Nov 01 03:33:16.108734 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:16.108703    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-210e6572-lxcdd"]
Nov 01 03:33:17.621615 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:17.621544    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dce274d0-64ca-459a-995e-651c9d548965" path="/var/lib/kubelet/pods/dce274d0-64ca-459a-995e-651c9d548965/volumes"
Nov 01 03:33:18.936490 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:18.936446    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:33:19.131379 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:33:19.131338    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:33:23.050892 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:23.050856    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53513363-zgf2q"]
Nov 01 03:33:23.100648 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:23.100613    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53513363-zgf2q"]
Nov 01 03:33:23.420095 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:23.420047    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:33:23.626838 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:23.626800    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8c55607-460a-4562-b07e-1f0274d9f917" path="/var/lib/kubelet/pods/c8c55607-460a-4562-b07e-1f0274d9f917/volumes"
Nov 01 03:33:25.055977 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:25.055936    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-518af558-8p2xt"]
Nov 01 03:33:25.107864 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:25.107822    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-518af558-8p2xt"]
Nov 01 03:33:25.471159 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:25.471116    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d262c155-9rls7"]
Nov 01 03:33:25.472265 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:25.472243    4930 cpu_manager.go:414] "containerMap: removing container" podUID="6b68fed2-7cce-4c2b-8476-9205ef4cf5e4" containerName="revoke-keys"
Nov 01 03:33:25.472330 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:25.472268    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b68fed2-7cce-4c2b-8476-9205ef4cf5e4" containerName="revoke-keys"
Nov 01 03:33:25.472458 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:25.472446    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b68fed2-7cce-4c2b-8476-9205ef4cf5e4" containerName="revoke-keys"
Nov 01 03:33:25.479980 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:25.479957    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d262c155-9rls7"
Nov 01 03:33:25.534186 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:25.534155    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:33:25.534388 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:25.534313    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:33:25.534470 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:25.534419    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:33:25.535050 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:25.535028    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:33:25.557575 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:25.539751    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d262c155-9rls7"]
Nov 01 03:33:25.590590 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:25.590029    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b77e2e1-bc9d-4451-93eb-66dddf7b0f74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d262c155-9rls7\" (UID: \"6b77e2e1-bc9d-4451-93eb-66dddf7b0f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d262c155-9rls7"
Nov 01 03:33:25.590590 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:25.590279    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4kf2\" (UniqueName: \"kubernetes.io/projected/6b77e2e1-bc9d-4451-93eb-66dddf7b0f74-kube-api-access-l4kf2\") pod \"maas-api-revoke-keys-e2e-migrate-d262c155-9rls7\" (UID: \"6b77e2e1-bc9d-4451-93eb-66dddf7b0f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d262c155-9rls7"
Nov 01 03:33:25.590590 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:25.590347    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b77e2e1-bc9d-4451-93eb-66dddf7b0f74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d262c155-9rls7\" (UID: \"6b77e2e1-bc9d-4451-93eb-66dddf7b0f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d262c155-9rls7"
Nov 01 03:33:25.626199 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:25.626159    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc5cffef-d29f-4ade-b650-f8904b356bcd" path="/var/lib/kubelet/pods/fc5cffef-d29f-4ade-b650-f8904b356bcd/volumes"
Nov 01 03:33:25.693483 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:25.692003    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b77e2e1-bc9d-4451-93eb-66dddf7b0f74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d262c155-9rls7\" (UID: \"6b77e2e1-bc9d-4451-93eb-66dddf7b0f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d262c155-9rls7"
Nov 01 03:33:25.693483 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:25.692317    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4kf2\" (UniqueName: \"kubernetes.io/projected/6b77e2e1-bc9d-4451-93eb-66dddf7b0f74-kube-api-access-l4kf2\") pod \"maas-api-revoke-keys-e2e-migrate-d262c155-9rls7\" (UID: \"6b77e2e1-bc9d-4451-93eb-66dddf7b0f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d262c155-9rls7"
Nov 01 03:33:25.693483 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:25.692390    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b77e2e1-bc9d-4451-93eb-66dddf7b0f74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d262c155-9rls7\" (UID: \"6b77e2e1-bc9d-4451-93eb-66dddf7b0f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d262c155-9rls7"
Nov 01 03:33:25.696596 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:25.696530    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b77e2e1-bc9d-4451-93eb-66dddf7b0f74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d262c155-9rls7\" (UID: \"6b77e2e1-bc9d-4451-93eb-66dddf7b0f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d262c155-9rls7"
Nov 01 03:33:25.698306 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:25.698282    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b77e2e1-bc9d-4451-93eb-66dddf7b0f74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d262c155-9rls7\" (UID: \"6b77e2e1-bc9d-4451-93eb-66dddf7b0f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d262c155-9rls7"
Nov 01 03:33:25.755791 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:25.755716    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4kf2\" (UniqueName: \"kubernetes.io/projected/6b77e2e1-bc9d-4451-93eb-66dddf7b0f74-kube-api-access-l4kf2\") pod \"maas-api-revoke-keys-e2e-migrate-d262c155-9rls7\" (UID: \"6b77e2e1-bc9d-4451-93eb-66dddf7b0f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d262c155-9rls7"
Nov 01 03:33:25.800194 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:25.800158    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d262c155-9rls7"
Nov 01 03:33:26.262089 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:33:26.262057    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b77e2e1_bc9d_4451_93eb_66dddf7b0f74.slice/crio-257e75b1325f5eda82cd432b99b3876a9fc677be29fa54733e0525bb42f4f1e5 WatchSource:0}: Error finding container 257e75b1325f5eda82cd432b99b3876a9fc677be29fa54733e0525bb42f4f1e5: Status 404 returned error can't find the container with id 257e75b1325f5eda82cd432b99b3876a9fc677be29fa54733e0525bb42f4f1e5
Nov 01 03:33:26.275229 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:26.265665    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d262c155-9rls7"]
Nov 01 03:33:26.675619 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:26.675582    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d262c155-9rls7" event={"ID":"6b77e2e1-bc9d-4451-93eb-66dddf7b0f74","Type":"ContainerStarted","Data":"257e75b1325f5eda82cd432b99b3876a9fc677be29fa54733e0525bb42f4f1e5"}
Nov 01 03:33:27.689640 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:27.689510    4930 generic.go:356] "Generic (PLEG): container finished" podID="6b77e2e1-bc9d-4451-93eb-66dddf7b0f74" containerID="5d73e91b053db311d1b537e81d2c3179bd6d0d6cb72e0ca7e66ade5cadaac9b6" exitCode=0
Nov 01 03:33:27.690212 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:27.689627    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d262c155-9rls7" event={"ID":"6b77e2e1-bc9d-4451-93eb-66dddf7b0f74","Type":"ContainerDied","Data":"5d73e91b053db311d1b537e81d2c3179bd6d0d6cb72e0ca7e66ade5cadaac9b6"}
Nov 01 03:33:28.700592 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:28.700465    4930 generic.go:356] "Generic (PLEG): container finished" podID="8deab213-f1c7-4883-9a64-72789a741faf" containerID="7be529d70abaa4f033fb767519510b1c2e98a21cbf5352fe59460d5b1aaebcde" exitCode=0
Nov 01 03:33:28.700592 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:28.700539    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm" event={"ID":"8deab213-f1c7-4883-9a64-72789a741faf","Type":"ContainerDied","Data":"7be529d70abaa4f033fb767519510b1c2e98a21cbf5352fe59460d5b1aaebcde"}
Nov 01 03:33:28.701187 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:28.701052    4930 scope.go:122] "RemoveContainer" containerID="7be529d70abaa4f033fb767519510b1c2e98a21cbf5352fe59460d5b1aaebcde"
Nov 01 03:33:28.702845 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:28.702811    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:33:28.702970 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:28.702891    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:33:28.702970 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:28.702931    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:33:28.775234 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:28.775181    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:33:29.014047 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:29.013967    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d262c155-9rls7"
Nov 01 03:33:29.182859 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:29.182808    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b77e2e1-bc9d-4451-93eb-66dddf7b0f74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b77e2e1-bc9d-4451-93eb-66dddf7b0f74-maas-api-xks-ca\") pod \"6b77e2e1-bc9d-4451-93eb-66dddf7b0f74\" (UID: \"6b77e2e1-bc9d-4451-93eb-66dddf7b0f74\") "
Nov 01 03:33:29.183082 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:29.183065    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b77e2e1-bc9d-4451-93eb-66dddf7b0f74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b77e2e1-bc9d-4451-93eb-66dddf7b0f74-maas-api-openshift-ca\") pod \"6b77e2e1-bc9d-4451-93eb-66dddf7b0f74\" (UID: \"6b77e2e1-bc9d-4451-93eb-66dddf7b0f74\") "
Nov 01 03:33:29.183283 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:29.183255    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b77e2e1-bc9d-4451-93eb-66dddf7b0f74-kube-api-access-l4kf2\" (UniqueName: \"kubernetes.io/projected/6b77e2e1-bc9d-4451-93eb-66dddf7b0f74-kube-api-access-l4kf2\") pod \"6b77e2e1-bc9d-4451-93eb-66dddf7b0f74\" (UID: \"6b77e2e1-bc9d-4451-93eb-66dddf7b0f74\") "
Nov 01 03:33:29.184187 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:29.184142    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b77e2e1-bc9d-4451-93eb-66dddf7b0f74-maas-api-openshift-ca" pod "6b77e2e1-bc9d-4451-93eb-66dddf7b0f74" (UID: "6b77e2e1-bc9d-4451-93eb-66dddf7b0f74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:33:29.202547 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:29.202504    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b77e2e1-bc9d-4451-93eb-66dddf7b0f74-maas-api-xks-ca" pod "6b77e2e1-bc9d-4451-93eb-66dddf7b0f74" (UID: "6b77e2e1-bc9d-4451-93eb-66dddf7b0f74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:33:29.202726 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:29.202620    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b77e2e1-bc9d-4451-93eb-66dddf7b0f74-kube-api-access-l4kf2" pod "6b77e2e1-bc9d-4451-93eb-66dddf7b0f74" (UID: "6b77e2e1-bc9d-4451-93eb-66dddf7b0f74"). InnerVolumeSpecName "kube-api-access-l4kf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:33:29.286581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:29.286486    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b77e2e1-bc9d-4451-93eb-66dddf7b0f74-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:33:29.286581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:29.286524    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4kf2\" (UniqueName: \"kubernetes.io/projected/6b77e2e1-bc9d-4451-93eb-66dddf7b0f74-kube-api-access-l4kf2\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:33:29.286581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:29.286541    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b77e2e1-bc9d-4451-93eb-66dddf7b0f74-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:33:29.712426 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:29.712389    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm" event={"ID":"8deab213-f1c7-4883-9a64-72789a741faf","Type":"ContainerStarted","Data":"682dc443e16ae0a10a65396ab37473b089b9c19e7228bd4a6a0470ff4010939a"}
Nov 01 03:33:29.714016 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:29.713994    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d262c155-9rls7"
Nov 01 03:33:29.714152 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:29.714008    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d262c155-9rls7" event={"ID":"6b77e2e1-bc9d-4451-93eb-66dddf7b0f74","Type":"ContainerDied","Data":"257e75b1325f5eda82cd432b99b3876a9fc677be29fa54733e0525bb42f4f1e5"}
Nov 01 03:33:29.714152 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:29.714055    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="257e75b1325f5eda82cd432b99b3876a9fc677be29fa54733e0525bb42f4f1e5"
Nov 01 03:33:30.431185 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:30.431149    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"
Nov 01 03:33:30.431424 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:30.431399    4930 patch_prober.go:29] interesting pod/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Nov 01 03:33:30.431521 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:30.431497    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm" podUID="8deab213-f1c7-4883-9a64-72789a741faf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Nov 01 03:33:31.414708 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:31.412537    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3513c2f1-dpfxx"]
Nov 01 03:33:31.414708 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:31.414162    4930 cpu_manager.go:414] "containerMap: removing container" podUID="6b77e2e1-bc9d-4451-93eb-66dddf7b0f74" containerName="revoke-keys"
Nov 01 03:33:31.414708 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:31.414184    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b77e2e1-bc9d-4451-93eb-66dddf7b0f74" containerName="revoke-keys"
Nov 01 03:33:31.414708 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:31.414525    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b77e2e1-bc9d-4451-93eb-66dddf7b0f74" containerName="revoke-keys"
Nov 01 03:33:31.418268 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:31.418246    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3513c2f1-dpfxx"
Nov 01 03:33:31.430631 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:31.430497    4930 patch_prober.go:29] interesting pod/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Nov 01 03:33:31.430631 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:31.430540    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm" podUID="8deab213-f1c7-4883-9a64-72789a741faf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Nov 01 03:33:31.477579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:31.459274    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3513c2f1-dpfxx"]
Nov 01 03:33:31.506610 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:31.502354    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:33:31.506610 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:31.502383    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:33:31.506610 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:31.502388    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:33:31.506610 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:31.502422    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:33:31.524700 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:31.524667    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/414a2706-7d57-4edb-a3fd-8946b9d07729-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-3513c2f1-dpfxx\" (UID: \"414a2706-7d57-4edb-a3fd-8946b9d07729\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3513c2f1-dpfxx"
Nov 01 03:33:31.524929 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:31.524714    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/414a2706-7d57-4edb-a3fd-8946b9d07729-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-3513c2f1-dpfxx\" (UID: \"414a2706-7d57-4edb-a3fd-8946b9d07729\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3513c2f1-dpfxx"
Nov 01 03:33:31.524929 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:31.524909    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn2w4\" (UniqueName: \"kubernetes.io/projected/414a2706-7d57-4edb-a3fd-8946b9d07729-kube-api-access-bn2w4\") pod \"maas-api-revoke-keys-e2e-derive-3513c2f1-dpfxx\" (UID: \"414a2706-7d57-4edb-a3fd-8946b9d07729\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3513c2f1-dpfxx"
Nov 01 03:33:31.626103 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:31.626067    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn2w4\" (UniqueName: \"kubernetes.io/projected/414a2706-7d57-4edb-a3fd-8946b9d07729-kube-api-access-bn2w4\") pod \"maas-api-revoke-keys-e2e-derive-3513c2f1-dpfxx\" (UID: \"414a2706-7d57-4edb-a3fd-8946b9d07729\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3513c2f1-dpfxx"
Nov 01 03:33:31.626330 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:31.626313    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/414a2706-7d57-4edb-a3fd-8946b9d07729-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-3513c2f1-dpfxx\" (UID: \"414a2706-7d57-4edb-a3fd-8946b9d07729\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3513c2f1-dpfxx"
Nov 01 03:33:31.626410 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:31.626365    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/414a2706-7d57-4edb-a3fd-8946b9d07729-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-3513c2f1-dpfxx\" (UID: \"414a2706-7d57-4edb-a3fd-8946b9d07729\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3513c2f1-dpfxx"
Nov 01 03:33:31.627754 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:31.627672    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/414a2706-7d57-4edb-a3fd-8946b9d07729-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-3513c2f1-dpfxx\" (UID: \"414a2706-7d57-4edb-a3fd-8946b9d07729\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3513c2f1-dpfxx"
Nov 01 03:33:31.629636 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:31.629611    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/414a2706-7d57-4edb-a3fd-8946b9d07729-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-3513c2f1-dpfxx\" (UID: \"414a2706-7d57-4edb-a3fd-8946b9d07729\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3513c2f1-dpfxx"
Nov 01 03:33:31.688199 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:31.688109    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn2w4\" (UniqueName: \"kubernetes.io/projected/414a2706-7d57-4edb-a3fd-8946b9d07729-kube-api-access-bn2w4\") pod \"maas-api-revoke-keys-e2e-derive-3513c2f1-dpfxx\" (UID: \"414a2706-7d57-4edb-a3fd-8946b9d07729\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3513c2f1-dpfxx"
Nov 01 03:33:31.735852 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:31.735823    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3513c2f1-dpfxx"
Nov 01 03:33:32.187367 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:32.186648    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3513c2f1-dpfxx"]
Nov 01 03:33:32.431072 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:32.431043    4930 patch_prober.go:29] interesting pod/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Nov 01 03:33:32.431408 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:32.431106    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm" podUID="8deab213-f1c7-4883-9a64-72789a741faf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Nov 01 03:33:32.745511 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:32.745421    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3513c2f1-dpfxx" event={"ID":"414a2706-7d57-4edb-a3fd-8946b9d07729","Type":"ContainerStarted","Data":"2681dcc05077d544151263f69c09dca31383983aed088dbb68e80711be751c7d"}
Nov 01 03:33:32.745511 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:32.745486    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3513c2f1-dpfxx" event={"ID":"414a2706-7d57-4edb-a3fd-8946b9d07729","Type":"ContainerStarted","Data":"061db564f5b89b7ab5d4afdd30a504631d5cf06f5885bc8b98e0c6227fb5aa5d"}
Nov 01 03:33:32.880539 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:32.880471    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3513c2f1-dpfxx" podStartSLOduration=1.880448587 podStartE2EDuration="1.880448587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:33:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:33:32.864213849 +0000 UTC m=+1617.387220437" watchObservedRunningTime="2026-11-01 03:33:32.880448587 +0000 UTC m=+1617.403455177"
Nov 01 03:33:33.430878 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:33.430844    4930 patch_prober.go:29] interesting pod/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Nov 01 03:33:33.431039 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:33.430905    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm" podUID="8deab213-f1c7-4883-9a64-72789a741faf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Nov 01 03:33:33.755945 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:33.755835    4930 generic.go:356] "Generic (PLEG): container finished" podID="414a2706-7d57-4edb-a3fd-8946b9d07729" containerID="2681dcc05077d544151263f69c09dca31383983aed088dbb68e80711be751c7d" exitCode=0
Nov 01 03:33:33.755945 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:33.755917    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3513c2f1-dpfxx" event={"ID":"414a2706-7d57-4edb-a3fd-8946b9d07729","Type":"ContainerDied","Data":"2681dcc05077d544151263f69c09dca31383983aed088dbb68e80711be751c7d"}
Nov 01 03:33:34.054653 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:34.051723    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:33:34.431028 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:34.430991    4930 patch_prober.go:29] interesting pod/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Nov 01 03:33:34.431215 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:34.431047    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm" podUID="8deab213-f1c7-4883-9a64-72789a741faf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Nov 01 03:33:34.740035 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:34.739954    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:33:34.933585 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:34.932930    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:33:34.981412 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:34.981386    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3513c2f1-dpfxx"
Nov 01 03:33:35.119177 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:35.119138    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/414a2706-7d57-4edb-a3fd-8946b9d07729-kube-api-access-bn2w4\" (UniqueName: \"kubernetes.io/projected/414a2706-7d57-4edb-a3fd-8946b9d07729-kube-api-access-bn2w4\") pod \"414a2706-7d57-4edb-a3fd-8946b9d07729\" (UID: \"414a2706-7d57-4edb-a3fd-8946b9d07729\") "
Nov 01 03:33:35.119666 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:35.119640    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/414a2706-7d57-4edb-a3fd-8946b9d07729-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/414a2706-7d57-4edb-a3fd-8946b9d07729-maas-api-xks-ca\") pod \"414a2706-7d57-4edb-a3fd-8946b9d07729\" (UID: \"414a2706-7d57-4edb-a3fd-8946b9d07729\") "
Nov 01 03:33:35.119842 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:35.119826    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/414a2706-7d57-4edb-a3fd-8946b9d07729-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/414a2706-7d57-4edb-a3fd-8946b9d07729-maas-api-openshift-ca\") pod \"414a2706-7d57-4edb-a3fd-8946b9d07729\" (UID: \"414a2706-7d57-4edb-a3fd-8946b9d07729\") "
Nov 01 03:33:35.121156 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:35.121133    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/414a2706-7d57-4edb-a3fd-8946b9d07729-maas-api-openshift-ca" pod "414a2706-7d57-4edb-a3fd-8946b9d07729" (UID: "414a2706-7d57-4edb-a3fd-8946b9d07729"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:33:35.124480 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:35.124457    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/414a2706-7d57-4edb-a3fd-8946b9d07729-maas-api-xks-ca" pod "414a2706-7d57-4edb-a3fd-8946b9d07729" (UID: "414a2706-7d57-4edb-a3fd-8946b9d07729"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:33:35.130453 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:35.130429    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/414a2706-7d57-4edb-a3fd-8946b9d07729-kube-api-access-bn2w4" pod "414a2706-7d57-4edb-a3fd-8946b9d07729" (UID: "414a2706-7d57-4edb-a3fd-8946b9d07729"). InnerVolumeSpecName "kube-api-access-bn2w4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:33:35.222102 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:35.222066    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/414a2706-7d57-4edb-a3fd-8946b9d07729-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:33:35.222102 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:35.222101    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/414a2706-7d57-4edb-a3fd-8946b9d07729-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:33:35.222311 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:35.222113    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn2w4\" (UniqueName: \"kubernetes.io/projected/414a2706-7d57-4edb-a3fd-8946b9d07729-kube-api-access-bn2w4\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:33:35.436147 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:35.436063    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"
Nov 01 03:33:35.436349 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:35.436270    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:33:35.436619 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:35.436591    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"
Nov 01 03:33:35.437253 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:35.437236    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"
Nov 01 03:33:35.806373 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:35.806278    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3513c2f1-dpfxx"
Nov 01 03:33:35.806825 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:35.806794    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3513c2f1-dpfxx" event={"ID":"414a2706-7d57-4edb-a3fd-8946b9d07729","Type":"ContainerDied","Data":"061db564f5b89b7ab5d4afdd30a504631d5cf06f5885bc8b98e0c6227fb5aa5d"}
Nov 01 03:33:35.807033 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:35.807007    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="061db564f5b89b7ab5d4afdd30a504631d5cf06f5885bc8b98e0c6227fb5aa5d"
Nov 01 03:33:36.051232 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:36.051196    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10863eb9-6zd52"]
Nov 01 03:33:36.055855 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:36.055830    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10863eb9-6zd52"]
Nov 01 03:33:36.681897 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:36.681865    4930 scope.go:122] "RemoveContainer" containerID="c5f1d6e5d57fe27cb0a8482c509f75f8ca456ca0151a7cb7c7bae76ceae32410"
Nov 01 03:33:36.693267 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:36.693244    4930 scope.go:122] "RemoveContainer" containerID="0f0c31c06f4d1873ee688f09fc18beba7597acbc871f93994877c34a26db583f"
Nov 01 03:33:36.704089 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:36.704053    4930 scope.go:122] "RemoveContainer" containerID="4bf3ff6d81c4497ca3407757663dc755565167c97960239edbddc30e2117e273"
Nov 01 03:33:36.714709 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:36.714686    4930 scope.go:122] "RemoveContainer" containerID="cb8d2b46ccff7c7c1f632d505b787f705a90af3e39fd92caa73385a99ab16ff3"
Nov 01 03:33:36.724596 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:36.724577    4930 scope.go:122] "RemoveContainer" containerID="53aee18bd15c501c67142113f3a365221036327971ceb60cdb3d4a3df51a1574"
Nov 01 03:33:36.736392 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:36.736374    4930 scope.go:122] "RemoveContainer" containerID="68db2c2c473cc995b6d6dee1d2994a64b642a01311a6daf14e0679a803c9baa4"
Nov 01 03:33:36.745897 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:36.745880    4930 scope.go:122] "RemoveContainer" containerID="587ff8897f2cea2f537c74d13a8538eac6f033e9c70ee5f1428ffcd5577b0afd"
Nov 01 03:33:37.580340 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:37.580307    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:33:37.623214 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:37.623178    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3934c25e-4360-44ca-a355-abb425ebed69" path="/var/lib/kubelet/pods/3934c25e-4360-44ca-a355-abb425ebed69/volumes"
Nov 01 03:33:37.767611 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:37.767522    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:33:38.954579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:38.949340    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:33:39.101288 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:39.101248    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:33:39.684511 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:39.684466    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:33:42.746024 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:42.745985    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:33:42.986019 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:42.985987    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:33:43.832460 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:43.832428    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:33:44.933113 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:44.933074    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:33:45.051008 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:45.050971    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a07782b1-wjp8r"]
Nov 01 03:33:45.101250 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:45.101207    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-367a7f78-w4t5v"]
Nov 01 03:33:45.105646 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:45.105623    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a07782b1-wjp8r"]
Nov 01 03:33:45.109436 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:45.109406    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-367a7f78-w4t5v"]
Nov 01 03:33:45.621993 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:45.621961    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="540c82b7-8782-49b8-86a4-576e10e21357" path="/var/lib/kubelet/pods/540c82b7-8782-49b8-86a4-576e10e21357/volumes"
Nov 01 03:33:45.622509 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:45.622492    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db76ac41-529c-4927-b1fc-b029877dd3fc" path="/var/lib/kubelet/pods/db76ac41-529c-4927-b1fc-b029877dd3fc/volumes"
Nov 01 03:33:47.607158 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:47.607104    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:33:48.138425 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:48.138393    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"]
Nov 01 03:33:48.139109 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:48.139092    4930 cpu_manager.go:414] "containerMap: removing container" podUID="414a2706-7d57-4edb-a3fd-8946b9d07729" containerName="revoke-keys"
Nov 01 03:33:48.139175 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:48.139112    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="414a2706-7d57-4edb-a3fd-8946b9d07729" containerName="revoke-keys"
Nov 01 03:33:48.139248 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:48.139238    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="414a2706-7d57-4edb-a3fd-8946b9d07729" containerName="revoke-keys"
Nov 01 03:33:48.143228 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:48.143207    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"
Nov 01 03:33:48.197972 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:48.195893    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-fdbdfefd-openshift-default-dockercfg-5lrhl\""
Nov 01 03:33:48.219676 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:48.219648    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-workload-socket\") pod \"e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9\" (UID: \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\") " pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"
Nov 01 03:33:48.219940 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:48.219919    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-istio-token\") pod \"e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9\" (UID: \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\") " pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"
Nov 01 03:33:48.220060 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:48.220045    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-istio-envoy\") pod \"e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9\" (UID: \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\") " pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"
Nov 01 03:33:48.220152 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:48.220072    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-istio-podinfo\") pod \"e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9\" (UID: \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\") " pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"
Nov 01 03:33:48.220152 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:48.220103    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-istio-data\") pod \"e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9\" (UID: \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\") " pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"
Nov 01 03:33:48.220152 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:48.220145    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-credential-socket\") pod \"e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9\" (UID: \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\") " pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"
Nov 01 03:33:48.220357 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:48.220211    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whqx5\" (UniqueName: \"kubernetes.io/projected/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-kube-api-access-whqx5\") pod \"e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9\" (UID: \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\") " pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"
Nov 01 03:33:48.220357 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:48.220312    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-workload-certs\") pod \"e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9\" (UID: \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\") " pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"
Nov 01 03:33:48.220461 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:48.220433    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-istiod-ca-cert\") pod \"e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9\" (UID: \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\") " pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"
Nov 01 03:33:48.233373 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:48.233340    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:33:48.253946 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:48.253908    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"]
Nov 01 03:33:48.321816 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:48.321782    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-istio-token\") pod \"e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9\" (UID: \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\") " pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"
Nov 01 03:33:48.322050 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:48.322019    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-istio-envoy\") pod \"e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9\" (UID: \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\") " pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"
Nov 01 03:33:48.322142 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:48.322055    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-istio-podinfo\") pod \"e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9\" (UID: \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\") " pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"
Nov 01 03:33:48.322142 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:48.322083    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-istio-data\") pod \"e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9\" (UID: \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\") " pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"
Nov 01 03:33:48.322142 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:48.322122    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-credential-socket\") pod \"e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9\" (UID: \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\") " pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"
Nov 01 03:33:48.322309 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:48.322188    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whqx5\" (UniqueName: \"kubernetes.io/projected/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-kube-api-access-whqx5\") pod \"e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9\" (UID: \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\") " pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"
Nov 01 03:33:48.322309 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:48.322289    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-workload-certs\") pod \"e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9\" (UID: \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\") " pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"
Nov 01 03:33:48.322425 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:48.322405    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-istiod-ca-cert\") pod \"e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9\" (UID: \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\") " pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"
Nov 01 03:33:48.322617 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:48.322595    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-workload-socket\") pod \"e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9\" (UID: \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\") " pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"
Nov 01 03:33:48.323074 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:48.323051    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-workload-socket\") pod \"e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9\" (UID: \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\") " pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"
Nov 01 03:33:48.323945 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:48.323921    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-credential-socket\") pod \"e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9\" (UID: \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\") " pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"
Nov 01 03:33:48.324446 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:48.324421    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-istio-data\") pod \"e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9\" (UID: \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\") " pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"
Nov 01 03:33:48.324689 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:48.324659    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-istiod-ca-cert\") pod \"e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9\" (UID: \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\") " pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"
Nov 01 03:33:48.324783 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:48.324757    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-workload-certs\") pod \"e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9\" (UID: \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\") " pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"
Nov 01 03:33:48.327940 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:48.327914    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-istio-envoy\") pod \"e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9\" (UID: \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\") " pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"
Nov 01 03:33:48.330066 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:48.330043    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-istio-podinfo\") pod \"e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9\" (UID: \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\") " pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"
Nov 01 03:33:48.429896 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:48.429821    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whqx5\" (UniqueName: \"kubernetes.io/projected/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-kube-api-access-whqx5\") pod \"e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9\" (UID: \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\") " pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"
Nov 01 03:33:48.456399 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:48.456371    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-istio-token\") pod \"e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9\" (UID: \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\") " pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"
Nov 01 03:33:48.460793 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:48.460762    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:33:48.461711 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:48.461690    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"
Nov 01 03:33:48.793939 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:48.793836    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:33:49.057137 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:49.057058    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27c1b5ea-qtg9h"]
Nov 01 03:33:49.063254 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:49.063220    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27c1b5ea-qtg9h"]
Nov 01 03:33:49.105204 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:49.105173    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"]
Nov 01 03:33:49.108127 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:33:49.108088    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb1aaf35_8c75_4c36_a2cd_193d7cf4fa20.slice/crio-f34512be1c8ad3a24bd9138caa0316ea624938b45beb62852c863811efe1873e WatchSource:0}: Error finding container f34512be1c8ad3a24bd9138caa0316ea624938b45beb62852c863811efe1873e: Status 404 returned error can't find the container with id f34512be1c8ad3a24bd9138caa0316ea624938b45beb62852c863811efe1873e
Nov 01 03:33:49.110716 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:49.110679    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:33:49.110843 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:49.110773    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:33:49.110843 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:49.110821    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:33:49.134961 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:33:49.134927    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:33:49.163171 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:49.163138    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9" event={"ID":"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20","Type":"ContainerStarted","Data":"f34512be1c8ad3a24bd9138caa0316ea624938b45beb62852c863811efe1873e"}
Nov 01 03:33:49.623296 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:49.623263    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="864d195a-d8a1-41e3-bcdf-fd2acefa7f8b" path="/var/lib/kubelet/pods/864d195a-d8a1-41e3-bcdf-fd2acefa7f8b/volumes"
Nov 01 03:33:50.082950 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:50.082870    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d17bc02d-82rld"]
Nov 01 03:33:50.130811 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:50.130775    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d17bc02d-82rld"]
Nov 01 03:33:50.176764 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:50.176726    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9" event={"ID":"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20","Type":"ContainerStarted","Data":"c2d5829d14a12ad87679cd55ed38ee14bcc992364fd70ad5c1245017a0bd6315"}
Nov 01 03:33:50.306836 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:50.306769    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9" podStartSLOduration=2.306750472 podStartE2EDuration="2.306750472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:33:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:33:50.30380045 +0000 UTC m=+1634.826807075" watchObservedRunningTime="2026-11-01 03:33:50.306750472 +0000 UTC m=+1634.829757061"
Nov 01 03:33:50.471615 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:50.469822    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"
Nov 01 03:33:50.493162 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:50.493125    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"
Nov 01 03:33:51.188366 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:51.188326    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"
Nov 01 03:33:51.189490 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:51.189466    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"
Nov 01 03:33:51.623124 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:51.623093    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac769fae-f7ba-48c1-b254-594c1ac5899e" path="/var/lib/kubelet/pods/ac769fae-f7ba-48c1-b254-594c1ac5899e/volumes"
Nov 01 03:33:53.113878 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:53.113847    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:33:53.360866 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:53.360825    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:33:53.645075 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:53.645034    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:33:55.050880 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:55.050839    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85680f45-cbjpd"]
Nov 01 03:33:55.100453 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:55.100416    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2408401a-m5s4p"]
Nov 01 03:33:55.105018 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:55.104992    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85680f45-cbjpd"]
Nov 01 03:33:55.108719 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:55.108695    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2408401a-m5s4p"]
Nov 01 03:33:55.328812 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:55.328718    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:33:55.624269 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:55.624237    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d8e03bf-33e5-4142-9bfe-723785dce2a0" path="/var/lib/kubelet/pods/3d8e03bf-33e5-4142-9bfe-723785dce2a0/volumes"
Nov 01 03:33:55.624752 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:55.624736    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46b23170-f01f-4af1-b13b-fa55f359c43e" path="/var/lib/kubelet/pods/46b23170-f01f-4af1-b13b-fa55f359c43e/volumes"
Nov 01 03:33:55.959067 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:55.958963    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:33:56.243868 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:56.243765    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:33:56.404357 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:56.404316    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:33:56.481672 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:56.481633    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-44379c05-vwm69"]
Nov 01 03:33:56.574370 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:56.574277    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:33:56.738782 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:56.738746    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-44379c05-vwm69"]
Nov 01 03:33:56.897621 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:56.897584    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:33:57.622058 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:57.622022    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b68fed2-7cce-4c2b-8476-9205ef4cf5e4" path="/var/lib/kubelet/pods/6b68fed2-7cce-4c2b-8476-9205ef4cf5e4/volumes"
Nov 01 03:33:58.505998 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:58.505959    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:33:59.008165 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:59.008134    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw"]
Nov 01 03:33:59.008683 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:59.008362    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw" podUID="9cd5db87-0172-46ce-b856-cfc6366d7cb2" containerName="istio-proxy" containerID="cri-o://42ae3b9e3c4d81ad83023657e8fe369041adb8c52cff478cdfa0e186410cd13a" gracePeriod=30
Nov 01 03:33:59.056812 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:59.056775    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n"]
Nov 01 03:33:59.061216 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:59.061186    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"]
Nov 01 03:33:59.061421 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:59.061397    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm" podUID="58f8e6d8-aafa-4a82-b1e4-63c358380207" containerName="istio-proxy" containerID="cri-o://a2a769f870a5917c452b00a2196b52e027797797f5ac49010ec4fec40e450dc6" gracePeriod=30
Nov 01 03:33:59.238450 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:59.238415    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q"]
Nov 01 03:33:59.238741 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:59.238712    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q" podUID="12d11ff3-9497-498d-8aeb-7f70f1874383" containerName="istio-proxy" containerID="cri-o://563b9115b7f7e666b6b16ccdff4721b02322eb406b1ed65d7a5fbc7ec0db8615" gracePeriod=30
Nov 01 03:33:59.244075 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:59.244046    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"]
Nov 01 03:33:59.244361 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:59.244264    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j" podUID="f3c4cdb7-e686-4b71-9ab9-3c82f4c24648" containerName="istio-proxy" containerID="cri-o://143d2317b405875bd15fb6fcbcc10bc0a02c0104fb4b8569a986b10ddc70ec71" gracePeriod=30
Nov 01 03:33:59.250608 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:59.250573    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz"]
Nov 01 03:33:59.251038 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:59.251008    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz" podUID="27bad768-f9df-4649-a979-2a459327f07d" containerName="istio-proxy" containerID="cri-o://1b6710396ea1c3a864a6e4972e728fe7c165c385b57dfe70cc5e911fa7144143" gracePeriod=30
Nov 01 03:33:59.294629 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:33:59.288257    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n" podUID="72c2d7cf-fac4-4d80-a5f1-82ae300a255a" containerName="istio-proxy" containerID="cri-o://dbf783d89d9705308d560551d328a4077052c6f3944362a3803f829548325208" gracePeriod=30
Nov 01 03:34:00.614640 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:00.614583    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:00.652230 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:00.652186    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7"]
Nov 01 03:34:00.652449 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:00.652422    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7" podUID="4b528dcb-1501-4764-bf26-d676a6e18474" containerName="istio-proxy" containerID="cri-o://8e3a98e5f62c2e50232c9ac38e464abe3bee0d658efdb9265bd52afb76278b1a" gracePeriod=30
Nov 01 03:34:00.661996 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:00.661964    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk"]
Nov 01 03:34:00.662229 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:00.662202    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk" podUID="fd824f02-6c5a-4428-a995-80cbf1b20571" containerName="istio-proxy" containerID="cri-o://5778c35aad39918487bc447f89f9ff1a99fc3fcf8d7df7b4060f4a6425121b77" gracePeriod=30
Nov 01 03:34:00.715001 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:00.714967    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t"]
Nov 01 03:34:00.715455 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:00.715419    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t" podUID="23ed3df4-075f-459b-be45-69da508ea513" containerName="istio-proxy" containerID="cri-o://b2780b594982117c704a8352f5ac248d50a664e116cbd6c9c3e888b35d2b2082" gracePeriod=30
Nov 01 03:34:00.721650 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:00.721611    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"]
Nov 01 03:34:00.722181 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:00.722125    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr" podUID="267415f5-02bf-4571-914a-45aa36ae41c9" containerName="istio-proxy" containerID="cri-o://e79fab7e2557d937d8010fea46cbac0a838d03f17abb66abf00e348bec67aa5d" gracePeriod=30
Nov 01 03:34:02.970793 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:02.970754    4930 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:34:02.971334 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:02.970837    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw" podUID="9cd5db87-0172-46ce-b856-cfc6366d7cb2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:34:03.108266 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:03.108223    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:03.285688 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:03.282844    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:03.469059 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:03.469021    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:03.483190 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:03.483159    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"]
Nov 01 03:34:03.483377 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:03.483357    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8" podUID="9165d842-245c-4270-a5ab-8789ab0a2e89" containerName="istio-proxy" containerID="cri-o://37ea7590884339c5454463228bc2325ec7d25f2daabc0802ce5d94f1bc4b3d4e" gracePeriod=30
Nov 01 03:34:03.992944 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:03.992906    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:04.193357 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.193304    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:04.220171 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.220133    4930 patch_prober.go:29] interesting pod/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:34:04.220355 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.220198    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n" podUID="72c2d7cf-fac4-4d80-a5f1-82ae300a255a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:34:04.292784 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.292749    4930 patch_prober.go:29] interesting pod/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:34:04.292904 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.292821    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8" podUID="9165d842-245c-4270-a5ab-8789ab0a2e89" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:34:04.351595 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.351374    4930 generic.go:356] "Generic (PLEG): container finished" podID="58f8e6d8-aafa-4a82-b1e4-63c358380207" containerID="a2a769f870a5917c452b00a2196b52e027797797f5ac49010ec4fec40e450dc6" exitCode=0
Nov 01 03:34:04.351595 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.351401    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm" event={"ID":"58f8e6d8-aafa-4a82-b1e4-63c358380207","Type":"ContainerDied","Data":"a2a769f870a5917c452b00a2196b52e027797797f5ac49010ec4fec40e450dc6"}
Nov 01 03:34:04.351595 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.351498    4930 scope.go:122] "RemoveContainer" containerID="aafc35f35a3dde77e5b573be2be98eaa2018cd07ef430dbbfdcc84e2359e07a1"
Nov 01 03:34:04.356639 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.356304    4930 generic.go:356] "Generic (PLEG): container finished" podID="9cd5db87-0172-46ce-b856-cfc6366d7cb2" containerID="42ae3b9e3c4d81ad83023657e8fe369041adb8c52cff478cdfa0e186410cd13a" exitCode=0
Nov 01 03:34:04.356639 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.356524    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw" event={"ID":"9cd5db87-0172-46ce-b856-cfc6366d7cb2","Type":"ContainerDied","Data":"42ae3b9e3c4d81ad83023657e8fe369041adb8c52cff478cdfa0e186410cd13a"}
Nov 01 03:34:04.512748 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.504483    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:04.548514 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.548411    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"
Nov 01 03:34:04.554573 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.553174    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw"
Nov 01 03:34:04.609979 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.609864    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q"
Nov 01 03:34:04.614998 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.614963    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cd5db87-0172-46ce-b856-cfc6366d7cb2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cd5db87-0172-46ce-b856-cfc6366d7cb2-istio-envoy\") pod \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\" (UID: \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\") "
Nov 01 03:34:04.615128 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.615022    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58f8e6d8-aafa-4a82-b1e4-63c358380207-kube-api-access-bghzq\" (UniqueName: \"kubernetes.io/projected/58f8e6d8-aafa-4a82-b1e4-63c358380207-kube-api-access-bghzq\") pod \"58f8e6d8-aafa-4a82-b1e4-63c358380207\" (UID: \"58f8e6d8-aafa-4a82-b1e4-63c358380207\") "
Nov 01 03:34:04.615128 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.615059    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58f8e6d8-aafa-4a82-b1e4-63c358380207-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58f8e6d8-aafa-4a82-b1e4-63c358380207-istio-envoy\") pod \"58f8e6d8-aafa-4a82-b1e4-63c358380207\" (UID: \"58f8e6d8-aafa-4a82-b1e4-63c358380207\") "
Nov 01 03:34:04.615268 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.615143    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58f8e6d8-aafa-4a82-b1e4-63c358380207-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58f8e6d8-aafa-4a82-b1e4-63c358380207-istiod-ca-cert\") pod \"58f8e6d8-aafa-4a82-b1e4-63c358380207\" (UID: \"58f8e6d8-aafa-4a82-b1e4-63c358380207\") "
Nov 01 03:34:04.615268 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.615178    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cd5db87-0172-46ce-b856-cfc6366d7cb2-kube-api-access-dzhdm\" (UniqueName: \"kubernetes.io/projected/9cd5db87-0172-46ce-b856-cfc6366d7cb2-kube-api-access-dzhdm\") pod \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\" (UID: \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\") "
Nov 01 03:34:04.615268 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.615228    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58f8e6d8-aafa-4a82-b1e4-63c358380207-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58f8e6d8-aafa-4a82-b1e4-63c358380207-workload-certs\") pod \"58f8e6d8-aafa-4a82-b1e4-63c358380207\" (UID: \"58f8e6d8-aafa-4a82-b1e4-63c358380207\") "
Nov 01 03:34:04.615443 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.615345    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cd5db87-0172-46ce-b856-cfc6366d7cb2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cd5db87-0172-46ce-b856-cfc6366d7cb2-istiod-ca-cert\") pod \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\" (UID: \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\") "
Nov 01 03:34:04.615443 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.615395    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58f8e6d8-aafa-4a82-b1e4-63c358380207-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58f8e6d8-aafa-4a82-b1e4-63c358380207-istio-data\") pod \"58f8e6d8-aafa-4a82-b1e4-63c358380207\" (UID: \"58f8e6d8-aafa-4a82-b1e4-63c358380207\") "
Nov 01 03:34:04.615595 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.615456    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58f8e6d8-aafa-4a82-b1e4-63c358380207-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58f8e6d8-aafa-4a82-b1e4-63c358380207-istio-podinfo\") pod \"58f8e6d8-aafa-4a82-b1e4-63c358380207\" (UID: \"58f8e6d8-aafa-4a82-b1e4-63c358380207\") "
Nov 01 03:34:04.615595 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.615582    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cd5db87-0172-46ce-b856-cfc6366d7cb2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd5db87-0172-46ce-b856-cfc6366d7cb2-workload-socket\") pod \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\" (UID: \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\") "
Nov 01 03:34:04.615722 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.615673    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cd5db87-0172-46ce-b856-cfc6366d7cb2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cd5db87-0172-46ce-b856-cfc6366d7cb2-istio-data\") pod \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\" (UID: \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\") "
Nov 01 03:34:04.615722 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.615715    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58f8e6d8-aafa-4a82-b1e4-63c358380207-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f8e6d8-aafa-4a82-b1e4-63c358380207-workload-socket\") pod \"58f8e6d8-aafa-4a82-b1e4-63c358380207\" (UID: \"58f8e6d8-aafa-4a82-b1e4-63c358380207\") "
Nov 01 03:34:04.615830 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.615745    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58f8e6d8-aafa-4a82-b1e4-63c358380207-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f8e6d8-aafa-4a82-b1e4-63c358380207-credential-socket\") pod \"58f8e6d8-aafa-4a82-b1e4-63c358380207\" (UID: \"58f8e6d8-aafa-4a82-b1e4-63c358380207\") "
Nov 01 03:34:04.615830 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.615795    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cd5db87-0172-46ce-b856-cfc6366d7cb2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cd5db87-0172-46ce-b856-cfc6366d7cb2-workload-certs\") pod \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\" (UID: \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\") "
Nov 01 03:34:04.615946 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.615858    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58f8e6d8-aafa-4a82-b1e4-63c358380207-istio-token\" (UniqueName: \"kubernetes.io/projected/58f8e6d8-aafa-4a82-b1e4-63c358380207-istio-token\") pod \"58f8e6d8-aafa-4a82-b1e4-63c358380207\" (UID: \"58f8e6d8-aafa-4a82-b1e4-63c358380207\") "
Nov 01 03:34:04.615946 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.615911    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cd5db87-0172-46ce-b856-cfc6366d7cb2-istio-token\" (UniqueName: \"kubernetes.io/projected/9cd5db87-0172-46ce-b856-cfc6366d7cb2-istio-token\") pod \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\" (UID: \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\") "
Nov 01 03:34:04.616063 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.616039    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cd5db87-0172-46ce-b856-cfc6366d7cb2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd5db87-0172-46ce-b856-cfc6366d7cb2-credential-socket\") pod \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\" (UID: \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\") "
Nov 01 03:34:04.616127 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.616084    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9cd5db87-0172-46ce-b856-cfc6366d7cb2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cd5db87-0172-46ce-b856-cfc6366d7cb2-istio-podinfo\") pod \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\" (UID: \"9cd5db87-0172-46ce-b856-cfc6366d7cb2\") "
Nov 01 03:34:04.616984 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.616458    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58f8e6d8-aafa-4a82-b1e4-63c358380207-istiod-ca-cert" pod "58f8e6d8-aafa-4a82-b1e4-63c358380207" (UID: "58f8e6d8-aafa-4a82-b1e4-63c358380207"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:34:04.616984 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.616895    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58f8e6d8-aafa-4a82-b1e4-63c358380207-workload-socket" pod "58f8e6d8-aafa-4a82-b1e4-63c358380207" (UID: "58f8e6d8-aafa-4a82-b1e4-63c358380207"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:04.617959 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.617444    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58f8e6d8-aafa-4a82-b1e4-63c358380207-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.617959 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.617472    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f8e6d8-aafa-4a82-b1e4-63c358380207-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.617959 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.617823    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58f8e6d8-aafa-4a82-b1e4-63c358380207-credential-socket" pod "58f8e6d8-aafa-4a82-b1e4-63c358380207" (UID: "58f8e6d8-aafa-4a82-b1e4-63c358380207"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:04.618369 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.618348    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58f8e6d8-aafa-4a82-b1e4-63c358380207-workload-certs" pod "58f8e6d8-aafa-4a82-b1e4-63c358380207" (UID: "58f8e6d8-aafa-4a82-b1e4-63c358380207"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:04.618515 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.618498    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd5db87-0172-46ce-b856-cfc6366d7cb2-workload-certs" pod "9cd5db87-0172-46ce-b856-cfc6366d7cb2" (UID: "9cd5db87-0172-46ce-b856-cfc6366d7cb2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:04.619341 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.618864    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cd5db87-0172-46ce-b856-cfc6366d7cb2-istiod-ca-cert" pod "9cd5db87-0172-46ce-b856-cfc6366d7cb2" (UID: "9cd5db87-0172-46ce-b856-cfc6366d7cb2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:34:04.619341 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.619274    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd5db87-0172-46ce-b856-cfc6366d7cb2-istio-data" pod "9cd5db87-0172-46ce-b856-cfc6366d7cb2" (UID: "9cd5db87-0172-46ce-b856-cfc6366d7cb2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:04.619533 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.619478    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58f8e6d8-aafa-4a82-b1e4-63c358380207-istio-data" pod "58f8e6d8-aafa-4a82-b1e4-63c358380207" (UID: "58f8e6d8-aafa-4a82-b1e4-63c358380207"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:04.619533 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.619519    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd5db87-0172-46ce-b856-cfc6366d7cb2-credential-socket" pod "9cd5db87-0172-46ce-b856-cfc6366d7cb2" (UID: "9cd5db87-0172-46ce-b856-cfc6366d7cb2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:04.619754 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.619729    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd5db87-0172-46ce-b856-cfc6366d7cb2-workload-socket" pod "9cd5db87-0172-46ce-b856-cfc6366d7cb2" (UID: "9cd5db87-0172-46ce-b856-cfc6366d7cb2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:04.621591 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.621533    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58f8e6d8-aafa-4a82-b1e4-63c358380207-istio-podinfo" pod "58f8e6d8-aafa-4a82-b1e4-63c358380207" (UID: "58f8e6d8-aafa-4a82-b1e4-63c358380207"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:34:04.624819 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.624791    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd5db87-0172-46ce-b856-cfc6366d7cb2-istio-token" pod "9cd5db87-0172-46ce-b856-cfc6366d7cb2" (UID: "9cd5db87-0172-46ce-b856-cfc6366d7cb2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:34:04.631017 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.630979    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58f8e6d8-aafa-4a82-b1e4-63c358380207-kube-api-access-bghzq" pod "58f8e6d8-aafa-4a82-b1e4-63c358380207" (UID: "58f8e6d8-aafa-4a82-b1e4-63c358380207"). InnerVolumeSpecName "kube-api-access-bghzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:34:04.631208 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.631181    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd5db87-0172-46ce-b856-cfc6366d7cb2-kube-api-access-dzhdm" pod "9cd5db87-0172-46ce-b856-cfc6366d7cb2" (UID: "9cd5db87-0172-46ce-b856-cfc6366d7cb2"). InnerVolumeSpecName "kube-api-access-dzhdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:34:04.631352 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.631329    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd5db87-0172-46ce-b856-cfc6366d7cb2-istio-envoy" pod "9cd5db87-0172-46ce-b856-cfc6366d7cb2" (UID: "9cd5db87-0172-46ce-b856-cfc6366d7cb2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:04.631520 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.631497    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58f8e6d8-aafa-4a82-b1e4-63c358380207-istio-envoy" pod "58f8e6d8-aafa-4a82-b1e4-63c358380207" (UID: "58f8e6d8-aafa-4a82-b1e4-63c358380207"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:04.634142 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.634110    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58f8e6d8-aafa-4a82-b1e4-63c358380207-istio-token" pod "58f8e6d8-aafa-4a82-b1e4-63c358380207" (UID: "58f8e6d8-aafa-4a82-b1e4-63c358380207"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:34:04.639932 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.639906    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9cd5db87-0172-46ce-b856-cfc6366d7cb2-istio-podinfo" pod "9cd5db87-0172-46ce-b856-cfc6366d7cb2" (UID: "9cd5db87-0172-46ce-b856-cfc6366d7cb2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:34:04.704154 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.704126    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"
Nov 01 03:34:04.712069 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.712042    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz"
Nov 01 03:34:04.718056 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.718027    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n"
Nov 01 03:34:04.718642 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.718615    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12d11ff3-9497-498d-8aeb-7f70f1874383-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12d11ff3-9497-498d-8aeb-7f70f1874383-istiod-ca-cert\") pod \"12d11ff3-9497-498d-8aeb-7f70f1874383\" (UID: \"12d11ff3-9497-498d-8aeb-7f70f1874383\") "
Nov 01 03:34:04.718759 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.718686    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d11ff3-9497-498d-8aeb-7f70f1874383-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12d11ff3-9497-498d-8aeb-7f70f1874383-istio-envoy\") pod \"12d11ff3-9497-498d-8aeb-7f70f1874383\" (UID: \"12d11ff3-9497-498d-8aeb-7f70f1874383\") "
Nov 01 03:34:04.718818 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.718754    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12d11ff3-9497-498d-8aeb-7f70f1874383-kube-api-access-2ff8v\" (UniqueName: \"kubernetes.io/projected/12d11ff3-9497-498d-8aeb-7f70f1874383-kube-api-access-2ff8v\") pod \"12d11ff3-9497-498d-8aeb-7f70f1874383\" (UID: \"12d11ff3-9497-498d-8aeb-7f70f1874383\") "
Nov 01 03:34:04.721618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.718883    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d11ff3-9497-498d-8aeb-7f70f1874383-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12d11ff3-9497-498d-8aeb-7f70f1874383-istio-data\") pod \"12d11ff3-9497-498d-8aeb-7f70f1874383\" (UID: \"12d11ff3-9497-498d-8aeb-7f70f1874383\") "
Nov 01 03:34:04.721618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.718985    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12d11ff3-9497-498d-8aeb-7f70f1874383-istio-token\" (UniqueName: \"kubernetes.io/projected/12d11ff3-9497-498d-8aeb-7f70f1874383-istio-token\") pod \"12d11ff3-9497-498d-8aeb-7f70f1874383\" (UID: \"12d11ff3-9497-498d-8aeb-7f70f1874383\") "
Nov 01 03:34:04.721618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.719045    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d11ff3-9497-498d-8aeb-7f70f1874383-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d11ff3-9497-498d-8aeb-7f70f1874383-workload-certs\") pod \"12d11ff3-9497-498d-8aeb-7f70f1874383\" (UID: \"12d11ff3-9497-498d-8aeb-7f70f1874383\") "
Nov 01 03:34:04.721618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.719061    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12d11ff3-9497-498d-8aeb-7f70f1874383-istiod-ca-cert" pod "12d11ff3-9497-498d-8aeb-7f70f1874383" (UID: "12d11ff3-9497-498d-8aeb-7f70f1874383"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:34:04.721618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.719079    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12d11ff3-9497-498d-8aeb-7f70f1874383-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12d11ff3-9497-498d-8aeb-7f70f1874383-istio-podinfo\") pod \"12d11ff3-9497-498d-8aeb-7f70f1874383\" (UID: \"12d11ff3-9497-498d-8aeb-7f70f1874383\") "
Nov 01 03:34:04.721618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.719097    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d11ff3-9497-498d-8aeb-7f70f1874383-istio-data" pod "12d11ff3-9497-498d-8aeb-7f70f1874383" (UID: "12d11ff3-9497-498d-8aeb-7f70f1874383"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:04.721618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.719135    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d11ff3-9497-498d-8aeb-7f70f1874383-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d11ff3-9497-498d-8aeb-7f70f1874383-credential-socket\") pod \"12d11ff3-9497-498d-8aeb-7f70f1874383\" (UID: \"12d11ff3-9497-498d-8aeb-7f70f1874383\") "
Nov 01 03:34:04.721618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.719185    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12d11ff3-9497-498d-8aeb-7f70f1874383-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d11ff3-9497-498d-8aeb-7f70f1874383-workload-socket\") pod \"12d11ff3-9497-498d-8aeb-7f70f1874383\" (UID: \"12d11ff3-9497-498d-8aeb-7f70f1874383\") "
Nov 01 03:34:04.721618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.719298    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d11ff3-9497-498d-8aeb-7f70f1874383-workload-certs" pod "12d11ff3-9497-498d-8aeb-7f70f1874383" (UID: "12d11ff3-9497-498d-8aeb-7f70f1874383"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:04.721618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.719489    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d11ff3-9497-498d-8aeb-7f70f1874383-credential-socket" pod "12d11ff3-9497-498d-8aeb-7f70f1874383" (UID: "12d11ff3-9497-498d-8aeb-7f70f1874383"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:04.721618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.719759    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d11ff3-9497-498d-8aeb-7f70f1874383-workload-socket" pod "12d11ff3-9497-498d-8aeb-7f70f1874383" (UID: "12d11ff3-9497-498d-8aeb-7f70f1874383"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:04.721618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.720851    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58f8e6d8-aafa-4a82-b1e4-63c358380207-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.721618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.720873    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12d11ff3-9497-498d-8aeb-7f70f1874383-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.721618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.720887    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzhdm\" (UniqueName: \"kubernetes.io/projected/9cd5db87-0172-46ce-b856-cfc6366d7cb2-kube-api-access-dzhdm\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.721618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.720901    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58f8e6d8-aafa-4a82-b1e4-63c358380207-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.721618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.720914    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12d11ff3-9497-498d-8aeb-7f70f1874383-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.721618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.720929    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d11ff3-9497-498d-8aeb-7f70f1874383-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.721618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.720943    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12d11ff3-9497-498d-8aeb-7f70f1874383-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.721618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.720957    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cd5db87-0172-46ce-b856-cfc6366d7cb2-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.721618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.720971    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58f8e6d8-aafa-4a82-b1e4-63c358380207-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.721618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.720984    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58f8e6d8-aafa-4a82-b1e4-63c358380207-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.721618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.720998    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd5db87-0172-46ce-b856-cfc6366d7cb2-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.721618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.721013    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cd5db87-0172-46ce-b856-cfc6366d7cb2-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.721618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.721027    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58f8e6d8-aafa-4a82-b1e4-63c358380207-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.721618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.721040    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cd5db87-0172-46ce-b856-cfc6366d7cb2-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.721618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.721053    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12d11ff3-9497-498d-8aeb-7f70f1874383-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.721618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.721068    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58f8e6d8-aafa-4a82-b1e4-63c358380207-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.721618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.721083    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cd5db87-0172-46ce-b856-cfc6366d7cb2-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.721618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.721097    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd5db87-0172-46ce-b856-cfc6366d7cb2-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.721618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.721113    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cd5db87-0172-46ce-b856-cfc6366d7cb2-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.721618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.721126    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cd5db87-0172-46ce-b856-cfc6366d7cb2-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.721618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.721140    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bghzq\" (UniqueName: \"kubernetes.io/projected/58f8e6d8-aafa-4a82-b1e4-63c358380207-kube-api-access-bghzq\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.725322 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.725291    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12d11ff3-9497-498d-8aeb-7f70f1874383-istio-envoy" pod "12d11ff3-9497-498d-8aeb-7f70f1874383" (UID: "12d11ff3-9497-498d-8aeb-7f70f1874383"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:04.725460 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.725328    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12d11ff3-9497-498d-8aeb-7f70f1874383-istio-token" pod "12d11ff3-9497-498d-8aeb-7f70f1874383" (UID: "12d11ff3-9497-498d-8aeb-7f70f1874383"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:34:04.726377 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.726351    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12d11ff3-9497-498d-8aeb-7f70f1874383-kube-api-access-2ff8v" pod "12d11ff3-9497-498d-8aeb-7f70f1874383" (UID: "12d11ff3-9497-498d-8aeb-7f70f1874383"). InnerVolumeSpecName "kube-api-access-2ff8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:34:04.726814 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.726789    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12d11ff3-9497-498d-8aeb-7f70f1874383-istio-podinfo" pod "12d11ff3-9497-498d-8aeb-7f70f1874383" (UID: "12d11ff3-9497-498d-8aeb-7f70f1874383"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:34:04.822119 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.822029    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-workload-socket\") pod \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\" (UID: \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\") "
Nov 01 03:34:04.822119 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.822068    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27bad768-f9df-4649-a979-2a459327f07d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27bad768-f9df-4649-a979-2a459327f07d-workload-socket\") pod \"27bad768-f9df-4649-a979-2a459327f07d\" (UID: \"27bad768-f9df-4649-a979-2a459327f07d\") "
Nov 01 03:34:04.822119 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.822093    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-credential-socket\") pod \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\" (UID: \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\") "
Nov 01 03:34:04.822414 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.822168    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-workload-certs\") pod \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\" (UID: \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\") "
Nov 01 03:34:04.822414 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.822204    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-workload-socket\") pod \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\" (UID: \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\") "
Nov 01 03:34:04.822414 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.822254    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-istio-data\") pod \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\" (UID: \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\") "
Nov 01 03:34:04.822414 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.822318    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-istio-data\") pod \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\" (UID: \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\") "
Nov 01 03:34:04.822414 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.822336    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-workload-socket" pod "72c2d7cf-fac4-4d80-a5f1-82ae300a255a" (UID: "72c2d7cf-fac4-4d80-a5f1-82ae300a255a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:04.822414 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.822349    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27bad768-f9df-4649-a979-2a459327f07d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27bad768-f9df-4649-a979-2a459327f07d-istio-podinfo\") pod \"27bad768-f9df-4649-a979-2a459327f07d\" (UID: \"27bad768-f9df-4649-a979-2a459327f07d\") "
Nov 01 03:34:04.822783 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.822343    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27bad768-f9df-4649-a979-2a459327f07d-workload-socket" pod "27bad768-f9df-4649-a979-2a459327f07d" (UID: "27bad768-f9df-4649-a979-2a459327f07d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:04.822783 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.822446    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27bad768-f9df-4649-a979-2a459327f07d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27bad768-f9df-4649-a979-2a459327f07d-workload-certs\") pod \"27bad768-f9df-4649-a979-2a459327f07d\" (UID: \"27bad768-f9df-4649-a979-2a459327f07d\") "
Nov 01 03:34:04.822783 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.822458    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-credential-socket" pod "72c2d7cf-fac4-4d80-a5f1-82ae300a255a" (UID: "72c2d7cf-fac4-4d80-a5f1-82ae300a255a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:04.822783 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.822501    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27bad768-f9df-4649-a979-2a459327f07d-kube-api-access-m7wxq\" (UniqueName: \"kubernetes.io/projected/27bad768-f9df-4649-a979-2a459327f07d-kube-api-access-m7wxq\") pod \"27bad768-f9df-4649-a979-2a459327f07d\" (UID: \"27bad768-f9df-4649-a979-2a459327f07d\") "
Nov 01 03:34:04.822783 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.822523    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-workload-certs\") pod \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\" (UID: \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\") "
Nov 01 03:34:04.822783 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.822543    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-kube-api-access-5kd2w\" (UniqueName: \"kubernetes.io/projected/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-kube-api-access-5kd2w\") pod \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\" (UID: \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\") "
Nov 01 03:34:04.822783 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.822542    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-istio-data" pod "72c2d7cf-fac4-4d80-a5f1-82ae300a255a" (UID: "72c2d7cf-fac4-4d80-a5f1-82ae300a255a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:04.822783 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.822629    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-istio-podinfo\") pod \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\" (UID: \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\") "
Nov 01 03:34:04.822783 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.822658    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27bad768-f9df-4649-a979-2a459327f07d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27bad768-f9df-4649-a979-2a459327f07d-istio-envoy\") pod \"27bad768-f9df-4649-a979-2a459327f07d\" (UID: \"27bad768-f9df-4649-a979-2a459327f07d\") "
Nov 01 03:34:04.822783 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.822686    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-kube-api-access-mzsp4\" (UniqueName: \"kubernetes.io/projected/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-kube-api-access-mzsp4\") pod \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\" (UID: \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\") "
Nov 01 03:34:04.822783 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.822706    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27bad768-f9df-4649-a979-2a459327f07d-workload-certs" pod "27bad768-f9df-4649-a979-2a459327f07d" (UID: "27bad768-f9df-4649-a979-2a459327f07d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:04.822783 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.822718    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-istiod-ca-cert\") pod \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\" (UID: \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\") "
Nov 01 03:34:04.824806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.822793    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-istiod-ca-cert\") pod \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\" (UID: \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\") "
Nov 01 03:34:04.824806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.822804    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-workload-certs" pod "72c2d7cf-fac4-4d80-a5f1-82ae300a255a" (UID: "72c2d7cf-fac4-4d80-a5f1-82ae300a255a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:04.824806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.822828    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-istio-token\" (UniqueName: \"kubernetes.io/projected/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-istio-token\") pod \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\" (UID: \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\") "
Nov 01 03:34:04.824806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.822917    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27bad768-f9df-4649-a979-2a459327f07d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27bad768-f9df-4649-a979-2a459327f07d-credential-socket\") pod \"27bad768-f9df-4649-a979-2a459327f07d\" (UID: \"27bad768-f9df-4649-a979-2a459327f07d\") "
Nov 01 03:34:04.824806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.822955    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-credential-socket\") pod \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\" (UID: \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\") "
Nov 01 03:34:04.824806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.822995    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-istio-podinfo\") pod \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\" (UID: \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\") "
Nov 01 03:34:04.824806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.823030    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-istiod-ca-cert" pod "72c2d7cf-fac4-4d80-a5f1-82ae300a255a" (UID: "72c2d7cf-fac4-4d80-a5f1-82ae300a255a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:34:04.824806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.823057    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27bad768-f9df-4649-a979-2a459327f07d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27bad768-f9df-4649-a979-2a459327f07d-istio-data\") pod \"27bad768-f9df-4649-a979-2a459327f07d\" (UID: \"27bad768-f9df-4649-a979-2a459327f07d\") "
Nov 01 03:34:04.824806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.823090    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27bad768-f9df-4649-a979-2a459327f07d-istio-token\" (UniqueName: \"kubernetes.io/projected/27bad768-f9df-4649-a979-2a459327f07d-istio-token\") pod \"27bad768-f9df-4649-a979-2a459327f07d\" (UID: \"27bad768-f9df-4649-a979-2a459327f07d\") "
Nov 01 03:34:04.824806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.823126    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-istio-envoy\") pod \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\" (UID: \"72c2d7cf-fac4-4d80-a5f1-82ae300a255a\") "
Nov 01 03:34:04.824806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.823232    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27bad768-f9df-4649-a979-2a459327f07d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27bad768-f9df-4649-a979-2a459327f07d-istiod-ca-cert\") pod \"27bad768-f9df-4649-a979-2a459327f07d\" (UID: \"27bad768-f9df-4649-a979-2a459327f07d\") "
Nov 01 03:34:04.824806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.823308    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-istio-token\" (UniqueName: \"kubernetes.io/projected/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-istio-token\") pod \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\" (UID: \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\") "
Nov 01 03:34:04.824806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.823372    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-workload-socket" pod "f3c4cdb7-e686-4b71-9ab9-3c82f4c24648" (UID: "f3c4cdb7-e686-4b71-9ab9-3c82f4c24648"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:04.824806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.823376    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-istio-envoy\") pod \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\" (UID: \"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648\") "
Nov 01 03:34:04.824806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.823468    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-workload-certs" pod "f3c4cdb7-e686-4b71-9ab9-3c82f4c24648" (UID: "f3c4cdb7-e686-4b71-9ab9-3c82f4c24648"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:04.824806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.823743    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-istiod-ca-cert" pod "f3c4cdb7-e686-4b71-9ab9-3c82f4c24648" (UID: "f3c4cdb7-e686-4b71-9ab9-3c82f4c24648"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:34:04.824806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.823947    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27bad768-f9df-4649-a979-2a459327f07d-istio-data" pod "27bad768-f9df-4649-a979-2a459327f07d" (UID: "27bad768-f9df-4649-a979-2a459327f07d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:04.824806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.824132    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27bad768-f9df-4649-a979-2a459327f07d-credential-socket" pod "27bad768-f9df-4649-a979-2a459327f07d" (UID: "27bad768-f9df-4649-a979-2a459327f07d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:04.824806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.824430    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27bad768-f9df-4649-a979-2a459327f07d-istiod-ca-cert" pod "27bad768-f9df-4649-a979-2a459327f07d" (UID: "27bad768-f9df-4649-a979-2a459327f07d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:34:04.824806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.824600    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27bad768-f9df-4649-a979-2a459327f07d-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.824806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.824628    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.824806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.824645    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.824806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.824665    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12d11ff3-9497-498d-8aeb-7f70f1874383-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.824806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.824680    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.824806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.824693    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ff8v\" (UniqueName: \"kubernetes.io/projected/12d11ff3-9497-498d-8aeb-7f70f1874383-kube-api-access-2ff8v\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.824806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.824715    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27bad768-f9df-4649-a979-2a459327f07d-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.824806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.824730    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27bad768-f9df-4649-a979-2a459327f07d-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.824806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.824748    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12d11ff3-9497-498d-8aeb-7f70f1874383-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.824806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.824764    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12d11ff3-9497-498d-8aeb-7f70f1874383-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.824806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.824780    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.824806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.824796    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27bad768-f9df-4649-a979-2a459327f07d-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.824806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.824812    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.824806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.824828    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.826826 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.824844    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.826826 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.824861    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.826826 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.825369    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-credential-socket" pod "f3c4cdb7-e686-4b71-9ab9-3c82f4c24648" (UID: "f3c4cdb7-e686-4b71-9ab9-3c82f4c24648"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:04.826826 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.825431    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-istio-data" pod "f3c4cdb7-e686-4b71-9ab9-3c82f4c24648" (UID: "f3c4cdb7-e686-4b71-9ab9-3c82f4c24648"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:04.826826 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.826791    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-istio-envoy" pod "f3c4cdb7-e686-4b71-9ab9-3c82f4c24648" (UID: "f3c4cdb7-e686-4b71-9ab9-3c82f4c24648"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:04.827300 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.827277    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-istio-envoy" pod "72c2d7cf-fac4-4d80-a5f1-82ae300a255a" (UID: "72c2d7cf-fac4-4d80-a5f1-82ae300a255a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:04.827386 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.827308    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-kube-api-access-5kd2w" pod "72c2d7cf-fac4-4d80-a5f1-82ae300a255a" (UID: "72c2d7cf-fac4-4d80-a5f1-82ae300a255a"). InnerVolumeSpecName "kube-api-access-5kd2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:34:04.827453 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.827422    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27bad768-f9df-4649-a979-2a459327f07d-istio-envoy" pod "27bad768-f9df-4649-a979-2a459327f07d" (UID: "27bad768-f9df-4649-a979-2a459327f07d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:04.827453 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.827435    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27bad768-f9df-4649-a979-2a459327f07d-istio-podinfo" pod "27bad768-f9df-4649-a979-2a459327f07d" (UID: "27bad768-f9df-4649-a979-2a459327f07d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:34:04.827593 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.827490    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-istio-token" pod "72c2d7cf-fac4-4d80-a5f1-82ae300a255a" (UID: "72c2d7cf-fac4-4d80-a5f1-82ae300a255a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:34:04.827668 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.827620    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-istio-podinfo" pod "f3c4cdb7-e686-4b71-9ab9-3c82f4c24648" (UID: "f3c4cdb7-e686-4b71-9ab9-3c82f4c24648"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:34:04.827784 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.827760    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-istio-token" pod "f3c4cdb7-e686-4b71-9ab9-3c82f4c24648" (UID: "f3c4cdb7-e686-4b71-9ab9-3c82f4c24648"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:34:04.827920 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.827872    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27bad768-f9df-4649-a979-2a459327f07d-kube-api-access-m7wxq" pod "27bad768-f9df-4649-a979-2a459327f07d" (UID: "27bad768-f9df-4649-a979-2a459327f07d"). InnerVolumeSpecName "kube-api-access-m7wxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:34:04.828200 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.828181    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-kube-api-access-mzsp4" pod "f3c4cdb7-e686-4b71-9ab9-3c82f4c24648" (UID: "f3c4cdb7-e686-4b71-9ab9-3c82f4c24648"). InnerVolumeSpecName "kube-api-access-mzsp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:34:04.828280 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.828252    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-istio-podinfo" pod "72c2d7cf-fac4-4d80-a5f1-82ae300a255a" (UID: "72c2d7cf-fac4-4d80-a5f1-82ae300a255a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:34:04.828354 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.828338    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27bad768-f9df-4649-a979-2a459327f07d-istio-token" pod "27bad768-f9df-4649-a979-2a459327f07d" (UID: "27bad768-f9df-4649-a979-2a459327f07d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:34:04.926347 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.926315    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.926347 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.926342    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27bad768-f9df-4649-a979-2a459327f07d-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.926347 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.926353    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7wxq\" (UniqueName: \"kubernetes.io/projected/27bad768-f9df-4649-a979-2a459327f07d-kube-api-access-m7wxq\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.926574 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.926364    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kd2w\" (UniqueName: \"kubernetes.io/projected/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-kube-api-access-5kd2w\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.926574 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.926373    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.926574 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.926382    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27bad768-f9df-4649-a979-2a459327f07d-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.926574 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.926390    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzsp4\" (UniqueName: \"kubernetes.io/projected/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-kube-api-access-mzsp4\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.926574 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.926399    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.926574 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.926407    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.926574 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.926415    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.926574 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.926425    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27bad768-f9df-4649-a979-2a459327f07d-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.926574 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.926433    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72c2d7cf-fac4-4d80-a5f1-82ae300a255a-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.926574 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.926441    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27bad768-f9df-4649-a979-2a459327f07d-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.926574 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.926451    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:04.926574 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:04.926460    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:05.222482 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.222214    4930 patch_prober.go:29] interesting pod/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:34:05.222482 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.222273    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t" podUID="23ed3df4-075f-459b-be45-69da508ea513" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:34:05.371067 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.371033    4930 generic.go:356] "Generic (PLEG): container finished" podID="f3c4cdb7-e686-4b71-9ab9-3c82f4c24648" containerID="143d2317b405875bd15fb6fcbcc10bc0a02c0104fb4b8569a986b10ddc70ec71" exitCode=0
Nov 01 03:34:05.371269 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.371104    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"
Nov 01 03:34:05.371269 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.371117    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j" event={"ID":"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648","Type":"ContainerDied","Data":"143d2317b405875bd15fb6fcbcc10bc0a02c0104fb4b8569a986b10ddc70ec71"}
Nov 01 03:34:05.371269 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.371182    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j" event={"ID":"f3c4cdb7-e686-4b71-9ab9-3c82f4c24648","Type":"ContainerDied","Data":"fe4aa40ed07a33a49e7ea585a8bed77550ef71e0afae2082b93c5a34dcbc451d"}
Nov 01 03:34:05.371269 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.371214    4930 scope.go:122] "RemoveContainer" containerID="143d2317b405875bd15fb6fcbcc10bc0a02c0104fb4b8569a986b10ddc70ec71"
Nov 01 03:34:05.374250 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.374224    4930 generic.go:356] "Generic (PLEG): container finished" podID="72c2d7cf-fac4-4d80-a5f1-82ae300a255a" containerID="dbf783d89d9705308d560551d328a4077052c6f3944362a3803f829548325208" exitCode=0
Nov 01 03:34:05.374406 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.374386    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n" event={"ID":"72c2d7cf-fac4-4d80-a5f1-82ae300a255a","Type":"ContainerDied","Data":"dbf783d89d9705308d560551d328a4077052c6f3944362a3803f829548325208"}
Nov 01 03:34:05.374492 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.374434    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n" event={"ID":"72c2d7cf-fac4-4d80-a5f1-82ae300a255a","Type":"ContainerDied","Data":"5448b858764ca320402cd209fc052704bff5b28df25aac1434dde5dbd46d84fa"}
Nov 01 03:34:05.374572 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.374545    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n"
Nov 01 03:34:05.392595 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.388717    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm" event={"ID":"58f8e6d8-aafa-4a82-b1e4-63c358380207","Type":"ContainerDied","Data":"0d9cd85c21544fdbe215eca8128d6f38c122be48785f9bd1d1b1615e87f5c705"}
Nov 01 03:34:05.392595 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.388815    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"
Nov 01 03:34:05.401647 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.401619    4930 generic.go:356] "Generic (PLEG): container finished" podID="27bad768-f9df-4649-a979-2a459327f07d" containerID="1b6710396ea1c3a864a6e4972e728fe7c165c385b57dfe70cc5e911fa7144143" exitCode=0
Nov 01 03:34:05.401772 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.401667    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz"
Nov 01 03:34:05.401772 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.401763    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz" event={"ID":"27bad768-f9df-4649-a979-2a459327f07d","Type":"ContainerDied","Data":"1b6710396ea1c3a864a6e4972e728fe7c165c385b57dfe70cc5e911fa7144143"}
Nov 01 03:34:05.401889 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.401811    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz" event={"ID":"27bad768-f9df-4649-a979-2a459327f07d","Type":"ContainerDied","Data":"463ea2ab4b419a8f31d8c067d7bb7498b579fd9aea88334b74bab09c59bc9401"}
Nov 01 03:34:05.402821 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.402791    4930 scope.go:122] "RemoveContainer" containerID="50e10515f8b8c5685a5aeb267f96fb5cbda7782d25a15cfad5613df7ba5078b2"
Nov 01 03:34:05.403721 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.403665    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw" event={"ID":"9cd5db87-0172-46ce-b856-cfc6366d7cb2","Type":"ContainerDied","Data":"cc45f2cd577517a9ce54704373293c5877fb25528b3ff8707deb8f737f75edcd"}
Nov 01 03:34:05.403919 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.403902    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw"
Nov 01 03:34:05.405376 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.405350    4930 generic.go:356] "Generic (PLEG): container finished" podID="12d11ff3-9497-498d-8aeb-7f70f1874383" containerID="563b9115b7f7e666b6b16ccdff4721b02322eb406b1ed65d7a5fbc7ec0db8615" exitCode=0
Nov 01 03:34:05.405484 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.405378    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q"
Nov 01 03:34:05.405484 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.405440    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q" event={"ID":"12d11ff3-9497-498d-8aeb-7f70f1874383","Type":"ContainerDied","Data":"563b9115b7f7e666b6b16ccdff4721b02322eb406b1ed65d7a5fbc7ec0db8615"}
Nov 01 03:34:05.405484 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.405481    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q" event={"ID":"12d11ff3-9497-498d-8aeb-7f70f1874383","Type":"ContainerDied","Data":"f08bd3aea28a609c8cc94c8803410df69cb218a9089a1c7510b3f1282dc47cb7"}
Nov 01 03:34:05.420784 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.420765    4930 scope.go:122] "RemoveContainer" containerID="143d2317b405875bd15fb6fcbcc10bc0a02c0104fb4b8569a986b10ddc70ec71"
Nov 01 03:34:05.421089 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:34:05.421061    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"143d2317b405875bd15fb6fcbcc10bc0a02c0104fb4b8569a986b10ddc70ec71\": container with ID starting with 143d2317b405875bd15fb6fcbcc10bc0a02c0104fb4b8569a986b10ddc70ec71 not found: ID does not exist" containerID="143d2317b405875bd15fb6fcbcc10bc0a02c0104fb4b8569a986b10ddc70ec71"
Nov 01 03:34:05.421202 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.421087    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"143d2317b405875bd15fb6fcbcc10bc0a02c0104fb4b8569a986b10ddc70ec71"} err="failed to get container status \"143d2317b405875bd15fb6fcbcc10bc0a02c0104fb4b8569a986b10ddc70ec71\": rpc error: code = NotFound desc = could not find container \"143d2317b405875bd15fb6fcbcc10bc0a02c0104fb4b8569a986b10ddc70ec71\": container with ID starting with 143d2317b405875bd15fb6fcbcc10bc0a02c0104fb4b8569a986b10ddc70ec71 not found: ID does not exist"
Nov 01 03:34:05.421202 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.421103    4930 scope.go:122] "RemoveContainer" containerID="50e10515f8b8c5685a5aeb267f96fb5cbda7782d25a15cfad5613df7ba5078b2"
Nov 01 03:34:05.421375 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:34:05.421359    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50e10515f8b8c5685a5aeb267f96fb5cbda7782d25a15cfad5613df7ba5078b2\": container with ID starting with 50e10515f8b8c5685a5aeb267f96fb5cbda7782d25a15cfad5613df7ba5078b2 not found: ID does not exist" containerID="50e10515f8b8c5685a5aeb267f96fb5cbda7782d25a15cfad5613df7ba5078b2"
Nov 01 03:34:05.421423 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.421380    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50e10515f8b8c5685a5aeb267f96fb5cbda7782d25a15cfad5613df7ba5078b2"} err="failed to get container status \"50e10515f8b8c5685a5aeb267f96fb5cbda7782d25a15cfad5613df7ba5078b2\": rpc error: code = NotFound desc = could not find container \"50e10515f8b8c5685a5aeb267f96fb5cbda7782d25a15cfad5613df7ba5078b2\": container with ID starting with 50e10515f8b8c5685a5aeb267f96fb5cbda7782d25a15cfad5613df7ba5078b2 not found: ID does not exist"
Nov 01 03:34:05.421423 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.421393    4930 scope.go:122] "RemoveContainer" containerID="dbf783d89d9705308d560551d328a4077052c6f3944362a3803f829548325208"
Nov 01 03:34:05.431667 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.431642    4930 scope.go:122] "RemoveContainer" containerID="dbf783d89d9705308d560551d328a4077052c6f3944362a3803f829548325208"
Nov 01 03:34:05.431933 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:34:05.431914    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbf783d89d9705308d560551d328a4077052c6f3944362a3803f829548325208\": container with ID starting with dbf783d89d9705308d560551d328a4077052c6f3944362a3803f829548325208 not found: ID does not exist" containerID="dbf783d89d9705308d560551d328a4077052c6f3944362a3803f829548325208"
Nov 01 03:34:05.431999 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.431941    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbf783d89d9705308d560551d328a4077052c6f3944362a3803f829548325208"} err="failed to get container status \"dbf783d89d9705308d560551d328a4077052c6f3944362a3803f829548325208\": rpc error: code = NotFound desc = could not find container \"dbf783d89d9705308d560551d328a4077052c6f3944362a3803f829548325208\": container with ID starting with dbf783d89d9705308d560551d328a4077052c6f3944362a3803f829548325208 not found: ID does not exist"
Nov 01 03:34:05.431999 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.431957    4930 scope.go:122] "RemoveContainer" containerID="a2a769f870a5917c452b00a2196b52e027797797f5ac49010ec4fec40e450dc6"
Nov 01 03:34:05.441844 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.441824    4930 scope.go:122] "RemoveContainer" containerID="1b6710396ea1c3a864a6e4972e728fe7c165c385b57dfe70cc5e911fa7144143"
Nov 01 03:34:05.451491 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.451472    4930 scope.go:122] "RemoveContainer" containerID="1b6710396ea1c3a864a6e4972e728fe7c165c385b57dfe70cc5e911fa7144143"
Nov 01 03:34:05.451853 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:34:05.451834    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b6710396ea1c3a864a6e4972e728fe7c165c385b57dfe70cc5e911fa7144143\": container with ID starting with 1b6710396ea1c3a864a6e4972e728fe7c165c385b57dfe70cc5e911fa7144143 not found: ID does not exist" containerID="1b6710396ea1c3a864a6e4972e728fe7c165c385b57dfe70cc5e911fa7144143"
Nov 01 03:34:05.451915 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.451860    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b6710396ea1c3a864a6e4972e728fe7c165c385b57dfe70cc5e911fa7144143"} err="failed to get container status \"1b6710396ea1c3a864a6e4972e728fe7c165c385b57dfe70cc5e911fa7144143\": rpc error: code = NotFound desc = could not find container \"1b6710396ea1c3a864a6e4972e728fe7c165c385b57dfe70cc5e911fa7144143\": container with ID starting with 1b6710396ea1c3a864a6e4972e728fe7c165c385b57dfe70cc5e911fa7144143 not found: ID does not exist"
Nov 01 03:34:05.451915 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.451874    4930 scope.go:122] "RemoveContainer" containerID="42ae3b9e3c4d81ad83023657e8fe369041adb8c52cff478cdfa0e186410cd13a"
Nov 01 03:34:05.473235 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.473210    4930 scope.go:122] "RemoveContainer" containerID="563b9115b7f7e666b6b16ccdff4721b02322eb406b1ed65d7a5fbc7ec0db8615"
Nov 01 03:34:05.484922 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.484897    4930 scope.go:122] "RemoveContainer" containerID="563b9115b7f7e666b6b16ccdff4721b02322eb406b1ed65d7a5fbc7ec0db8615"
Nov 01 03:34:05.485282 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:34:05.485248    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"563b9115b7f7e666b6b16ccdff4721b02322eb406b1ed65d7a5fbc7ec0db8615\": container with ID starting with 563b9115b7f7e666b6b16ccdff4721b02322eb406b1ed65d7a5fbc7ec0db8615 not found: ID does not exist" containerID="563b9115b7f7e666b6b16ccdff4721b02322eb406b1ed65d7a5fbc7ec0db8615"
Nov 01 03:34:05.485420 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.485292    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"563b9115b7f7e666b6b16ccdff4721b02322eb406b1ed65d7a5fbc7ec0db8615"} err="failed to get container status \"563b9115b7f7e666b6b16ccdff4721b02322eb406b1ed65d7a5fbc7ec0db8615\": rpc error: code = NotFound desc = could not find container \"563b9115b7f7e666b6b16ccdff4721b02322eb406b1ed65d7a5fbc7ec0db8615\": container with ID starting with 563b9115b7f7e666b6b16ccdff4721b02322eb406b1ed65d7a5fbc7ec0db8615 not found: ID does not exist"
Nov 01 03:34:05.576345 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.575492    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw"]
Nov 01 03:34:05.600566 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.600518    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-kz7lw"]
Nov 01 03:34:05.622168 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.622138    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cd5db87-0172-46ce-b856-cfc6366d7cb2" path="/var/lib/kubelet/pods/9cd5db87-0172-46ce-b856-cfc6366d7cb2/volumes"
Nov 01 03:34:05.709034 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.709007    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz"]
Nov 01 03:34:05.720029 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.720005    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-zbbsz"]
Nov 01 03:34:05.919390 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.909069    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"]
Nov 01 03:34:05.919390 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:05.916423    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-rn4nm"]
Nov 01 03:34:06.035081 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.032162    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:06.039822 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.039794    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"]
Nov 01 03:34:06.046721 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.046698    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-wld6j"]
Nov 01 03:34:06.048959 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.048935    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7"
Nov 01 03:34:06.143618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.142780    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t"
Nov 01 03:34:06.169082 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.167746    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b528dcb-1501-4764-bf26-d676a6e18474-istio-token\" (UniqueName: \"kubernetes.io/projected/4b528dcb-1501-4764-bf26-d676a6e18474-istio-token\") pod \"4b528dcb-1501-4764-bf26-d676a6e18474\" (UID: \"4b528dcb-1501-4764-bf26-d676a6e18474\") "
Nov 01 03:34:06.169082 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.167790    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b528dcb-1501-4764-bf26-d676a6e18474-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b528dcb-1501-4764-bf26-d676a6e18474-credential-socket\") pod \"4b528dcb-1501-4764-bf26-d676a6e18474\" (UID: \"4b528dcb-1501-4764-bf26-d676a6e18474\") "
Nov 01 03:34:06.169082 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.167830    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b528dcb-1501-4764-bf26-d676a6e18474-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b528dcb-1501-4764-bf26-d676a6e18474-workload-socket\") pod \"4b528dcb-1501-4764-bf26-d676a6e18474\" (UID: \"4b528dcb-1501-4764-bf26-d676a6e18474\") "
Nov 01 03:34:06.169082 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.167863    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b528dcb-1501-4764-bf26-d676a6e18474-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b528dcb-1501-4764-bf26-d676a6e18474-istiod-ca-cert\") pod \"4b528dcb-1501-4764-bf26-d676a6e18474\" (UID: \"4b528dcb-1501-4764-bf26-d676a6e18474\") "
Nov 01 03:34:06.169082 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.167936    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b528dcb-1501-4764-bf26-d676a6e18474-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b528dcb-1501-4764-bf26-d676a6e18474-istio-envoy\") pod \"4b528dcb-1501-4764-bf26-d676a6e18474\" (UID: \"4b528dcb-1501-4764-bf26-d676a6e18474\") "
Nov 01 03:34:06.169082 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.168037    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b528dcb-1501-4764-bf26-d676a6e18474-kube-api-access-j9wln\" (UniqueName: \"kubernetes.io/projected/4b528dcb-1501-4764-bf26-d676a6e18474-kube-api-access-j9wln\") pod \"4b528dcb-1501-4764-bf26-d676a6e18474\" (UID: \"4b528dcb-1501-4764-bf26-d676a6e18474\") "
Nov 01 03:34:06.169082 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.168086    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b528dcb-1501-4764-bf26-d676a6e18474-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b528dcb-1501-4764-bf26-d676a6e18474-istio-data\") pod \"4b528dcb-1501-4764-bf26-d676a6e18474\" (UID: \"4b528dcb-1501-4764-bf26-d676a6e18474\") "
Nov 01 03:34:06.169082 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.168103    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b528dcb-1501-4764-bf26-d676a6e18474-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b528dcb-1501-4764-bf26-d676a6e18474-workload-certs\") pod \"4b528dcb-1501-4764-bf26-d676a6e18474\" (UID: \"4b528dcb-1501-4764-bf26-d676a6e18474\") "
Nov 01 03:34:06.169082 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.168157    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b528dcb-1501-4764-bf26-d676a6e18474-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b528dcb-1501-4764-bf26-d676a6e18474-istio-podinfo\") pod \"4b528dcb-1501-4764-bf26-d676a6e18474\" (UID: \"4b528dcb-1501-4764-bf26-d676a6e18474\") "
Nov 01 03:34:06.170323 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.170269    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b528dcb-1501-4764-bf26-d676a6e18474-istiod-ca-cert" pod "4b528dcb-1501-4764-bf26-d676a6e18474" (UID: "4b528dcb-1501-4764-bf26-d676a6e18474"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:34:06.170631 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.169649    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q"]
Nov 01 03:34:06.170983 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.170874    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b528dcb-1501-4764-bf26-d676a6e18474-credential-socket" pod "4b528dcb-1501-4764-bf26-d676a6e18474" (UID: "4b528dcb-1501-4764-bf26-d676a6e18474"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:06.171112 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.171088    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b528dcb-1501-4764-bf26-d676a6e18474-workload-socket" pod "4b528dcb-1501-4764-bf26-d676a6e18474" (UID: "4b528dcb-1501-4764-bf26-d676a6e18474"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:06.171740 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.171708    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b528dcb-1501-4764-bf26-d676a6e18474-istio-podinfo" pod "4b528dcb-1501-4764-bf26-d676a6e18474" (UID: "4b528dcb-1501-4764-bf26-d676a6e18474"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:34:06.171994 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.171969    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b528dcb-1501-4764-bf26-d676a6e18474-istio-data" pod "4b528dcb-1501-4764-bf26-d676a6e18474" (UID: "4b528dcb-1501-4764-bf26-d676a6e18474"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:06.172228 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.172203    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b528dcb-1501-4764-bf26-d676a6e18474-workload-certs" pod "4b528dcb-1501-4764-bf26-d676a6e18474" (UID: "4b528dcb-1501-4764-bf26-d676a6e18474"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:06.174482 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.174277    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b528dcb-1501-4764-bf26-d676a6e18474-kube-api-access-j9wln" pod "4b528dcb-1501-4764-bf26-d676a6e18474" (UID: "4b528dcb-1501-4764-bf26-d676a6e18474"). InnerVolumeSpecName "kube-api-access-j9wln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:34:06.176370 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.176338    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b528dcb-1501-4764-bf26-d676a6e18474-istio-token" pod "4b528dcb-1501-4764-bf26-d676a6e18474" (UID: "4b528dcb-1501-4764-bf26-d676a6e18474"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:34:06.180996 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.180965    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b528dcb-1501-4764-bf26-d676a6e18474-istio-envoy" pod "4b528dcb-1501-4764-bf26-d676a6e18474" (UID: "4b528dcb-1501-4764-bf26-d676a6e18474"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:06.181290 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.181277    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-19faa5-openshift-default-59c44967-57q7q"]
Nov 01 03:34:06.233599 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.233566    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"
Nov 01 03:34:06.235836 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.235815    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk"
Nov 01 03:34:06.269278 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.269203    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23ed3df4-075f-459b-be45-69da508ea513-kube-api-access-tzkt4\" (UniqueName: \"kubernetes.io/projected/23ed3df4-075f-459b-be45-69da508ea513-kube-api-access-tzkt4\") pod \"23ed3df4-075f-459b-be45-69da508ea513\" (UID: \"23ed3df4-075f-459b-be45-69da508ea513\") "
Nov 01 03:34:06.269428 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.269299    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23ed3df4-075f-459b-be45-69da508ea513-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23ed3df4-075f-459b-be45-69da508ea513-istio-data\") pod \"23ed3df4-075f-459b-be45-69da508ea513\" (UID: \"23ed3df4-075f-459b-be45-69da508ea513\") "
Nov 01 03:34:06.269428 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.269387    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23ed3df4-075f-459b-be45-69da508ea513-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ed3df4-075f-459b-be45-69da508ea513-workload-socket\") pod \"23ed3df4-075f-459b-be45-69da508ea513\" (UID: \"23ed3df4-075f-459b-be45-69da508ea513\") "
Nov 01 03:34:06.269428 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.269415    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23ed3df4-075f-459b-be45-69da508ea513-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23ed3df4-075f-459b-be45-69da508ea513-istio-podinfo\") pod \"23ed3df4-075f-459b-be45-69da508ea513\" (UID: \"23ed3df4-075f-459b-be45-69da508ea513\") "
Nov 01 03:34:06.269543 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.269483    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23ed3df4-075f-459b-be45-69da508ea513-istio-token\" (UniqueName: \"kubernetes.io/projected/23ed3df4-075f-459b-be45-69da508ea513-istio-token\") pod \"23ed3df4-075f-459b-be45-69da508ea513\" (UID: \"23ed3df4-075f-459b-be45-69da508ea513\") "
Nov 01 03:34:06.269677 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.269632    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ed3df4-075f-459b-be45-69da508ea513-istio-data" pod "23ed3df4-075f-459b-be45-69da508ea513" (UID: "23ed3df4-075f-459b-be45-69da508ea513"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:06.269677 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.269663    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23ed3df4-075f-459b-be45-69da508ea513-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ed3df4-075f-459b-be45-69da508ea513-credential-socket\") pod \"23ed3df4-075f-459b-be45-69da508ea513\" (UID: \"23ed3df4-075f-459b-be45-69da508ea513\") "
Nov 01 03:34:06.269815 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.269706    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ed3df4-075f-459b-be45-69da508ea513-workload-socket" pod "23ed3df4-075f-459b-be45-69da508ea513" (UID: "23ed3df4-075f-459b-be45-69da508ea513"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:06.269815 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.269723    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23ed3df4-075f-459b-be45-69da508ea513-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23ed3df4-075f-459b-be45-69da508ea513-istiod-ca-cert\") pod \"23ed3df4-075f-459b-be45-69da508ea513\" (UID: \"23ed3df4-075f-459b-be45-69da508ea513\") "
Nov 01 03:34:06.269912 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.269830    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23ed3df4-075f-459b-be45-69da508ea513-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23ed3df4-075f-459b-be45-69da508ea513-istio-envoy\") pod \"23ed3df4-075f-459b-be45-69da508ea513\" (UID: \"23ed3df4-075f-459b-be45-69da508ea513\") "
Nov 01 03:34:06.269967 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.269921    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23ed3df4-075f-459b-be45-69da508ea513-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ed3df4-075f-459b-be45-69da508ea513-workload-certs\") pod \"23ed3df4-075f-459b-be45-69da508ea513\" (UID: \"23ed3df4-075f-459b-be45-69da508ea513\") "
Nov 01 03:34:06.269967 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.269934    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ed3df4-075f-459b-be45-69da508ea513-credential-socket" pod "23ed3df4-075f-459b-be45-69da508ea513" (UID: "23ed3df4-075f-459b-be45-69da508ea513"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:06.270632 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.270009    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23ed3df4-075f-459b-be45-69da508ea513-istiod-ca-cert" pod "23ed3df4-075f-459b-be45-69da508ea513" (UID: "23ed3df4-075f-459b-be45-69da508ea513"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:34:06.270632 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.270300    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ed3df4-075f-459b-be45-69da508ea513-workload-certs" pod "23ed3df4-075f-459b-be45-69da508ea513" (UID: "23ed3df4-075f-459b-be45-69da508ea513"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:06.270815 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.270784    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ed3df4-075f-459b-be45-69da508ea513-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:06.270919 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.270821    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23ed3df4-075f-459b-be45-69da508ea513-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:06.270919 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.270839    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b528dcb-1501-4764-bf26-d676a6e18474-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:06.270919 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.270854    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b528dcb-1501-4764-bf26-d676a6e18474-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:06.270919 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.270886    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b528dcb-1501-4764-bf26-d676a6e18474-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:06.270919 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.270902    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23ed3df4-075f-459b-be45-69da508ea513-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:06.270919 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.270917    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b528dcb-1501-4764-bf26-d676a6e18474-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:06.271101 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.270933    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b528dcb-1501-4764-bf26-d676a6e18474-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:06.271101 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.270948    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23ed3df4-075f-459b-be45-69da508ea513-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:06.271101 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.270966    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9wln\" (UniqueName: \"kubernetes.io/projected/4b528dcb-1501-4764-bf26-d676a6e18474-kube-api-access-j9wln\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:06.271101 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.270981    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23ed3df4-075f-459b-be45-69da508ea513-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:06.271101 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.270995    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b528dcb-1501-4764-bf26-d676a6e18474-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:06.271101 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.271012    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b528dcb-1501-4764-bf26-d676a6e18474-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:06.271101 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.271026    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b528dcb-1501-4764-bf26-d676a6e18474-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:06.271768 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.271741    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23ed3df4-075f-459b-be45-69da508ea513-kube-api-access-tzkt4" pod "23ed3df4-075f-459b-be45-69da508ea513" (UID: "23ed3df4-075f-459b-be45-69da508ea513"). InnerVolumeSpecName "kube-api-access-tzkt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:34:06.271858 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.271817    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23ed3df4-075f-459b-be45-69da508ea513-istio-podinfo" pod "23ed3df4-075f-459b-be45-69da508ea513" (UID: "23ed3df4-075f-459b-be45-69da508ea513"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:34:06.271961 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.271946    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23ed3df4-075f-459b-be45-69da508ea513-istio-token" pod "23ed3df4-075f-459b-be45-69da508ea513" (UID: "23ed3df4-075f-459b-be45-69da508ea513"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:34:06.272073 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.272059    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23ed3df4-075f-459b-be45-69da508ea513-istio-envoy" pod "23ed3df4-075f-459b-be45-69da508ea513" (UID: "23ed3df4-075f-459b-be45-69da508ea513"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:06.280985 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.280962    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n"]
Nov 01 03:34:06.287876 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.287855    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-dsx9n"]
Nov 01 03:34:06.372373 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.372348    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd824f02-6c5a-4428-a995-80cbf1b20571-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd824f02-6c5a-4428-a995-80cbf1b20571-istiod-ca-cert\") pod \"fd824f02-6c5a-4428-a995-80cbf1b20571\" (UID: \"fd824f02-6c5a-4428-a995-80cbf1b20571\") "
Nov 01 03:34:06.372477 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.372394    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/267415f5-02bf-4571-914a-45aa36ae41c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/267415f5-02bf-4571-914a-45aa36ae41c9-credential-socket\") pod \"267415f5-02bf-4571-914a-45aa36ae41c9\" (UID: \"267415f5-02bf-4571-914a-45aa36ae41c9\") "
Nov 01 03:34:06.372477 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.372418    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/267415f5-02bf-4571-914a-45aa36ae41c9-istio-token\" (UniqueName: \"kubernetes.io/projected/267415f5-02bf-4571-914a-45aa36ae41c9-istio-token\") pod \"267415f5-02bf-4571-914a-45aa36ae41c9\" (UID: \"267415f5-02bf-4571-914a-45aa36ae41c9\") "
Nov 01 03:34:06.372617 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.372530    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd824f02-6c5a-4428-a995-80cbf1b20571-kube-api-access-bsh5t\" (UniqueName: \"kubernetes.io/projected/fd824f02-6c5a-4428-a995-80cbf1b20571-kube-api-access-bsh5t\") pod \"fd824f02-6c5a-4428-a995-80cbf1b20571\" (UID: \"fd824f02-6c5a-4428-a995-80cbf1b20571\") "
Nov 01 03:34:06.372686 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.372657    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/267415f5-02bf-4571-914a-45aa36ae41c9-credential-socket" pod "267415f5-02bf-4571-914a-45aa36ae41c9" (UID: "267415f5-02bf-4571-914a-45aa36ae41c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:06.372742 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.372695    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd824f02-6c5a-4428-a995-80cbf1b20571-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd824f02-6c5a-4428-a995-80cbf1b20571-workload-socket\") pod \"fd824f02-6c5a-4428-a995-80cbf1b20571\" (UID: \"fd824f02-6c5a-4428-a995-80cbf1b20571\") "
Nov 01 03:34:06.372797 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.372747    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/267415f5-02bf-4571-914a-45aa36ae41c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/267415f5-02bf-4571-914a-45aa36ae41c9-istio-podinfo\") pod \"267415f5-02bf-4571-914a-45aa36ae41c9\" (UID: \"267415f5-02bf-4571-914a-45aa36ae41c9\") "
Nov 01 03:34:06.372863 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.372815    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd824f02-6c5a-4428-a995-80cbf1b20571-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd824f02-6c5a-4428-a995-80cbf1b20571-credential-socket\") pod \"fd824f02-6c5a-4428-a995-80cbf1b20571\" (UID: \"fd824f02-6c5a-4428-a995-80cbf1b20571\") "
Nov 01 03:34:06.372918 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.372878    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd824f02-6c5a-4428-a995-80cbf1b20571-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd824f02-6c5a-4428-a995-80cbf1b20571-workload-certs\") pod \"fd824f02-6c5a-4428-a995-80cbf1b20571\" (UID: \"fd824f02-6c5a-4428-a995-80cbf1b20571\") "
Nov 01 03:34:06.372970 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.372812    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd824f02-6c5a-4428-a995-80cbf1b20571-istiod-ca-cert" pod "fd824f02-6c5a-4428-a995-80cbf1b20571" (UID: "fd824f02-6c5a-4428-a995-80cbf1b20571"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:34:06.372970 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.372948    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/267415f5-02bf-4571-914a-45aa36ae41c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/267415f5-02bf-4571-914a-45aa36ae41c9-workload-socket\") pod \"267415f5-02bf-4571-914a-45aa36ae41c9\" (UID: \"267415f5-02bf-4571-914a-45aa36ae41c9\") "
Nov 01 03:34:06.373080 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.373021    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd824f02-6c5a-4428-a995-80cbf1b20571-workload-socket" pod "fd824f02-6c5a-4428-a995-80cbf1b20571" (UID: "fd824f02-6c5a-4428-a995-80cbf1b20571"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:06.373143 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.373100    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd824f02-6c5a-4428-a995-80cbf1b20571-credential-socket" pod "fd824f02-6c5a-4428-a995-80cbf1b20571" (UID: "fd824f02-6c5a-4428-a995-80cbf1b20571"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:06.373201 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.373169    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd824f02-6c5a-4428-a995-80cbf1b20571-workload-certs" pod "fd824f02-6c5a-4428-a995-80cbf1b20571" (UID: "fd824f02-6c5a-4428-a995-80cbf1b20571"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:06.373259 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.373204    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/267415f5-02bf-4571-914a-45aa36ae41c9-kube-api-access-wxg92\" (UniqueName: \"kubernetes.io/projected/267415f5-02bf-4571-914a-45aa36ae41c9-kube-api-access-wxg92\") pod \"267415f5-02bf-4571-914a-45aa36ae41c9\" (UID: \"267415f5-02bf-4571-914a-45aa36ae41c9\") "
Nov 01 03:34:06.373259 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.373245    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/267415f5-02bf-4571-914a-45aa36ae41c9-workload-socket" pod "267415f5-02bf-4571-914a-45aa36ae41c9" (UID: "267415f5-02bf-4571-914a-45aa36ae41c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:06.373367 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.373254    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/267415f5-02bf-4571-914a-45aa36ae41c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/267415f5-02bf-4571-914a-45aa36ae41c9-istio-envoy\") pod \"267415f5-02bf-4571-914a-45aa36ae41c9\" (UID: \"267415f5-02bf-4571-914a-45aa36ae41c9\") "
Nov 01 03:34:06.373367 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.373324    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd824f02-6c5a-4428-a995-80cbf1b20571-istio-token\" (UniqueName: \"kubernetes.io/projected/fd824f02-6c5a-4428-a995-80cbf1b20571-istio-token\") pod \"fd824f02-6c5a-4428-a995-80cbf1b20571\" (UID: \"fd824f02-6c5a-4428-a995-80cbf1b20571\") "
Nov 01 03:34:06.373367 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.373358    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/267415f5-02bf-4571-914a-45aa36ae41c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/267415f5-02bf-4571-914a-45aa36ae41c9-workload-certs\") pod \"267415f5-02bf-4571-914a-45aa36ae41c9\" (UID: \"267415f5-02bf-4571-914a-45aa36ae41c9\") "
Nov 01 03:34:06.373531 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.373424    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd824f02-6c5a-4428-a995-80cbf1b20571-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd824f02-6c5a-4428-a995-80cbf1b20571-istio-podinfo\") pod \"fd824f02-6c5a-4428-a995-80cbf1b20571\" (UID: \"fd824f02-6c5a-4428-a995-80cbf1b20571\") "
Nov 01 03:34:06.373531 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.373470    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd824f02-6c5a-4428-a995-80cbf1b20571-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd824f02-6c5a-4428-a995-80cbf1b20571-istio-data\") pod \"fd824f02-6c5a-4428-a995-80cbf1b20571\" (UID: \"fd824f02-6c5a-4428-a995-80cbf1b20571\") "
Nov 01 03:34:06.373531 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.373507    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/267415f5-02bf-4571-914a-45aa36ae41c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/267415f5-02bf-4571-914a-45aa36ae41c9-istio-data\") pod \"267415f5-02bf-4571-914a-45aa36ae41c9\" (UID: \"267415f5-02bf-4571-914a-45aa36ae41c9\") "
Nov 01 03:34:06.373727 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.373543    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/267415f5-02bf-4571-914a-45aa36ae41c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/267415f5-02bf-4571-914a-45aa36ae41c9-istiod-ca-cert\") pod \"267415f5-02bf-4571-914a-45aa36ae41c9\" (UID: \"267415f5-02bf-4571-914a-45aa36ae41c9\") "
Nov 01 03:34:06.373727 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.373665    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd824f02-6c5a-4428-a995-80cbf1b20571-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd824f02-6c5a-4428-a995-80cbf1b20571-istio-envoy\") pod \"fd824f02-6c5a-4428-a995-80cbf1b20571\" (UID: \"fd824f02-6c5a-4428-a995-80cbf1b20571\") "
Nov 01 03:34:06.374448 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.374420    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzkt4\" (UniqueName: \"kubernetes.io/projected/23ed3df4-075f-459b-be45-69da508ea513-kube-api-access-tzkt4\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:06.374550 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.374453    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd824f02-6c5a-4428-a995-80cbf1b20571-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:06.374550 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.374468    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd824f02-6c5a-4428-a995-80cbf1b20571-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:06.374550 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.374484    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd824f02-6c5a-4428-a995-80cbf1b20571-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:06.374550 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.374498    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/267415f5-02bf-4571-914a-45aa36ae41c9-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:06.374550 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.374514    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23ed3df4-075f-459b-be45-69da508ea513-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:06.374550 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.374528    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23ed3df4-075f-459b-be45-69da508ea513-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:06.374550 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.374542    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd824f02-6c5a-4428-a995-80cbf1b20571-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:06.374550 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.374572    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/267415f5-02bf-4571-914a-45aa36ae41c9-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:06.374550 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.374587    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23ed3df4-075f-459b-be45-69da508ea513-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:06.375199 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.375172    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/267415f5-02bf-4571-914a-45aa36ae41c9-istio-data" pod "267415f5-02bf-4571-914a-45aa36ae41c9" (UID: "267415f5-02bf-4571-914a-45aa36ae41c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:06.375277 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.375186    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/267415f5-02bf-4571-914a-45aa36ae41c9-workload-certs" pod "267415f5-02bf-4571-914a-45aa36ae41c9" (UID: "267415f5-02bf-4571-914a-45aa36ae41c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:06.375534 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.375512    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/267415f5-02bf-4571-914a-45aa36ae41c9-istiod-ca-cert" pod "267415f5-02bf-4571-914a-45aa36ae41c9" (UID: "267415f5-02bf-4571-914a-45aa36ae41c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:34:06.375677 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.375653    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/267415f5-02bf-4571-914a-45aa36ae41c9-istio-token" pod "267415f5-02bf-4571-914a-45aa36ae41c9" (UID: "267415f5-02bf-4571-914a-45aa36ae41c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:34:06.375791 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.375672    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/267415f5-02bf-4571-914a-45aa36ae41c9-kube-api-access-wxg92" pod "267415f5-02bf-4571-914a-45aa36ae41c9" (UID: "267415f5-02bf-4571-914a-45aa36ae41c9"). InnerVolumeSpecName "kube-api-access-wxg92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:34:06.375855 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.375798    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/267415f5-02bf-4571-914a-45aa36ae41c9-istio-podinfo" pod "267415f5-02bf-4571-914a-45aa36ae41c9" (UID: "267415f5-02bf-4571-914a-45aa36ae41c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:34:06.375946 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.375916    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd824f02-6c5a-4428-a995-80cbf1b20571-istio-token" pod "fd824f02-6c5a-4428-a995-80cbf1b20571" (UID: "fd824f02-6c5a-4428-a995-80cbf1b20571"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:34:06.376072 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.376048    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/267415f5-02bf-4571-914a-45aa36ae41c9-istio-envoy" pod "267415f5-02bf-4571-914a-45aa36ae41c9" (UID: "267415f5-02bf-4571-914a-45aa36ae41c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:06.376140 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.376064    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd824f02-6c5a-4428-a995-80cbf1b20571-istio-data" pod "fd824f02-6c5a-4428-a995-80cbf1b20571" (UID: "fd824f02-6c5a-4428-a995-80cbf1b20571"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:06.376340 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.376317    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd824f02-6c5a-4428-a995-80cbf1b20571-kube-api-access-bsh5t" pod "fd824f02-6c5a-4428-a995-80cbf1b20571" (UID: "fd824f02-6c5a-4428-a995-80cbf1b20571"). InnerVolumeSpecName "kube-api-access-bsh5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:34:06.376804 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.376779    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd824f02-6c5a-4428-a995-80cbf1b20571-istio-envoy" pod "fd824f02-6c5a-4428-a995-80cbf1b20571" (UID: "fd824f02-6c5a-4428-a995-80cbf1b20571"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:06.376942 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.376925    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd824f02-6c5a-4428-a995-80cbf1b20571-istio-podinfo" pod "fd824f02-6c5a-4428-a995-80cbf1b20571" (UID: "fd824f02-6c5a-4428-a995-80cbf1b20571"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:34:06.421288 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.421259    4930 generic.go:356] "Generic (PLEG): container finished" podID="267415f5-02bf-4571-914a-45aa36ae41c9" containerID="e79fab7e2557d937d8010fea46cbac0a838d03f17abb66abf00e348bec67aa5d" exitCode=0
Nov 01 03:34:06.421494 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.421332    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"
Nov 01 03:34:06.421494 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.421348    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr" event={"ID":"267415f5-02bf-4571-914a-45aa36ae41c9","Type":"ContainerDied","Data":"e79fab7e2557d937d8010fea46cbac0a838d03f17abb66abf00e348bec67aa5d"}
Nov 01 03:34:06.421494 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.421417    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr" event={"ID":"267415f5-02bf-4571-914a-45aa36ae41c9","Type":"ContainerDied","Data":"8bdb2dcda1f847538c19a2d1b802fb4573e7cd87b90d3c4b7703b21f8e68614b"}
Nov 01 03:34:06.421494 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.421456    4930 scope.go:122] "RemoveContainer" containerID="e79fab7e2557d937d8010fea46cbac0a838d03f17abb66abf00e348bec67aa5d"
Nov 01 03:34:06.424954 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.424930    4930 generic.go:356] "Generic (PLEG): container finished" podID="fd824f02-6c5a-4428-a995-80cbf1b20571" containerID="5778c35aad39918487bc447f89f9ff1a99fc3fcf8d7df7b4060f4a6425121b77" exitCode=0
Nov 01 03:34:06.425099 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.425082    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk"
Nov 01 03:34:06.426853 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.426814    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk" event={"ID":"fd824f02-6c5a-4428-a995-80cbf1b20571","Type":"ContainerDied","Data":"5778c35aad39918487bc447f89f9ff1a99fc3fcf8d7df7b4060f4a6425121b77"}
Nov 01 03:34:06.428132 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.427395    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk" event={"ID":"fd824f02-6c5a-4428-a995-80cbf1b20571","Type":"ContainerDied","Data":"77904612833e66680b4f4011e7e039e7c264c8ebac051cb65a0c3f6a121504c1"}
Nov 01 03:34:06.435550 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.430679    4930 generic.go:356] "Generic (PLEG): container finished" podID="4b528dcb-1501-4764-bf26-d676a6e18474" containerID="8e3a98e5f62c2e50232c9ac38e464abe3bee0d658efdb9265bd52afb76278b1a" exitCode=0
Nov 01 03:34:06.435550 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.430771    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7"
Nov 01 03:34:06.435550 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.430868    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7" event={"ID":"4b528dcb-1501-4764-bf26-d676a6e18474","Type":"ContainerDied","Data":"8e3a98e5f62c2e50232c9ac38e464abe3bee0d658efdb9265bd52afb76278b1a"}
Nov 01 03:34:06.435550 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.430924    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7" event={"ID":"4b528dcb-1501-4764-bf26-d676a6e18474","Type":"ContainerDied","Data":"fe158cb237022690dc27d46e631057bced9c33dbd953125b2474f45fbbfc3003"}
Nov 01 03:34:06.440055 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.439619    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t"
Nov 01 03:34:06.440055 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.439639    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t" event={"ID":"23ed3df4-075f-459b-be45-69da508ea513","Type":"ContainerDied","Data":"b2780b594982117c704a8352f5ac248d50a664e116cbd6c9c3e888b35d2b2082"}
Nov 01 03:34:06.440055 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.439617    4930 generic.go:356] "Generic (PLEG): container finished" podID="23ed3df4-075f-459b-be45-69da508ea513" containerID="b2780b594982117c704a8352f5ac248d50a664e116cbd6c9c3e888b35d2b2082" exitCode=0
Nov 01 03:34:06.440055 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.439747    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t" event={"ID":"23ed3df4-075f-459b-be45-69da508ea513","Type":"ContainerDied","Data":"5667d2e9ab92d40e137bf5e30c9dc66f280425826b44cbdac6a4186789b572a6"}
Nov 01 03:34:06.440265 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.440102    4930 scope.go:122] "RemoveContainer" containerID="414127804b6b9477bd68c04db236c25a65d858c5da049a77a83d81fa391fd91b"
Nov 01 03:34:06.453917 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.453899    4930 scope.go:122] "RemoveContainer" containerID="e79fab7e2557d937d8010fea46cbac0a838d03f17abb66abf00e348bec67aa5d"
Nov 01 03:34:06.454369 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:34:06.454343    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e79fab7e2557d937d8010fea46cbac0a838d03f17abb66abf00e348bec67aa5d\": container with ID starting with e79fab7e2557d937d8010fea46cbac0a838d03f17abb66abf00e348bec67aa5d not found: ID does not exist" containerID="e79fab7e2557d937d8010fea46cbac0a838d03f17abb66abf00e348bec67aa5d"
Nov 01 03:34:06.454463 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.454378    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e79fab7e2557d937d8010fea46cbac0a838d03f17abb66abf00e348bec67aa5d"} err="failed to get container status \"e79fab7e2557d937d8010fea46cbac0a838d03f17abb66abf00e348bec67aa5d\": rpc error: code = NotFound desc = could not find container \"e79fab7e2557d937d8010fea46cbac0a838d03f17abb66abf00e348bec67aa5d\": container with ID starting with e79fab7e2557d937d8010fea46cbac0a838d03f17abb66abf00e348bec67aa5d not found: ID does not exist"
Nov 01 03:34:06.454463 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.454399    4930 scope.go:122] "RemoveContainer" containerID="414127804b6b9477bd68c04db236c25a65d858c5da049a77a83d81fa391fd91b"
Nov 01 03:34:06.455059 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:34:06.454967    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"414127804b6b9477bd68c04db236c25a65d858c5da049a77a83d81fa391fd91b\": container with ID starting with 414127804b6b9477bd68c04db236c25a65d858c5da049a77a83d81fa391fd91b not found: ID does not exist" containerID="414127804b6b9477bd68c04db236c25a65d858c5da049a77a83d81fa391fd91b"
Nov 01 03:34:06.455059 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.454995    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"414127804b6b9477bd68c04db236c25a65d858c5da049a77a83d81fa391fd91b"} err="failed to get container status \"414127804b6b9477bd68c04db236c25a65d858c5da049a77a83d81fa391fd91b\": rpc error: code = NotFound desc = could not find container \"414127804b6b9477bd68c04db236c25a65d858c5da049a77a83d81fa391fd91b\": container with ID starting with 414127804b6b9477bd68c04db236c25a65d858c5da049a77a83d81fa391fd91b not found: ID does not exist"
Nov 01 03:34:06.455059 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.455019    4930 scope.go:122] "RemoveContainer" containerID="5778c35aad39918487bc447f89f9ff1a99fc3fcf8d7df7b4060f4a6425121b77"
Nov 01 03:34:06.468378 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.468355    4930 scope.go:122] "RemoveContainer" containerID="5778c35aad39918487bc447f89f9ff1a99fc3fcf8d7df7b4060f4a6425121b77"
Nov 01 03:34:06.468734 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:34:06.468705    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5778c35aad39918487bc447f89f9ff1a99fc3fcf8d7df7b4060f4a6425121b77\": container with ID starting with 5778c35aad39918487bc447f89f9ff1a99fc3fcf8d7df7b4060f4a6425121b77 not found: ID does not exist" containerID="5778c35aad39918487bc447f89f9ff1a99fc3fcf8d7df7b4060f4a6425121b77"
Nov 01 03:34:06.468834 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.468743    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5778c35aad39918487bc447f89f9ff1a99fc3fcf8d7df7b4060f4a6425121b77"} err="failed to get container status \"5778c35aad39918487bc447f89f9ff1a99fc3fcf8d7df7b4060f4a6425121b77\": rpc error: code = NotFound desc = could not find container \"5778c35aad39918487bc447f89f9ff1a99fc3fcf8d7df7b4060f4a6425121b77\": container with ID starting with 5778c35aad39918487bc447f89f9ff1a99fc3fcf8d7df7b4060f4a6425121b77 not found: ID does not exist"
Nov 01 03:34:06.468834 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.468765    4930 scope.go:122] "RemoveContainer" containerID="8e3a98e5f62c2e50232c9ac38e464abe3bee0d658efdb9265bd52afb76278b1a"
Nov 01 03:34:06.476302 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.476276    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/267415f5-02bf-4571-914a-45aa36ae41c9-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:06.476386 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.476309    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsh5t\" (UniqueName: \"kubernetes.io/projected/fd824f02-6c5a-4428-a995-80cbf1b20571-kube-api-access-bsh5t\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:06.476386 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.476327    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/267415f5-02bf-4571-914a-45aa36ae41c9-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:06.476386 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.476342    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxg92\" (UniqueName: \"kubernetes.io/projected/267415f5-02bf-4571-914a-45aa36ae41c9-kube-api-access-wxg92\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:06.476386 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.476356    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/267415f5-02bf-4571-914a-45aa36ae41c9-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:06.476386 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.476370    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd824f02-6c5a-4428-a995-80cbf1b20571-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:06.476386 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.476384    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/267415f5-02bf-4571-914a-45aa36ae41c9-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:06.476599 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.476398    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd824f02-6c5a-4428-a995-80cbf1b20571-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:06.476599 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.476421    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd824f02-6c5a-4428-a995-80cbf1b20571-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:06.476599 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.476435    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/267415f5-02bf-4571-914a-45aa36ae41c9-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:06.476599 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.476448    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/267415f5-02bf-4571-914a-45aa36ae41c9-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:06.476599 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.476463    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd824f02-6c5a-4428-a995-80cbf1b20571-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:06.479586 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.479566    4930 scope.go:122] "RemoveContainer" containerID="8e3a98e5f62c2e50232c9ac38e464abe3bee0d658efdb9265bd52afb76278b1a"
Nov 01 03:34:06.479858 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:34:06.479839    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e3a98e5f62c2e50232c9ac38e464abe3bee0d658efdb9265bd52afb76278b1a\": container with ID starting with 8e3a98e5f62c2e50232c9ac38e464abe3bee0d658efdb9265bd52afb76278b1a not found: ID does not exist" containerID="8e3a98e5f62c2e50232c9ac38e464abe3bee0d658efdb9265bd52afb76278b1a"
Nov 01 03:34:06.479912 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.479864    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e3a98e5f62c2e50232c9ac38e464abe3bee0d658efdb9265bd52afb76278b1a"} err="failed to get container status \"8e3a98e5f62c2e50232c9ac38e464abe3bee0d658efdb9265bd52afb76278b1a\": rpc error: code = NotFound desc = could not find container \"8e3a98e5f62c2e50232c9ac38e464abe3bee0d658efdb9265bd52afb76278b1a\": container with ID starting with 8e3a98e5f62c2e50232c9ac38e464abe3bee0d658efdb9265bd52afb76278b1a not found: ID does not exist"
Nov 01 03:34:06.479912 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.479882    4930 scope.go:122] "RemoveContainer" containerID="b2780b594982117c704a8352f5ac248d50a664e116cbd6c9c3e888b35d2b2082"
Nov 01 03:34:06.489752 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.489734    4930 scope.go:122] "RemoveContainer" containerID="b2780b594982117c704a8352f5ac248d50a664e116cbd6c9c3e888b35d2b2082"
Nov 01 03:34:06.490016 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:34:06.489996    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2780b594982117c704a8352f5ac248d50a664e116cbd6c9c3e888b35d2b2082\": container with ID starting with b2780b594982117c704a8352f5ac248d50a664e116cbd6c9c3e888b35d2b2082 not found: ID does not exist" containerID="b2780b594982117c704a8352f5ac248d50a664e116cbd6c9c3e888b35d2b2082"
Nov 01 03:34:06.490080 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.490024    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2780b594982117c704a8352f5ac248d50a664e116cbd6c9c3e888b35d2b2082"} err="failed to get container status \"b2780b594982117c704a8352f5ac248d50a664e116cbd6c9c3e888b35d2b2082\": rpc error: code = NotFound desc = could not find container \"b2780b594982117c704a8352f5ac248d50a664e116cbd6c9c3e888b35d2b2082\": container with ID starting with b2780b594982117c704a8352f5ac248d50a664e116cbd6c9c3e888b35d2b2082 not found: ID does not exist"
Nov 01 03:34:06.899759 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.899718    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"]
Nov 01 03:34:06.915390 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:06.915355    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-zbrfr"]
Nov 01 03:34:07.022605 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:07.022543    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk"]
Nov 01 03:34:07.029604 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:07.029573    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-lrkwk"]
Nov 01 03:34:07.138941 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:07.138906    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t"]
Nov 01 03:34:07.147489 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:07.147451    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-jb68t"]
Nov 01 03:34:07.254752 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:07.254670    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7"]
Nov 01 03:34:07.258293 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:07.258263    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c8cb9a-openshift-default-5dbc574c7c-nkcj7"]
Nov 01 03:34:07.627855 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:07.627823    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12d11ff3-9497-498d-8aeb-7f70f1874383" path="/var/lib/kubelet/pods/12d11ff3-9497-498d-8aeb-7f70f1874383/volumes"
Nov 01 03:34:07.628329 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:07.628314    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23ed3df4-075f-459b-be45-69da508ea513" path="/var/lib/kubelet/pods/23ed3df4-075f-459b-be45-69da508ea513/volumes"
Nov 01 03:34:07.628830 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:07.628816    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="267415f5-02bf-4571-914a-45aa36ae41c9" path="/var/lib/kubelet/pods/267415f5-02bf-4571-914a-45aa36ae41c9/volumes"
Nov 01 03:34:07.629630 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:07.629615    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27bad768-f9df-4649-a979-2a459327f07d" path="/var/lib/kubelet/pods/27bad768-f9df-4649-a979-2a459327f07d/volumes"
Nov 01 03:34:07.630097 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:07.630083    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b528dcb-1501-4764-bf26-d676a6e18474" path="/var/lib/kubelet/pods/4b528dcb-1501-4764-bf26-d676a6e18474/volumes"
Nov 01 03:34:07.630532 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:07.630518    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58f8e6d8-aafa-4a82-b1e4-63c358380207" path="/var/lib/kubelet/pods/58f8e6d8-aafa-4a82-b1e4-63c358380207/volumes"
Nov 01 03:34:07.631271 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:07.631256    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72c2d7cf-fac4-4d80-a5f1-82ae300a255a" path="/var/lib/kubelet/pods/72c2d7cf-fac4-4d80-a5f1-82ae300a255a/volumes"
Nov 01 03:34:07.631722 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:07.631709    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3c4cdb7-e686-4b71-9ab9-3c82f4c24648" path="/var/lib/kubelet/pods/f3c4cdb7-e686-4b71-9ab9-3c82f4c24648/volumes"
Nov 01 03:34:07.632399 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:07.632385    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd824f02-6c5a-4428-a995-80cbf1b20571" path="/var/lib/kubelet/pods/fd824f02-6c5a-4428-a995-80cbf1b20571/volumes"
Nov 01 03:34:08.783025 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:08.783003    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"
Nov 01 03:34:08.841406 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:08.841378    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9165d842-245c-4270-a5ab-8789ab0a2e89-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9165d842-245c-4270-a5ab-8789ab0a2e89-istio-data\") pod \"9165d842-245c-4270-a5ab-8789ab0a2e89\" (UID: \"9165d842-245c-4270-a5ab-8789ab0a2e89\") "
Nov 01 03:34:08.841540 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:08.841423    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9165d842-245c-4270-a5ab-8789ab0a2e89-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9165d842-245c-4270-a5ab-8789ab0a2e89-istio-envoy\") pod \"9165d842-245c-4270-a5ab-8789ab0a2e89\" (UID: \"9165d842-245c-4270-a5ab-8789ab0a2e89\") "
Nov 01 03:34:08.841540 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:08.841466    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9165d842-245c-4270-a5ab-8789ab0a2e89-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9165d842-245c-4270-a5ab-8789ab0a2e89-workload-certs\") pod \"9165d842-245c-4270-a5ab-8789ab0a2e89\" (UID: \"9165d842-245c-4270-a5ab-8789ab0a2e89\") "
Nov 01 03:34:08.841540 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:08.841535    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9165d842-245c-4270-a5ab-8789ab0a2e89-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9165d842-245c-4270-a5ab-8789ab0a2e89-istiod-ca-cert\") pod \"9165d842-245c-4270-a5ab-8789ab0a2e89\" (UID: \"9165d842-245c-4270-a5ab-8789ab0a2e89\") "
Nov 01 03:34:08.841725 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:08.841577    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9165d842-245c-4270-a5ab-8789ab0a2e89-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9165d842-245c-4270-a5ab-8789ab0a2e89-istio-podinfo\") pod \"9165d842-245c-4270-a5ab-8789ab0a2e89\" (UID: \"9165d842-245c-4270-a5ab-8789ab0a2e89\") "
Nov 01 03:34:08.841725 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:08.841606    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9165d842-245c-4270-a5ab-8789ab0a2e89-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9165d842-245c-4270-a5ab-8789ab0a2e89-credential-socket\") pod \"9165d842-245c-4270-a5ab-8789ab0a2e89\" (UID: \"9165d842-245c-4270-a5ab-8789ab0a2e89\") "
Nov 01 03:34:08.841725 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:08.841646    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9165d842-245c-4270-a5ab-8789ab0a2e89-kube-api-access-sjq2p\" (UniqueName: \"kubernetes.io/projected/9165d842-245c-4270-a5ab-8789ab0a2e89-kube-api-access-sjq2p\") pod \"9165d842-245c-4270-a5ab-8789ab0a2e89\" (UID: \"9165d842-245c-4270-a5ab-8789ab0a2e89\") "
Nov 01 03:34:08.841725 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:08.841661    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9165d842-245c-4270-a5ab-8789ab0a2e89-istio-data" pod "9165d842-245c-4270-a5ab-8789ab0a2e89" (UID: "9165d842-245c-4270-a5ab-8789ab0a2e89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:08.841918 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:08.841851    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9165d842-245c-4270-a5ab-8789ab0a2e89-workload-certs" pod "9165d842-245c-4270-a5ab-8789ab0a2e89" (UID: "9165d842-245c-4270-a5ab-8789ab0a2e89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:08.841918 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:08.841861    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9165d842-245c-4270-a5ab-8789ab0a2e89-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9165d842-245c-4270-a5ab-8789ab0a2e89-workload-socket\") pod \"9165d842-245c-4270-a5ab-8789ab0a2e89\" (UID: \"9165d842-245c-4270-a5ab-8789ab0a2e89\") "
Nov 01 03:34:08.842022 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:08.841989    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9165d842-245c-4270-a5ab-8789ab0a2e89-credential-socket" pod "9165d842-245c-4270-a5ab-8789ab0a2e89" (UID: "9165d842-245c-4270-a5ab-8789ab0a2e89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:08.842080 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:08.842023    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9165d842-245c-4270-a5ab-8789ab0a2e89-istio-token\" (UniqueName: \"kubernetes.io/projected/9165d842-245c-4270-a5ab-8789ab0a2e89-istio-token\") pod \"9165d842-245c-4270-a5ab-8789ab0a2e89\" (UID: \"9165d842-245c-4270-a5ab-8789ab0a2e89\") "
Nov 01 03:34:08.842197 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:08.842158    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9165d842-245c-4270-a5ab-8789ab0a2e89-workload-socket" pod "9165d842-245c-4270-a5ab-8789ab0a2e89" (UID: "9165d842-245c-4270-a5ab-8789ab0a2e89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:08.842197 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:08.842093    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9165d842-245c-4270-a5ab-8789ab0a2e89-istiod-ca-cert" pod "9165d842-245c-4270-a5ab-8789ab0a2e89" (UID: "9165d842-245c-4270-a5ab-8789ab0a2e89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:34:08.842763 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:08.842735    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9165d842-245c-4270-a5ab-8789ab0a2e89-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:08.842854 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:08.842764    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9165d842-245c-4270-a5ab-8789ab0a2e89-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:08.842854 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:08.842781    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9165d842-245c-4270-a5ab-8789ab0a2e89-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:08.842854 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:08.842796    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9165d842-245c-4270-a5ab-8789ab0a2e89-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:08.842854 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:08.842811    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9165d842-245c-4270-a5ab-8789ab0a2e89-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:08.843784 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:08.843759    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9165d842-245c-4270-a5ab-8789ab0a2e89-istio-envoy" pod "9165d842-245c-4270-a5ab-8789ab0a2e89" (UID: "9165d842-245c-4270-a5ab-8789ab0a2e89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:08.843991 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:08.843973    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9165d842-245c-4270-a5ab-8789ab0a2e89-istio-podinfo" pod "9165d842-245c-4270-a5ab-8789ab0a2e89" (UID: "9165d842-245c-4270-a5ab-8789ab0a2e89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:34:08.843991 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:08.843979    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9165d842-245c-4270-a5ab-8789ab0a2e89-kube-api-access-sjq2p" pod "9165d842-245c-4270-a5ab-8789ab0a2e89" (UID: "9165d842-245c-4270-a5ab-8789ab0a2e89"). InnerVolumeSpecName "kube-api-access-sjq2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:34:08.844218 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:08.844200    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9165d842-245c-4270-a5ab-8789ab0a2e89-istio-token" pod "9165d842-245c-4270-a5ab-8789ab0a2e89" (UID: "9165d842-245c-4270-a5ab-8789ab0a2e89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:34:08.944133 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:08.944098    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjq2p\" (UniqueName: \"kubernetes.io/projected/9165d842-245c-4270-a5ab-8789ab0a2e89-kube-api-access-sjq2p\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:08.944262 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:08.944134    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9165d842-245c-4270-a5ab-8789ab0a2e89-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:08.944262 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:08.944151    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9165d842-245c-4270-a5ab-8789ab0a2e89-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:08.944262 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:08.944166    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9165d842-245c-4270-a5ab-8789ab0a2e89-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:09.473254 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:09.473221    4930 generic.go:356] "Generic (PLEG): container finished" podID="9165d842-245c-4270-a5ab-8789ab0a2e89" containerID="37ea7590884339c5454463228bc2325ec7d25f2daabc0802ce5d94f1bc4b3d4e" exitCode=0
Nov 01 03:34:09.473437 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:09.473271    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"
Nov 01 03:34:09.473437 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:09.473309    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8" event={"ID":"9165d842-245c-4270-a5ab-8789ab0a2e89","Type":"ContainerDied","Data":"37ea7590884339c5454463228bc2325ec7d25f2daabc0802ce5d94f1bc4b3d4e"}
Nov 01 03:34:09.473437 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:09.473366    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8" event={"ID":"9165d842-245c-4270-a5ab-8789ab0a2e89","Type":"ContainerDied","Data":"05683958c8e15ff743fc672041e2bf99a99a15cba82787962a5dd7a591d70f42"}
Nov 01 03:34:09.473437 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:09.473391    4930 scope.go:122] "RemoveContainer" containerID="37ea7590884339c5454463228bc2325ec7d25f2daabc0802ce5d94f1bc4b3d4e"
Nov 01 03:34:09.485258 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:09.485237    4930 scope.go:122] "RemoveContainer" containerID="f792968e49e2f52ea64875f94f3cb60fc433bea27522ddf2d426f5032482b857"
Nov 01 03:34:09.495076 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:09.495053    4930 scope.go:122] "RemoveContainer" containerID="37ea7590884339c5454463228bc2325ec7d25f2daabc0802ce5d94f1bc4b3d4e"
Nov 01 03:34:09.495374 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:34:09.495352    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37ea7590884339c5454463228bc2325ec7d25f2daabc0802ce5d94f1bc4b3d4e\": container with ID starting with 37ea7590884339c5454463228bc2325ec7d25f2daabc0802ce5d94f1bc4b3d4e not found: ID does not exist" containerID="37ea7590884339c5454463228bc2325ec7d25f2daabc0802ce5d94f1bc4b3d4e"
Nov 01 03:34:09.495414 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:09.495383    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37ea7590884339c5454463228bc2325ec7d25f2daabc0802ce5d94f1bc4b3d4e"} err="failed to get container status \"37ea7590884339c5454463228bc2325ec7d25f2daabc0802ce5d94f1bc4b3d4e\": rpc error: code = NotFound desc = could not find container \"37ea7590884339c5454463228bc2325ec7d25f2daabc0802ce5d94f1bc4b3d4e\": container with ID starting with 37ea7590884339c5454463228bc2325ec7d25f2daabc0802ce5d94f1bc4b3d4e not found: ID does not exist"
Nov 01 03:34:09.495414 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:09.495402    4930 scope.go:122] "RemoveContainer" containerID="f792968e49e2f52ea64875f94f3cb60fc433bea27522ddf2d426f5032482b857"
Nov 01 03:34:09.495696 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:34:09.495678    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f792968e49e2f52ea64875f94f3cb60fc433bea27522ddf2d426f5032482b857\": container with ID starting with f792968e49e2f52ea64875f94f3cb60fc433bea27522ddf2d426f5032482b857 not found: ID does not exist" containerID="f792968e49e2f52ea64875f94f3cb60fc433bea27522ddf2d426f5032482b857"
Nov 01 03:34:09.495745 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:09.495702    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f792968e49e2f52ea64875f94f3cb60fc433bea27522ddf2d426f5032482b857"} err="failed to get container status \"f792968e49e2f52ea64875f94f3cb60fc433bea27522ddf2d426f5032482b857\": rpc error: code = NotFound desc = could not find container \"f792968e49e2f52ea64875f94f3cb60fc433bea27522ddf2d426f5032482b857\": container with ID starting with f792968e49e2f52ea64875f94f3cb60fc433bea27522ddf2d426f5032482b857 not found: ID does not exist"
Nov 01 03:34:09.690616 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:09.690527    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"]
Nov 01 03:34:09.740481 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:09.740413    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-44379c05-openshift-default-c5bb5d5c-zs8b8"]
Nov 01 03:34:11.208683 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:11.208641    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:11.410745 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:11.410705    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:11.621535 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:11.621493    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9165d842-245c-4270-a5ab-8789ab0a2e89" path="/var/lib/kubelet/pods/9165d842-245c-4270-a5ab-8789ab0a2e89/volumes"
Nov 01 03:34:11.983804 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:11.983699    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:12.187412 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:12.187367    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:13.298252 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:13.298211    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:13.444238 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:13.444193    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:15.587080 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:15.587046    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:16.049854 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:16.049765    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3751a433-bz4tr"]
Nov 01 03:34:16.101570 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:16.101527    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3751a433-bz4tr"]
Nov 01 03:34:16.755409 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:16.755300    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:17.143039 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:17.140915    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:17.425913 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:17.425826    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:17.622542 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:17.622485    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b6239c6-f921-4896-b615-1206f64527bf" path="/var/lib/kubelet/pods/3b6239c6-f921-4896-b615-1206f64527bf/volumes"
Nov 01 03:34:17.662365 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:17.662329    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d262c155-9rls7"]
Nov 01 03:34:17.744665 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:17.744496    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d262c155-9rls7"]
Nov 01 03:34:17.769834 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:17.769641    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:18.506722 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:18.506681    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:18.653292 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:18.653255    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:18.804659 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:18.804571    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:18.957727 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:18.957686    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:19.055483 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.055410    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"]
Nov 01 03:34:19.056047 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056030    4930 cpu_manager.go:414] "containerMap: removing container" podUID="23ed3df4-075f-459b-be45-69da508ea513" containerName="istio-proxy"
Nov 01 03:34:19.056095 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056049    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23ed3df4-075f-459b-be45-69da508ea513" containerName="istio-proxy"
Nov 01 03:34:19.056095 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056065    4930 cpu_manager.go:414] "containerMap: removing container" podUID="27bad768-f9df-4649-a979-2a459327f07d" containerName="istio-proxy"
Nov 01 03:34:19.056095 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056070    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27bad768-f9df-4649-a979-2a459327f07d" containerName="istio-proxy"
Nov 01 03:34:19.056095 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056080    4930 cpu_manager.go:414] "containerMap: removing container" podUID="4b528dcb-1501-4764-bf26-d676a6e18474" containerName="istio-proxy"
Nov 01 03:34:19.056095 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056086    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b528dcb-1501-4764-bf26-d676a6e18474" containerName="istio-proxy"
Nov 01 03:34:19.056253 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056101    4930 cpu_manager.go:414] "containerMap: removing container" podUID="9cd5db87-0172-46ce-b856-cfc6366d7cb2" containerName="istio-proxy"
Nov 01 03:34:19.056253 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056106    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cd5db87-0172-46ce-b856-cfc6366d7cb2" containerName="istio-proxy"
Nov 01 03:34:19.056253 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056116    4930 cpu_manager.go:414] "containerMap: removing container" podUID="72c2d7cf-fac4-4d80-a5f1-82ae300a255a" containerName="istio-proxy"
Nov 01 03:34:19.056253 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056120    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72c2d7cf-fac4-4d80-a5f1-82ae300a255a" containerName="istio-proxy"
Nov 01 03:34:19.056253 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056130    4930 cpu_manager.go:414] "containerMap: removing container" podUID="f3c4cdb7-e686-4b71-9ab9-3c82f4c24648" containerName="istio-proxy"
Nov 01 03:34:19.056253 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056134    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3c4cdb7-e686-4b71-9ab9-3c82f4c24648" containerName="istio-proxy"
Nov 01 03:34:19.056253 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056141    4930 cpu_manager.go:414] "containerMap: removing container" podUID="58f8e6d8-aafa-4a82-b1e4-63c358380207" containerName="istio-proxy"
Nov 01 03:34:19.056253 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056145    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58f8e6d8-aafa-4a82-b1e4-63c358380207" containerName="istio-proxy"
Nov 01 03:34:19.056253 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056156    4930 cpu_manager.go:414] "containerMap: removing container" podUID="267415f5-02bf-4571-914a-45aa36ae41c9" containerName="istio-proxy"
Nov 01 03:34:19.056253 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056161    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="267415f5-02bf-4571-914a-45aa36ae41c9" containerName="istio-proxy"
Nov 01 03:34:19.056253 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056168    4930 cpu_manager.go:414] "containerMap: removing container" podUID="f3c4cdb7-e686-4b71-9ab9-3c82f4c24648" containerName="istio-proxy"
Nov 01 03:34:19.056253 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056172    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3c4cdb7-e686-4b71-9ab9-3c82f4c24648" containerName="istio-proxy"
Nov 01 03:34:19.056253 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056180    4930 cpu_manager.go:414] "containerMap: removing container" podUID="267415f5-02bf-4571-914a-45aa36ae41c9" containerName="istio-proxy"
Nov 01 03:34:19.056253 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056184    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="267415f5-02bf-4571-914a-45aa36ae41c9" containerName="istio-proxy"
Nov 01 03:34:19.056253 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056192    4930 cpu_manager.go:414] "containerMap: removing container" podUID="12d11ff3-9497-498d-8aeb-7f70f1874383" containerName="istio-proxy"
Nov 01 03:34:19.056253 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056196    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12d11ff3-9497-498d-8aeb-7f70f1874383" containerName="istio-proxy"
Nov 01 03:34:19.056253 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056202    4930 cpu_manager.go:414] "containerMap: removing container" podUID="9165d842-245c-4270-a5ab-8789ab0a2e89" containerName="istio-proxy"
Nov 01 03:34:19.056253 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056206    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9165d842-245c-4270-a5ab-8789ab0a2e89" containerName="istio-proxy"
Nov 01 03:34:19.056253 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056214    4930 cpu_manager.go:414] "containerMap: removing container" podUID="fd824f02-6c5a-4428-a995-80cbf1b20571" containerName="istio-proxy"
Nov 01 03:34:19.056253 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056218    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd824f02-6c5a-4428-a995-80cbf1b20571" containerName="istio-proxy"
Nov 01 03:34:19.056253 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056225    4930 cpu_manager.go:414] "containerMap: removing container" podUID="58f8e6d8-aafa-4a82-b1e4-63c358380207" containerName="istio-proxy"
Nov 01 03:34:19.056253 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056229    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58f8e6d8-aafa-4a82-b1e4-63c358380207" containerName="istio-proxy"
Nov 01 03:34:19.056253 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056238    4930 cpu_manager.go:414] "containerMap: removing container" podUID="9165d842-245c-4270-a5ab-8789ab0a2e89" containerName="istio-proxy"
Nov 01 03:34:19.056253 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056242    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9165d842-245c-4270-a5ab-8789ab0a2e89" containerName="istio-proxy"
Nov 01 03:34:19.057080 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056352    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="23ed3df4-075f-459b-be45-69da508ea513" containerName="istio-proxy"
Nov 01 03:34:19.057080 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056361    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="27bad768-f9df-4649-a979-2a459327f07d" containerName="istio-proxy"
Nov 01 03:34:19.057080 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056369    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b528dcb-1501-4764-bf26-d676a6e18474" containerName="istio-proxy"
Nov 01 03:34:19.057080 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056378    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cd5db87-0172-46ce-b856-cfc6366d7cb2" containerName="istio-proxy"
Nov 01 03:34:19.057080 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056383    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="72c2d7cf-fac4-4d80-a5f1-82ae300a255a" containerName="istio-proxy"
Nov 01 03:34:19.057080 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056390    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3c4cdb7-e686-4b71-9ab9-3c82f4c24648" containerName="istio-proxy"
Nov 01 03:34:19.057080 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056395    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="58f8e6d8-aafa-4a82-b1e4-63c358380207" containerName="istio-proxy"
Nov 01 03:34:19.057080 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056401    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="267415f5-02bf-4571-914a-45aa36ae41c9" containerName="istio-proxy"
Nov 01 03:34:19.057080 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056408    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3c4cdb7-e686-4b71-9ab9-3c82f4c24648" containerName="istio-proxy"
Nov 01 03:34:19.057080 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056414    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="267415f5-02bf-4571-914a-45aa36ae41c9" containerName="istio-proxy"
Nov 01 03:34:19.057080 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056418    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="12d11ff3-9497-498d-8aeb-7f70f1874383" containerName="istio-proxy"
Nov 01 03:34:19.057080 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056423    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="9165d842-245c-4270-a5ab-8789ab0a2e89" containerName="istio-proxy"
Nov 01 03:34:19.057080 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056430    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd824f02-6c5a-4428-a995-80cbf1b20571" containerName="istio-proxy"
Nov 01 03:34:19.057080 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056440    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="58f8e6d8-aafa-4a82-b1e4-63c358380207" containerName="istio-proxy"
Nov 01 03:34:19.057080 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.056447    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="9165d842-245c-4270-a5ab-8789ab0a2e89" containerName="istio-proxy"
Nov 01 03:34:19.061378 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.061355    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"
Nov 01 03:34:19.099758 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.099709    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4cae562-15e5-4d0b-bcef-62ca774e2369-credential-socket\") pod \"e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh\" (UID: \"a4cae562-15e5-4d0b-bcef-62ca774e2369\") " pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"
Nov 01 03:34:19.099933 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.099769    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4cae562-15e5-4d0b-bcef-62ca774e2369-workload-socket\") pod \"e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh\" (UID: \"a4cae562-15e5-4d0b-bcef-62ca774e2369\") " pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"
Nov 01 03:34:19.099933 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.099857    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4cae562-15e5-4d0b-bcef-62ca774e2369-istio-envoy\") pod \"e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh\" (UID: \"a4cae562-15e5-4d0b-bcef-62ca774e2369\") " pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"
Nov 01 03:34:19.099933 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.099910    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrtjk\" (UniqueName: \"kubernetes.io/projected/a4cae562-15e5-4d0b-bcef-62ca774e2369-kube-api-access-nrtjk\") pod \"e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh\" (UID: \"a4cae562-15e5-4d0b-bcef-62ca774e2369\") " pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"
Nov 01 03:34:19.100095 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.099985    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4cae562-15e5-4d0b-bcef-62ca774e2369-istio-podinfo\") pod \"e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh\" (UID: \"a4cae562-15e5-4d0b-bcef-62ca774e2369\") " pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"
Nov 01 03:34:19.100095 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.100031    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4cae562-15e5-4d0b-bcef-62ca774e2369-istio-token\") pod \"e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh\" (UID: \"a4cae562-15e5-4d0b-bcef-62ca774e2369\") " pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"
Nov 01 03:34:19.100095 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.100062    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4cae562-15e5-4d0b-bcef-62ca774e2369-istiod-ca-cert\") pod \"e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh\" (UID: \"a4cae562-15e5-4d0b-bcef-62ca774e2369\") " pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"
Nov 01 03:34:19.100459 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.100144    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4cae562-15e5-4d0b-bcef-62ca774e2369-workload-certs\") pod \"e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh\" (UID: \"a4cae562-15e5-4d0b-bcef-62ca774e2369\") " pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"
Nov 01 03:34:19.100459 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.100247    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4cae562-15e5-4d0b-bcef-62ca774e2369-istio-data\") pod \"e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh\" (UID: \"a4cae562-15e5-4d0b-bcef-62ca774e2369\") " pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"
Nov 01 03:34:19.123423 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.123037    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-e8c77d-openshift-default-dockercfg-tkps2\""
Nov 01 03:34:19.123423 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.123094    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3513c2f1-dpfxx"]
Nov 01 03:34:19.146636 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.142263    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:19.206139 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.206052    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3513c2f1-dpfxx"]
Nov 01 03:34:19.223065 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.223024    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4cae562-15e5-4d0b-bcef-62ca774e2369-workload-certs\") pod \"e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh\" (UID: \"a4cae562-15e5-4d0b-bcef-62ca774e2369\") " pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"
Nov 01 03:34:19.255433 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:34:19.240836    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:34:19.255433 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.241850    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"]
Nov 01 03:34:19.286577 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.261074    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4cae562-15e5-4d0b-bcef-62ca774e2369-workload-certs\") pod \"e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh\" (UID: \"a4cae562-15e5-4d0b-bcef-62ca774e2369\") " pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"
Nov 01 03:34:19.306626 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.293648    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4cae562-15e5-4d0b-bcef-62ca774e2369-istio-data\") pod \"e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh\" (UID: \"a4cae562-15e5-4d0b-bcef-62ca774e2369\") " pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"
Nov 01 03:34:19.306626 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.293768    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4cae562-15e5-4d0b-bcef-62ca774e2369-credential-socket\") pod \"e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh\" (UID: \"a4cae562-15e5-4d0b-bcef-62ca774e2369\") " pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"
Nov 01 03:34:19.306626 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.293810    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4cae562-15e5-4d0b-bcef-62ca774e2369-workload-socket\") pod \"e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh\" (UID: \"a4cae562-15e5-4d0b-bcef-62ca774e2369\") " pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"
Nov 01 03:34:19.306626 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.294207    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4cae562-15e5-4d0b-bcef-62ca774e2369-credential-socket\") pod \"e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh\" (UID: \"a4cae562-15e5-4d0b-bcef-62ca774e2369\") " pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"
Nov 01 03:34:19.306626 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.294519    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4cae562-15e5-4d0b-bcef-62ca774e2369-workload-socket\") pod \"e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh\" (UID: \"a4cae562-15e5-4d0b-bcef-62ca774e2369\") " pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"
Nov 01 03:34:19.306626 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.294767    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4cae562-15e5-4d0b-bcef-62ca774e2369-istio-data\") pod \"e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh\" (UID: \"a4cae562-15e5-4d0b-bcef-62ca774e2369\") " pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"
Nov 01 03:34:19.306626 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.294890    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4cae562-15e5-4d0b-bcef-62ca774e2369-istio-envoy\") pod \"e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh\" (UID: \"a4cae562-15e5-4d0b-bcef-62ca774e2369\") " pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"
Nov 01 03:34:19.306626 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.294960    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrtjk\" (UniqueName: \"kubernetes.io/projected/a4cae562-15e5-4d0b-bcef-62ca774e2369-kube-api-access-nrtjk\") pod \"e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh\" (UID: \"a4cae562-15e5-4d0b-bcef-62ca774e2369\") " pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"
Nov 01 03:34:19.306626 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.295889    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4cae562-15e5-4d0b-bcef-62ca774e2369-istio-podinfo\") pod \"e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh\" (UID: \"a4cae562-15e5-4d0b-bcef-62ca774e2369\") " pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"
Nov 01 03:34:19.306626 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.296329    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4cae562-15e5-4d0b-bcef-62ca774e2369-istio-token\") pod \"e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh\" (UID: \"a4cae562-15e5-4d0b-bcef-62ca774e2369\") " pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"
Nov 01 03:34:19.306626 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.296671    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4cae562-15e5-4d0b-bcef-62ca774e2369-istiod-ca-cert\") pod \"e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh\" (UID: \"a4cae562-15e5-4d0b-bcef-62ca774e2369\") " pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"
Nov 01 03:34:19.306626 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.297365    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4cae562-15e5-4d0b-bcef-62ca774e2369-istiod-ca-cert\") pod \"e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh\" (UID: \"a4cae562-15e5-4d0b-bcef-62ca774e2369\") " pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"
Nov 01 03:34:19.306626 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.300744    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:19.306626 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.304159    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4cae562-15e5-4d0b-bcef-62ca774e2369-istio-podinfo\") pod \"e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh\" (UID: \"a4cae562-15e5-4d0b-bcef-62ca774e2369\") " pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"
Nov 01 03:34:19.318336 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.317472    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"]
Nov 01 03:34:19.318336 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.317739    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm" podUID="8deab213-f1c7-4883-9a64-72789a741faf" containerName="istio-proxy" containerID="cri-o://682dc443e16ae0a10a65396ab37473b089b9c19e7228bd4a6a0470ff4010939a" gracePeriod=30
Nov 01 03:34:19.319922 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.319816    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4cae562-15e5-4d0b-bcef-62ca774e2369-istio-envoy\") pod \"e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh\" (UID: \"a4cae562-15e5-4d0b-bcef-62ca774e2369\") " pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"
Nov 01 03:34:19.366985 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.366921    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4cae562-15e5-4d0b-bcef-62ca774e2369-istio-token\") pod \"e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh\" (UID: \"a4cae562-15e5-4d0b-bcef-62ca774e2369\") " pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"
Nov 01 03:34:19.367201 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.367172    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrtjk\" (UniqueName: \"kubernetes.io/projected/a4cae562-15e5-4d0b-bcef-62ca774e2369-kube-api-access-nrtjk\") pod \"e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh\" (UID: \"a4cae562-15e5-4d0b-bcef-62ca774e2369\") " pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"
Nov 01 03:34:19.389431 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.389291    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"
Nov 01 03:34:19.647376 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.647341    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="414a2706-7d57-4edb-a3fd-8946b9d07729" path="/var/lib/kubelet/pods/414a2706-7d57-4edb-a3fd-8946b9d07729/volumes"
Nov 01 03:34:19.649267 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.649236    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:19.649639 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.649618    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b77e2e1-bc9d-4451-93eb-66dddf7b0f74" path="/var/lib/kubelet/pods/6b77e2e1-bc9d-4451-93eb-66dddf7b0f74/volumes"
Nov 01 03:34:19.898006 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.897978    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"]
Nov 01 03:34:19.900455 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:34:19.900425    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4cae562_15e5_4d0b_bcef_62ca774e2369.slice/crio-3158af2c7942a3a4bd0f801909f024c67da30e6e6a6d51d25d84e405cef82451 WatchSource:0}: Error finding container 3158af2c7942a3a4bd0f801909f024c67da30e6e6a6d51d25d84e405cef82451: Status 404 returned error can't find the container with id 3158af2c7942a3a4bd0f801909f024c67da30e6e6a6d51d25d84e405cef82451
Nov 01 03:34:19.903948 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.903910    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:34:19.904049 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.903990    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:34:19.904049 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.904037    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:34:19.954604 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:19.954568    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:20.438956 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.438920    4930 patch_prober.go:29] interesting pod/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:34:20.439158 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.438984    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm" podUID="8deab213-f1c7-4883-9a64-72789a741faf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:34:20.506578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.506139    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afab651a-bvr2q"]
Nov 01 03:34:20.512154 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.512128    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afab651a-bvr2q"
Nov 01 03:34:20.567587 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.564568    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:34:20.576635 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.576602    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"]
Nov 01 03:34:20.583369 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.583346    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"
Nov 01 03:34:20.589780 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.589758    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afab651a-bvr2q"]
Nov 01 03:34:20.605854 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.604836    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:34:20.605854 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.604978    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:34:20.605854 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.605074    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:34:20.661093 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.661055    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-credential-socket\") pod \"e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz\" (UID: \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\") " pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"
Nov 01 03:34:20.679693 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.679653    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-istio-data\") pod \"e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz\" (UID: \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\") " pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"
Nov 01 03:34:20.679941 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.679923    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab67eed1-b327-4933-ac53-fd2b8c6e7a66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afab651a-bvr2q\" (UID: \"ab67eed1-b327-4933-ac53-fd2b8c6e7a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afab651a-bvr2q"
Nov 01 03:34:20.680104 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.680091    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-istio-envoy\") pod \"e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz\" (UID: \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\") " pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"
Nov 01 03:34:20.680244 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.680223    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-istiod-ca-cert\") pod \"e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz\" (UID: \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\") " pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"
Nov 01 03:34:20.680392 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.680376    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j42l5\" (UniqueName: \"kubernetes.io/projected/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-kube-api-access-j42l5\") pod \"e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz\" (UID: \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\") " pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"
Nov 01 03:34:20.680523 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.680510    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-workload-socket\") pod \"e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz\" (UID: \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\") " pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"
Nov 01 03:34:20.680782 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.680702    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-istio-podinfo\") pod \"e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz\" (UID: \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\") " pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"
Nov 01 03:34:20.681252 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.681232    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt2n7\" (UniqueName: \"kubernetes.io/projected/ab67eed1-b327-4933-ac53-fd2b8c6e7a66-kube-api-access-pt2n7\") pod \"maas-api-revoke-sub-afab651a-bvr2q\" (UID: \"ab67eed1-b327-4933-ac53-fd2b8c6e7a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afab651a-bvr2q"
Nov 01 03:34:20.698756 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.698672    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab67eed1-b327-4933-ac53-fd2b8c6e7a66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afab651a-bvr2q\" (UID: \"ab67eed1-b327-4933-ac53-fd2b8c6e7a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afab651a-bvr2q"
Nov 01 03:34:20.702717 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.702689    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-istio-token\") pod \"e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz\" (UID: \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\") " pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"
Nov 01 03:34:20.706750 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.702896    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-workload-certs\") pod \"e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz\" (UID: \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\") " pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"
Nov 01 03:34:20.707108 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.699088    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-0cb10b-openshift-default-dockercfg-l2r47\""
Nov 01 03:34:20.750948 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.723631    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:20.777275 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.777142    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"]
Nov 01 03:34:20.789900 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.789863    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh" event={"ID":"a4cae562-15e5-4d0b-bcef-62ca774e2369","Type":"ContainerStarted","Data":"de5005bcce6c56f7eea7f9d783b30dcf09982aa9a56cc44a65a2306b807a57be"}
Nov 01 03:34:20.790092 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.789926    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh" event={"ID":"a4cae562-15e5-4d0b-bcef-62ca774e2369","Type":"ContainerStarted","Data":"3158af2c7942a3a4bd0f801909f024c67da30e6e6a6d51d25d84e405cef82451"}
Nov 01 03:34:20.853825 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.853791    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-istio-token\") pod \"e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz\" (UID: \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\") " pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"
Nov 01 03:34:20.854027 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.853952    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-workload-certs\") pod \"e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz\" (UID: \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\") " pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"
Nov 01 03:34:20.854104 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.854031    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-credential-socket\") pod \"e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz\" (UID: \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\") " pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"
Nov 01 03:34:20.854165 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.854099    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-istio-data\") pod \"e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz\" (UID: \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\") " pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"
Nov 01 03:34:20.856977 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.854166    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab67eed1-b327-4933-ac53-fd2b8c6e7a66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afab651a-bvr2q\" (UID: \"ab67eed1-b327-4933-ac53-fd2b8c6e7a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afab651a-bvr2q"
Nov 01 03:34:20.856977 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.854351    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-istio-envoy\") pod \"e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz\" (UID: \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\") " pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"
Nov 01 03:34:20.856977 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.854426    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-istiod-ca-cert\") pod \"e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz\" (UID: \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\") " pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"
Nov 01 03:34:20.856977 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.854518    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j42l5\" (UniqueName: \"kubernetes.io/projected/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-kube-api-access-j42l5\") pod \"e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz\" (UID: \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\") " pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"
Nov 01 03:34:20.856977 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.854786    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-credential-socket\") pod \"e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz\" (UID: \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\") " pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"
Nov 01 03:34:20.856977 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.855146    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-workload-socket\") pod \"e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz\" (UID: \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\") " pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"
Nov 01 03:34:20.856977 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.855185    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-istio-podinfo\") pod \"e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz\" (UID: \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\") " pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"
Nov 01 03:34:20.856977 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.855416    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-istio-data\") pod \"e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz\" (UID: \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\") " pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"
Nov 01 03:34:20.856977 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.855507    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt2n7\" (UniqueName: \"kubernetes.io/projected/ab67eed1-b327-4933-ac53-fd2b8c6e7a66-kube-api-access-pt2n7\") pod \"maas-api-revoke-sub-afab651a-bvr2q\" (UID: \"ab67eed1-b327-4933-ac53-fd2b8c6e7a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afab651a-bvr2q"
Nov 01 03:34:20.856977 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.856032    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab67eed1-b327-4933-ac53-fd2b8c6e7a66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afab651a-bvr2q\" (UID: \"ab67eed1-b327-4933-ac53-fd2b8c6e7a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afab651a-bvr2q"
Nov 01 03:34:20.856977 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.856702    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab67eed1-b327-4933-ac53-fd2b8c6e7a66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afab651a-bvr2q\" (UID: \"ab67eed1-b327-4933-ac53-fd2b8c6e7a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afab651a-bvr2q"
Nov 01 03:34:20.856977 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.856947    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-workload-certs\") pod \"e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz\" (UID: \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\") " pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"
Nov 01 03:34:20.857846 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.857587    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-istiod-ca-cert\") pod \"e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz\" (UID: \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\") " pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"
Nov 01 03:34:20.857846 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.857833    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-workload-socket\") pod \"e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz\" (UID: \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\") " pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"
Nov 01 03:34:20.884064 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.883951    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-istio-envoy\") pod \"e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz\" (UID: \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\") " pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"
Nov 01 03:34:20.888243 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.887508    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab67eed1-b327-4933-ac53-fd2b8c6e7a66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afab651a-bvr2q\" (UID: \"ab67eed1-b327-4933-ac53-fd2b8c6e7a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afab651a-bvr2q"
Nov 01 03:34:20.889253 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.889230    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-istio-podinfo\") pod \"e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz\" (UID: \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\") " pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"
Nov 01 03:34:20.915378 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.915343    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-istio-token\") pod \"e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz\" (UID: \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\") " pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"
Nov 01 03:34:20.924053 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.924021    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt2n7\" (UniqueName: \"kubernetes.io/projected/ab67eed1-b327-4933-ac53-fd2b8c6e7a66-kube-api-access-pt2n7\") pod \"maas-api-revoke-sub-afab651a-bvr2q\" (UID: \"ab67eed1-b327-4933-ac53-fd2b8c6e7a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afab651a-bvr2q"
Nov 01 03:34:20.924674 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.924650    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j42l5\" (UniqueName: \"kubernetes.io/projected/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-kube-api-access-j42l5\") pod \"e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz\" (UID: \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\") " pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"
Nov 01 03:34:20.969653 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:20.969508    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh" podStartSLOduration=1.96949251 podStartE2EDuration="1.96949251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:34:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:34:20.965205937 +0000 UTC m=+1665.488212527" watchObservedRunningTime="2026-11-01 03:34:20.96949251 +0000 UTC m=+1665.492499097"
Nov 01 03:34:21.056894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:21.056865    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"
Nov 01 03:34:21.060122 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:21.060088    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:21.134580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:21.129289    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afab651a-bvr2q"
Nov 01 03:34:21.390681 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:21.390648    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"
Nov 01 03:34:21.396404 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:21.396378    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"
Nov 01 03:34:21.521494 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:21.521339    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"]
Nov 01 03:34:21.522204 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:34:21.522174    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecf14a6b_aa67_4d3b_be7a_63b65eaa51e6.slice/crio-82e5619b8968817fabfffb49ccf3c7a50bbf046b62506a2237bde8677d9acb92 WatchSource:0}: Error finding container 82e5619b8968817fabfffb49ccf3c7a50bbf046b62506a2237bde8677d9acb92: Status 404 returned error can't find the container with id 82e5619b8968817fabfffb49ccf3c7a50bbf046b62506a2237bde8677d9acb92
Nov 01 03:34:21.525160 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:21.525123    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:34:21.525250 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:21.525202    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:34:21.525250 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:21.525244    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:34:21.573155 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:34:21.573125    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab67eed1_b327_4933_ac53_fd2b8c6e7a66.slice/crio-654466cee3f91db5d0e0794adc23c303f0e1045f43ece1e60ae50fafa0be14f7 WatchSource:0}: Error finding container 654466cee3f91db5d0e0794adc23c303f0e1045f43ece1e60ae50fafa0be14f7: Status 404 returned error can't find the container with id 654466cee3f91db5d0e0794adc23c303f0e1045f43ece1e60ae50fafa0be14f7
Nov 01 03:34:21.579020 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:21.578997    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afab651a-bvr2q"]
Nov 01 03:34:21.809756 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:21.809715    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afab651a-bvr2q" event={"ID":"ab67eed1-b327-4933-ac53-fd2b8c6e7a66","Type":"ContainerStarted","Data":"654466cee3f91db5d0e0794adc23c303f0e1045f43ece1e60ae50fafa0be14f7"}
Nov 01 03:34:21.812862 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:21.812833    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz" event={"ID":"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6","Type":"ContainerStarted","Data":"34dc61e7d67b7c666d5691fecdf5db6b2f65960eaa7d26ce8c08582ae6e53bb6"}
Nov 01 03:34:21.813005 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:21.812882    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz" event={"ID":"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6","Type":"ContainerStarted","Data":"82e5619b8968817fabfffb49ccf3c7a50bbf046b62506a2237bde8677d9acb92"}
Nov 01 03:34:21.813005 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:21.812914    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"
Nov 01 03:34:21.814053 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:21.814030    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"
Nov 01 03:34:21.934678 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:21.934615    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz" podStartSLOduration=1.934590534 podStartE2EDuration="1.934590534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:34:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:34:21.933825477 +0000 UTC m=+1666.456832066" watchObservedRunningTime="2026-11-01 03:34:21.934590534 +0000 UTC m=+1666.457597122"
Nov 01 03:34:22.059687 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:22.059522    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"
Nov 01 03:34:22.094534 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:22.094193    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"
Nov 01 03:34:22.877661 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:22.876828    4930 generic.go:356] "Generic (PLEG): container finished" podID="ab67eed1-b327-4933-ac53-fd2b8c6e7a66" containerID="713cd81d0c64e18c01734345c7b21605bcb695d3f54ff1e8c04683ac9cb17abb" exitCode=0
Nov 01 03:34:22.877661 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:22.877040    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afab651a-bvr2q" event={"ID":"ab67eed1-b327-4933-ac53-fd2b8c6e7a66","Type":"ContainerDied","Data":"713cd81d0c64e18c01734345c7b21605bcb695d3f54ff1e8c04683ac9cb17abb"}
Nov 01 03:34:22.877953 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:22.877933    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"
Nov 01 03:34:22.881238 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:22.881215    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"
Nov 01 03:34:23.906081 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:23.906043    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:24.087759 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.087719    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afab651a-bvr2q"
Nov 01 03:34:24.178717 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.178158    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:24.252383 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.252346    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab67eed1-b327-4933-ac53-fd2b8c6e7a66-kube-api-access-pt2n7\" (UniqueName: \"kubernetes.io/projected/ab67eed1-b327-4933-ac53-fd2b8c6e7a66-kube-api-access-pt2n7\") pod \"ab67eed1-b327-4933-ac53-fd2b8c6e7a66\" (UID: \"ab67eed1-b327-4933-ac53-fd2b8c6e7a66\") "
Nov 01 03:34:24.252549 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.252451    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab67eed1-b327-4933-ac53-fd2b8c6e7a66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab67eed1-b327-4933-ac53-fd2b8c6e7a66-maas-api-xks-ca\") pod \"ab67eed1-b327-4933-ac53-fd2b8c6e7a66\" (UID: \"ab67eed1-b327-4933-ac53-fd2b8c6e7a66\") "
Nov 01 03:34:24.252725 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.252709    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab67eed1-b327-4933-ac53-fd2b8c6e7a66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab67eed1-b327-4933-ac53-fd2b8c6e7a66-maas-api-openshift-ca\") pod \"ab67eed1-b327-4933-ac53-fd2b8c6e7a66\" (UID: \"ab67eed1-b327-4933-ac53-fd2b8c6e7a66\") "
Nov 01 03:34:24.254798 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.254770    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab67eed1-b327-4933-ac53-fd2b8c6e7a66-maas-api-openshift-ca" pod "ab67eed1-b327-4933-ac53-fd2b8c6e7a66" (UID: "ab67eed1-b327-4933-ac53-fd2b8c6e7a66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:34:24.256332 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.256306    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab67eed1-b327-4933-ac53-fd2b8c6e7a66-kube-api-access-pt2n7" pod "ab67eed1-b327-4933-ac53-fd2b8c6e7a66" (UID: "ab67eed1-b327-4933-ac53-fd2b8c6e7a66"). InnerVolumeSpecName "kube-api-access-pt2n7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:34:24.269869 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.269809    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab67eed1-b327-4933-ac53-fd2b8c6e7a66-maas-api-xks-ca" pod "ab67eed1-b327-4933-ac53-fd2b8c6e7a66" (UID: "ab67eed1-b327-4933-ac53-fd2b8c6e7a66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:34:24.362030 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.361976    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt2n7\" (UniqueName: \"kubernetes.io/projected/ab67eed1-b327-4933-ac53-fd2b8c6e7a66-kube-api-access-pt2n7\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:24.362030 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.362021    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab67eed1-b327-4933-ac53-fd2b8c6e7a66-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:24.362302 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.362042    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab67eed1-b327-4933-ac53-fd2b8c6e7a66-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:24.368499 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.368458    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:24.663723 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.663698    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"
Nov 01 03:34:24.677984 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.677953    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:24.769938 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.769862    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8deab213-f1c7-4883-9a64-72789a741faf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8deab213-f1c7-4883-9a64-72789a741faf-workload-certs\") pod \"8deab213-f1c7-4883-9a64-72789a741faf\" (UID: \"8deab213-f1c7-4883-9a64-72789a741faf\") "
Nov 01 03:34:24.769938 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.769920    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8deab213-f1c7-4883-9a64-72789a741faf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8deab213-f1c7-4883-9a64-72789a741faf-istio-podinfo\") pod \"8deab213-f1c7-4883-9a64-72789a741faf\" (UID: \"8deab213-f1c7-4883-9a64-72789a741faf\") "
Nov 01 03:34:24.770148 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.769958    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8deab213-f1c7-4883-9a64-72789a741faf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8deab213-f1c7-4883-9a64-72789a741faf-istio-data\") pod \"8deab213-f1c7-4883-9a64-72789a741faf\" (UID: \"8deab213-f1c7-4883-9a64-72789a741faf\") "
Nov 01 03:34:24.770148 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.770025    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8deab213-f1c7-4883-9a64-72789a741faf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8deab213-f1c7-4883-9a64-72789a741faf-workload-socket\") pod \"8deab213-f1c7-4883-9a64-72789a741faf\" (UID: \"8deab213-f1c7-4883-9a64-72789a741faf\") "
Nov 01 03:34:24.770261 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.770147    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8deab213-f1c7-4883-9a64-72789a741faf-workload-certs" pod "8deab213-f1c7-4883-9a64-72789a741faf" (UID: "8deab213-f1c7-4883-9a64-72789a741faf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:24.770261 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.770183    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8deab213-f1c7-4883-9a64-72789a741faf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8deab213-f1c7-4883-9a64-72789a741faf-istiod-ca-cert\") pod \"8deab213-f1c7-4883-9a64-72789a741faf\" (UID: \"8deab213-f1c7-4883-9a64-72789a741faf\") "
Nov 01 03:34:24.770374 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.770273    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8deab213-f1c7-4883-9a64-72789a741faf-kube-api-access-wm5lb\" (UniqueName: \"kubernetes.io/projected/8deab213-f1c7-4883-9a64-72789a741faf-kube-api-access-wm5lb\") pod \"8deab213-f1c7-4883-9a64-72789a741faf\" (UID: \"8deab213-f1c7-4883-9a64-72789a741faf\") "
Nov 01 03:34:24.770374 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.770293    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8deab213-f1c7-4883-9a64-72789a741faf-istio-data" pod "8deab213-f1c7-4883-9a64-72789a741faf" (UID: "8deab213-f1c7-4883-9a64-72789a741faf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:24.770374 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.770339    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8deab213-f1c7-4883-9a64-72789a741faf-istio-token\" (UniqueName: \"kubernetes.io/projected/8deab213-f1c7-4883-9a64-72789a741faf-istio-token\") pod \"8deab213-f1c7-4883-9a64-72789a741faf\" (UID: \"8deab213-f1c7-4883-9a64-72789a741faf\") "
Nov 01 03:34:24.770521 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.770383    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8deab213-f1c7-4883-9a64-72789a741faf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8deab213-f1c7-4883-9a64-72789a741faf-istio-envoy\") pod \"8deab213-f1c7-4883-9a64-72789a741faf\" (UID: \"8deab213-f1c7-4883-9a64-72789a741faf\") "
Nov 01 03:34:24.770521 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.770383    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8deab213-f1c7-4883-9a64-72789a741faf-workload-socket" pod "8deab213-f1c7-4883-9a64-72789a741faf" (UID: "8deab213-f1c7-4883-9a64-72789a741faf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:24.770521 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.770444    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8deab213-f1c7-4883-9a64-72789a741faf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8deab213-f1c7-4883-9a64-72789a741faf-credential-socket\") pod \"8deab213-f1c7-4883-9a64-72789a741faf\" (UID: \"8deab213-f1c7-4883-9a64-72789a741faf\") "
Nov 01 03:34:24.770748 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.770531    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8deab213-f1c7-4883-9a64-72789a741faf-istiod-ca-cert" pod "8deab213-f1c7-4883-9a64-72789a741faf" (UID: "8deab213-f1c7-4883-9a64-72789a741faf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:34:24.770807 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.770782    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8deab213-f1c7-4883-9a64-72789a741faf-credential-socket" pod "8deab213-f1c7-4883-9a64-72789a741faf" (UID: "8deab213-f1c7-4883-9a64-72789a741faf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:24.771355 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.771326    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8deab213-f1c7-4883-9a64-72789a741faf-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:24.771476 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.771357    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8deab213-f1c7-4883-9a64-72789a741faf-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:24.771476 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.771373    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8deab213-f1c7-4883-9a64-72789a741faf-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:24.771476 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.771389    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8deab213-f1c7-4883-9a64-72789a741faf-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:24.771476 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.771404    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8deab213-f1c7-4883-9a64-72789a741faf-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:24.772403 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.772380    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8deab213-f1c7-4883-9a64-72789a741faf-istio-podinfo" pod "8deab213-f1c7-4883-9a64-72789a741faf" (UID: "8deab213-f1c7-4883-9a64-72789a741faf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:34:24.772596 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.772564    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8deab213-f1c7-4883-9a64-72789a741faf-istio-token" pod "8deab213-f1c7-4883-9a64-72789a741faf" (UID: "8deab213-f1c7-4883-9a64-72789a741faf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:34:24.773026 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.773005    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8deab213-f1c7-4883-9a64-72789a741faf-istio-envoy" pod "8deab213-f1c7-4883-9a64-72789a741faf" (UID: "8deab213-f1c7-4883-9a64-72789a741faf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:24.773088 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.773047    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8deab213-f1c7-4883-9a64-72789a741faf-kube-api-access-wm5lb" pod "8deab213-f1c7-4883-9a64-72789a741faf" (UID: "8deab213-f1c7-4883-9a64-72789a741faf"). InnerVolumeSpecName "kube-api-access-wm5lb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:34:24.857902 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.857865    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:24.872789 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.872761    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm5lb\" (UniqueName: \"kubernetes.io/projected/8deab213-f1c7-4883-9a64-72789a741faf-kube-api-access-wm5lb\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:24.872789 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.872788    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8deab213-f1c7-4883-9a64-72789a741faf-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:24.872942 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.872799    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8deab213-f1c7-4883-9a64-72789a741faf-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:24.872942 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.872808    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8deab213-f1c7-4883-9a64-72789a741faf-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:24.896296 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.896272    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afab651a-bvr2q"
Nov 01 03:34:24.896296 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.896278    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afab651a-bvr2q" event={"ID":"ab67eed1-b327-4933-ac53-fd2b8c6e7a66","Type":"ContainerDied","Data":"654466cee3f91db5d0e0794adc23c303f0e1045f43ece1e60ae50fafa0be14f7"}
Nov 01 03:34:24.896443 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.896319    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="654466cee3f91db5d0e0794adc23c303f0e1045f43ece1e60ae50fafa0be14f7"
Nov 01 03:34:24.898291 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.898264    4930 generic.go:356] "Generic (PLEG): container finished" podID="8deab213-f1c7-4883-9a64-72789a741faf" containerID="682dc443e16ae0a10a65396ab37473b089b9c19e7228bd4a6a0470ff4010939a" exitCode=0
Nov 01 03:34:24.898405 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.898297    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"
Nov 01 03:34:24.898405 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.898341    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm" event={"ID":"8deab213-f1c7-4883-9a64-72789a741faf","Type":"ContainerDied","Data":"682dc443e16ae0a10a65396ab37473b089b9c19e7228bd4a6a0470ff4010939a"}
Nov 01 03:34:24.898405 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.898384    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm" event={"ID":"8deab213-f1c7-4883-9a64-72789a741faf","Type":"ContainerDied","Data":"74c28ce75257b1a93aa9fd70b5776b7c49c6c989d51b4d438c7abe24c375ffec"}
Nov 01 03:34:24.898534 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.898408    4930 scope.go:122] "RemoveContainer" containerID="682dc443e16ae0a10a65396ab37473b089b9c19e7228bd4a6a0470ff4010939a"
Nov 01 03:34:24.912452 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.912429    4930 scope.go:122] "RemoveContainer" containerID="7be529d70abaa4f033fb767519510b1c2e98a21cbf5352fe59460d5b1aaebcde"
Nov 01 03:34:24.926241 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.926222    4930 scope.go:122] "RemoveContainer" containerID="682dc443e16ae0a10a65396ab37473b089b9c19e7228bd4a6a0470ff4010939a"
Nov 01 03:34:24.926537 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:34:24.926517    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"682dc443e16ae0a10a65396ab37473b089b9c19e7228bd4a6a0470ff4010939a\": container with ID starting with 682dc443e16ae0a10a65396ab37473b089b9c19e7228bd4a6a0470ff4010939a not found: ID does not exist" containerID="682dc443e16ae0a10a65396ab37473b089b9c19e7228bd4a6a0470ff4010939a"
Nov 01 03:34:24.926642 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.926544    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"682dc443e16ae0a10a65396ab37473b089b9c19e7228bd4a6a0470ff4010939a"} err="failed to get container status \"682dc443e16ae0a10a65396ab37473b089b9c19e7228bd4a6a0470ff4010939a\": rpc error: code = NotFound desc = could not find container \"682dc443e16ae0a10a65396ab37473b089b9c19e7228bd4a6a0470ff4010939a\": container with ID starting with 682dc443e16ae0a10a65396ab37473b089b9c19e7228bd4a6a0470ff4010939a not found: ID does not exist"
Nov 01 03:34:24.926642 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.926582    4930 scope.go:122] "RemoveContainer" containerID="7be529d70abaa4f033fb767519510b1c2e98a21cbf5352fe59460d5b1aaebcde"
Nov 01 03:34:24.926865 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:34:24.926842    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7be529d70abaa4f033fb767519510b1c2e98a21cbf5352fe59460d5b1aaebcde\": container with ID starting with 7be529d70abaa4f033fb767519510b1c2e98a21cbf5352fe59460d5b1aaebcde not found: ID does not exist" containerID="7be529d70abaa4f033fb767519510b1c2e98a21cbf5352fe59460d5b1aaebcde"
Nov 01 03:34:24.926908 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:24.926873    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7be529d70abaa4f033fb767519510b1c2e98a21cbf5352fe59460d5b1aaebcde"} err="failed to get container status \"7be529d70abaa4f033fb767519510b1c2e98a21cbf5352fe59460d5b1aaebcde\": rpc error: code = NotFound desc = could not find container \"7be529d70abaa4f033fb767519510b1c2e98a21cbf5352fe59460d5b1aaebcde\": container with ID starting with 7be529d70abaa4f033fb767519510b1c2e98a21cbf5352fe59460d5b1aaebcde not found: ID does not exist"
Nov 01 03:34:25.063421 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:25.063391    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"]
Nov 01 03:34:25.064435 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:25.064409    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:25.067699 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:25.067677    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-3513c2f1-openshift-default-7f984548fc-gjpkm"]
Nov 01 03:34:25.622625 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:25.622585    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8deab213-f1c7-4883-9a64-72789a741faf" path="/var/lib/kubelet/pods/8deab213-f1c7-4883-9a64-72789a741faf/volumes"
Nov 01 03:34:27.036495 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:27.036461    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:27.258876 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:27.258839    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:27.464093 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:27.464055    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e8c77d-68444c5d47-bj8bs"]
Nov 01 03:34:27.465324 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:27.465023    4930 cpu_manager.go:414] "containerMap: removing container" podUID="8deab213-f1c7-4883-9a64-72789a741faf" containerName="istio-proxy"
Nov 01 03:34:27.465324 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:27.465048    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8deab213-f1c7-4883-9a64-72789a741faf" containerName="istio-proxy"
Nov 01 03:34:27.465324 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:27.465073    4930 cpu_manager.go:414] "containerMap: removing container" podUID="8deab213-f1c7-4883-9a64-72789a741faf" containerName="istio-proxy"
Nov 01 03:34:27.465324 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:27.465084    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8deab213-f1c7-4883-9a64-72789a741faf" containerName="istio-proxy"
Nov 01 03:34:27.465324 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:27.465152    4930 cpu_manager.go:414] "containerMap: removing container" podUID="ab67eed1-b327-4933-ac53-fd2b8c6e7a66" containerName="revoke-keys"
Nov 01 03:34:27.465324 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:27.465160    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab67eed1-b327-4933-ac53-fd2b8c6e7a66" containerName="revoke-keys"
Nov 01 03:34:27.465801 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:27.465432    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab67eed1-b327-4933-ac53-fd2b8c6e7a66" containerName="revoke-keys"
Nov 01 03:34:27.465801 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:27.465452    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="8deab213-f1c7-4883-9a64-72789a741faf" containerName="istio-proxy"
Nov 01 03:34:27.469662 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:27.469643    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e8c77d-68444c5d47-bj8bs"
Nov 01 03:34:27.543578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:27.530285    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e8c77d-68444c5d47-bj8bs"]
Nov 01 03:34:27.597209 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:27.581343    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-e8c77d-dockercfg-g79rl\""
Nov 01 03:34:27.597209 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:27.581417    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-e8c77d\""
Nov 01 03:34:27.656678 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:27.654836    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-e8c77d-75ffdb94b5-7klwh"]
Nov 01 03:34:27.656678 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:27.655524    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="8deab213-f1c7-4883-9a64-72789a741faf" containerName="istio-proxy"
Nov 01 03:34:27.697939 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:27.697907    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e8c77d-75ffdb94b5-7klwh"
Nov 01 03:34:27.719501 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:27.719418    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-e8c77d-75ffdb94b5-7klwh"]
Nov 01 03:34:27.777622 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:27.762670    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97r6f\" (UniqueName: \"kubernetes.io/projected/56f52541-2d41-4f11-a47f-489bfce8b284-kube-api-access-97r6f\") pod \"payload-pre-processing-e2e-shared-a-w0-e8c77d-68444c5d47-bj8bs\" (UID: \"56f52541-2d41-4f11-a47f-489bfce8b284\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e8c77d-68444c5d47-bj8bs"
Nov 01 03:34:27.778059 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:27.778024    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56f52541-2d41-4f11-a47f-489bfce8b284-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-e8c77d-68444c5d47-bj8bs\" (UID: \"56f52541-2d41-4f11-a47f-489bfce8b284\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e8c77d-68444c5d47-bj8bs"
Nov 01 03:34:27.778494 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:27.778465    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5cc9\" (UniqueName: \"kubernetes.io/projected/c4ef178c-04fb-43cf-a0e9-e6a42a087df5-kube-api-access-g5cc9\") pod \"payload-processing-e2e-shared-a-w0-e8c77d-75ffdb94b5-7klwh\" (UID: \"c4ef178c-04fb-43cf-a0e9-e6a42a087df5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e8c77d-75ffdb94b5-7klwh"
Nov 01 03:34:27.778884 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:27.778853    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4ef178c-04fb-43cf-a0e9-e6a42a087df5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-e8c77d-75ffdb94b5-7klwh\" (UID: \"c4ef178c-04fb-43cf-a0e9-e6a42a087df5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e8c77d-75ffdb94b5-7klwh"
Nov 01 03:34:27.880815 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:27.880777    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56f52541-2d41-4f11-a47f-489bfce8b284-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-e8c77d-68444c5d47-bj8bs\" (UID: \"56f52541-2d41-4f11-a47f-489bfce8b284\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e8c77d-68444c5d47-bj8bs"
Nov 01 03:34:27.881020 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:27.880865    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5cc9\" (UniqueName: \"kubernetes.io/projected/c4ef178c-04fb-43cf-a0e9-e6a42a087df5-kube-api-access-g5cc9\") pod \"payload-processing-e2e-shared-a-w0-e8c77d-75ffdb94b5-7klwh\" (UID: \"c4ef178c-04fb-43cf-a0e9-e6a42a087df5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e8c77d-75ffdb94b5-7klwh"
Nov 01 03:34:27.881020 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:27.880916    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4ef178c-04fb-43cf-a0e9-e6a42a087df5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-e8c77d-75ffdb94b5-7klwh\" (UID: \"c4ef178c-04fb-43cf-a0e9-e6a42a087df5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e8c77d-75ffdb94b5-7klwh"
Nov 01 03:34:27.881144 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:27.881081    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97r6f\" (UniqueName: \"kubernetes.io/projected/56f52541-2d41-4f11-a47f-489bfce8b284-kube-api-access-97r6f\") pod \"payload-pre-processing-e2e-shared-a-w0-e8c77d-68444c5d47-bj8bs\" (UID: \"56f52541-2d41-4f11-a47f-489bfce8b284\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e8c77d-68444c5d47-bj8bs"
Nov 01 03:34:27.881660 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:27.881635    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56f52541-2d41-4f11-a47f-489bfce8b284-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-e8c77d-68444c5d47-bj8bs\" (UID: \"56f52541-2d41-4f11-a47f-489bfce8b284\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e8c77d-68444c5d47-bj8bs"
Nov 01 03:34:27.881763 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:27.881727    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4ef178c-04fb-43cf-a0e9-e6a42a087df5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-e8c77d-75ffdb94b5-7klwh\" (UID: \"c4ef178c-04fb-43cf-a0e9-e6a42a087df5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e8c77d-75ffdb94b5-7klwh"
Nov 01 03:34:27.941982 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:27.941945    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97r6f\" (UniqueName: \"kubernetes.io/projected/56f52541-2d41-4f11-a47f-489bfce8b284-kube-api-access-97r6f\") pod \"payload-pre-processing-e2e-shared-a-w0-e8c77d-68444c5d47-bj8bs\" (UID: \"56f52541-2d41-4f11-a47f-489bfce8b284\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e8c77d-68444c5d47-bj8bs"
Nov 01 03:34:27.942128 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:27.942072    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5cc9\" (UniqueName: \"kubernetes.io/projected/c4ef178c-04fb-43cf-a0e9-e6a42a087df5-kube-api-access-g5cc9\") pod \"payload-processing-e2e-shared-a-w0-e8c77d-75ffdb94b5-7klwh\" (UID: \"c4ef178c-04fb-43cf-a0e9-e6a42a087df5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e8c77d-75ffdb94b5-7klwh"
Nov 01 03:34:28.027086 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:28.027007    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e8c77d-75ffdb94b5-7klwh"
Nov 01 03:34:28.088322 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:28.087918    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e8c77d-68444c5d47-bj8bs"
Nov 01 03:34:28.422291 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:34:28.422261    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4ef178c_04fb_43cf_a0e9_e6a42a087df5.slice/crio-67b2e946bb12892529fdeca56608134e7b2b4e20703eebe14e0a50c30b0d977a WatchSource:0}: Error finding container 67b2e946bb12892529fdeca56608134e7b2b4e20703eebe14e0a50c30b0d977a: Status 404 returned error can't find the container with id 67b2e946bb12892529fdeca56608134e7b2b4e20703eebe14e0a50c30b0d977a
Nov 01 03:34:28.423123 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:28.423090    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-e8c77d-75ffdb94b5-7klwh"]
Nov 01 03:34:28.522685 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:28.522655    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e8c77d-68444c5d47-bj8bs"]
Nov 01 03:34:28.523483 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:34:28.523460    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56f52541_2d41_4f11_a47f_489bfce8b284.slice/crio-a7acc2fc3624ba5bf569cd658a04a102493d541aa1780b224a0df6a6acdd3453 WatchSource:0}: Error finding container a7acc2fc3624ba5bf569cd658a04a102493d541aa1780b224a0df6a6acdd3453: Status 404 returned error can't find the container with id a7acc2fc3624ba5bf569cd658a04a102493d541aa1780b224a0df6a6acdd3453
Nov 01 03:34:28.938989 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:28.938900    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e8c77d-68444c5d47-bj8bs" event={"ID":"56f52541-2d41-4f11-a47f-489bfce8b284","Type":"ContainerStarted","Data":"08f24f5ab0b16965c447ba252ba9210546d2e5920e047780a2a4e3335ea58a8f"}
Nov 01 03:34:28.939162 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:28.939009    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e8c77d-68444c5d47-bj8bs" event={"ID":"56f52541-2d41-4f11-a47f-489bfce8b284","Type":"ContainerStarted","Data":"a7acc2fc3624ba5bf569cd658a04a102493d541aa1780b224a0df6a6acdd3453"}
Nov 01 03:34:28.939162 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:28.939041    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e8c77d-68444c5d47-bj8bs"
Nov 01 03:34:28.940737 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:28.940709    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e8c77d-75ffdb94b5-7klwh" event={"ID":"c4ef178c-04fb-43cf-a0e9-e6a42a087df5","Type":"ContainerStarted","Data":"bfd1c81df946a3cc590cf2546b06cf760eef451e2a6f88f525db23fd432f9a48"}
Nov 01 03:34:28.940865 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:28.940756    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e8c77d-75ffdb94b5-7klwh" event={"ID":"c4ef178c-04fb-43cf-a0e9-e6a42a087df5","Type":"ContainerStarted","Data":"67b2e946bb12892529fdeca56608134e7b2b4e20703eebe14e0a50c30b0d977a"}
Nov 01 03:34:28.940865 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:28.940797    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e8c77d-75ffdb94b5-7klwh"
Nov 01 03:34:29.057608 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:29.057532    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e8c77d-68444c5d47-bj8bs" podStartSLOduration=2.057514495 podStartE2EDuration="2.057514495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:34:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:34:29.055127333 +0000 UTC m=+1673.578133922" watchObservedRunningTime="2026-11-01 03:34:29.057514495 +0000 UTC m=+1673.580521082"
Nov 01 03:34:29.182220 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:29.182165    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e8c77d-75ffdb94b5-7klwh" podStartSLOduration=2.182150962 podStartE2EDuration="2.182150962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:34:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:34:29.180054082 +0000 UTC m=+1673.703060672" watchObservedRunningTime="2026-11-01 03:34:29.182150962 +0000 UTC m=+1673.705157549"
Nov 01 03:34:29.877842 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:29.877811    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n"]
Nov 01 03:34:29.878020 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:29.877997    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n" podUID="7f4df86a-a104-4c26-9b8e-9b556b23afcf" containerName="istio-proxy" containerID="cri-o://179e253cd8413cfc95c8c24785d42b0588a3e9eaa91dc1d70d189a7512de9171" gracePeriod=30
Nov 01 03:34:29.884091 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:29.883782    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr"]
Nov 01 03:34:29.884270 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:29.884184    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr" podUID="f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8" containerName="istio-proxy" containerID="cri-o://dbfcbf6fc66767eb265f4df5b42d750a5c8403f2ed14262eef96a0a9a97f651c" gracePeriod=30
Nov 01 03:34:29.926069 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:29.926030    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll"]
Nov 01 03:34:29.930004 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:29.929975    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"]
Nov 01 03:34:29.930173 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:29.930154    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r" podUID="5a115577-6d0a-4047-ab8b-8eb78f51acbe" containerName="istio-proxy" containerID="cri-o://20b525519fe9a48c409c20991b41e9576708c11064082a14690f7b6aa020727b" gracePeriod=30
Nov 01 03:34:29.950117 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:29.950072    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll" podUID="0405e5bf-ff6e-467d-9f4c-9bd36005c3ee" containerName="istio-proxy" containerID="cri-o://6f0ca25fbe6521eb5afe7e04a2a8d2422747e1d80f90124ec2776a591e3a8351" gracePeriod=30
Nov 01 03:34:30.010256 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:30.010221    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:31.215204 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:31.215166    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:32.097896 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:32.097865    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:32.309376 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:32.309301    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:32.501344 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:32.501261    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0cb10b-5597fdc799-qmc8n"]
Nov 01 03:34:32.505828 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:32.505806    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0cb10b-5597fdc799-qmc8n"
Nov 01 03:34:32.565203 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:32.565177    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0cb10b-5597fdc799-qmc8n"]
Nov 01 03:34:32.565713 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:32.565688    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9788fdf0-ba01-4cd7-8afb-5cd53881ffbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-0cb10b-5597fdc799-qmc8n\" (UID: \"9788fdf0-ba01-4cd7-8afb-5cd53881ffbf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0cb10b-5597fdc799-qmc8n"
Nov 01 03:34:32.565874 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:32.565847    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t96tv\" (UniqueName: \"kubernetes.io/projected/9788fdf0-ba01-4cd7-8afb-5cd53881ffbf-kube-api-access-t96tv\") pod \"payload-pre-processing-e2e-shared-a-w1-0cb10b-5597fdc799-qmc8n\" (UID: \"9788fdf0-ba01-4cd7-8afb-5cd53881ffbf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0cb10b-5597fdc799-qmc8n"
Nov 01 03:34:32.608861 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:32.608823    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-0cb10b\""
Nov 01 03:34:32.702751 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:32.692323    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9788fdf0-ba01-4cd7-8afb-5cd53881ffbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-0cb10b-5597fdc799-qmc8n\" (UID: \"9788fdf0-ba01-4cd7-8afb-5cd53881ffbf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0cb10b-5597fdc799-qmc8n"
Nov 01 03:34:32.702751 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:32.692484    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t96tv\" (UniqueName: \"kubernetes.io/projected/9788fdf0-ba01-4cd7-8afb-5cd53881ffbf-kube-api-access-t96tv\") pod \"payload-pre-processing-e2e-shared-a-w1-0cb10b-5597fdc799-qmc8n\" (UID: \"9788fdf0-ba01-4cd7-8afb-5cd53881ffbf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0cb10b-5597fdc799-qmc8n"
Nov 01 03:34:32.716143 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:32.716103    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9788fdf0-ba01-4cd7-8afb-5cd53881ffbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-0cb10b-5597fdc799-qmc8n\" (UID: \"9788fdf0-ba01-4cd7-8afb-5cd53881ffbf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0cb10b-5597fdc799-qmc8n"
Nov 01 03:34:32.748583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:32.746306    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0cb10b-6cb84f4cf5-qkv5p"]
Nov 01 03:34:32.765681 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:32.756027    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0cb10b-6cb84f4cf5-qkv5p"
Nov 01 03:34:32.765681 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:32.763711    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t96tv\" (UniqueName: \"kubernetes.io/projected/9788fdf0-ba01-4cd7-8afb-5cd53881ffbf-kube-api-access-t96tv\") pod \"payload-pre-processing-e2e-shared-a-w1-0cb10b-5597fdc799-qmc8n\" (UID: \"9788fdf0-ba01-4cd7-8afb-5cd53881ffbf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0cb10b-5597fdc799-qmc8n"
Nov 01 03:34:32.796182 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:32.796150    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0cb10b-6cb84f4cf5-qkv5p"]
Nov 01 03:34:32.833772 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:32.831900    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-0cb10b-dockercfg-8gntr\""
Nov 01 03:34:32.834195 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:32.834169    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0cb10b-5597fdc799-qmc8n"
Nov 01 03:34:32.896638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:32.896320    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwv6j\" (UniqueName: \"kubernetes.io/projected/279f59c4-604c-4352-bc67-8f2bd727f105-kube-api-access-dwv6j\") pod \"payload-processing-e2e-shared-a-w1-0cb10b-6cb84f4cf5-qkv5p\" (UID: \"279f59c4-604c-4352-bc67-8f2bd727f105\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0cb10b-6cb84f4cf5-qkv5p"
Nov 01 03:34:32.896638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:32.896385    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/279f59c4-604c-4352-bc67-8f2bd727f105-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-0cb10b-6cb84f4cf5-qkv5p\" (UID: \"279f59c4-604c-4352-bc67-8f2bd727f105\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0cb10b-6cb84f4cf5-qkv5p"
Nov 01 03:34:32.997938 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:32.997901    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/279f59c4-604c-4352-bc67-8f2bd727f105-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-0cb10b-6cb84f4cf5-qkv5p\" (UID: \"279f59c4-604c-4352-bc67-8f2bd727f105\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0cb10b-6cb84f4cf5-qkv5p"
Nov 01 03:34:32.998311 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:32.998291    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwv6j\" (UniqueName: \"kubernetes.io/projected/279f59c4-604c-4352-bc67-8f2bd727f105-kube-api-access-dwv6j\") pod \"payload-processing-e2e-shared-a-w1-0cb10b-6cb84f4cf5-qkv5p\" (UID: \"279f59c4-604c-4352-bc67-8f2bd727f105\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0cb10b-6cb84f4cf5-qkv5p"
Nov 01 03:34:32.998968 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:32.998940    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/279f59c4-604c-4352-bc67-8f2bd727f105-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-0cb10b-6cb84f4cf5-qkv5p\" (UID: \"279f59c4-604c-4352-bc67-8f2bd727f105\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0cb10b-6cb84f4cf5-qkv5p"
Nov 01 03:34:33.061085 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:33.061023    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwv6j\" (UniqueName: \"kubernetes.io/projected/279f59c4-604c-4352-bc67-8f2bd727f105-kube-api-access-dwv6j\") pod \"payload-processing-e2e-shared-a-w1-0cb10b-6cb84f4cf5-qkv5p\" (UID: \"279f59c4-604c-4352-bc67-8f2bd727f105\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0cb10b-6cb84f4cf5-qkv5p"
Nov 01 03:34:33.097979 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:33.097954    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0cb10b-6cb84f4cf5-qkv5p"
Nov 01 03:34:33.251226 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:34:33.251192    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9788fdf0_ba01_4cd7_8afb_5cd53881ffbf.slice/crio-dc4c5b5c70744cd80ea941c1d571141bd51715905ea067e4ee9c1e6988d9e809 WatchSource:0}: Error finding container dc4c5b5c70744cd80ea941c1d571141bd51715905ea067e4ee9c1e6988d9e809: Status 404 returned error can't find the container with id dc4c5b5c70744cd80ea941c1d571141bd51715905ea067e4ee9c1e6988d9e809
Nov 01 03:34:33.254540 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:33.254506    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0cb10b-5597fdc799-qmc8n"]
Nov 01 03:34:33.485652 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:34:33.485620    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod279f59c4_604c_4352_bc67_8f2bd727f105.slice/crio-ee52978f03144abf9de27031cd8593961622fdfd69553e70c919d3e3b8f19932 WatchSource:0}: Error finding container ee52978f03144abf9de27031cd8593961622fdfd69553e70c919d3e3b8f19932: Status 404 returned error can't find the container with id ee52978f03144abf9de27031cd8593961622fdfd69553e70c919d3e3b8f19932
Nov 01 03:34:33.487767 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:33.487736    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0cb10b-6cb84f4cf5-qkv5p"]
Nov 01 03:34:33.711850 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:33.711807    4930 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:34:33.712024 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:33.711872    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr" podUID="f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:34:33.987648 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:33.987542    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0cb10b-6cb84f4cf5-qkv5p" event={"ID":"279f59c4-604c-4352-bc67-8f2bd727f105","Type":"ContainerStarted","Data":"a466f818cf9e639733733f1c1b8a0dfe4f4715e975b56342f76a7eb0ff0505fd"}
Nov 01 03:34:33.987648 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:33.987622    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0cb10b-6cb84f4cf5-qkv5p" event={"ID":"279f59c4-604c-4352-bc67-8f2bd727f105","Type":"ContainerStarted","Data":"ee52978f03144abf9de27031cd8593961622fdfd69553e70c919d3e3b8f19932"}
Nov 01 03:34:33.987888 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:33.987678    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0cb10b-6cb84f4cf5-qkv5p"
Nov 01 03:34:33.989688 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:33.989651    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0cb10b-5597fdc799-qmc8n" event={"ID":"9788fdf0-ba01-4cd7-8afb-5cd53881ffbf","Type":"ContainerStarted","Data":"461e20ffc571b8a786ec44aba1082f0fb2f1b75446963723e57d380c7aad31ea"}
Nov 01 03:34:33.989810 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:33.989706    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0cb10b-5597fdc799-qmc8n" event={"ID":"9788fdf0-ba01-4cd7-8afb-5cd53881ffbf","Type":"ContainerStarted","Data":"dc4c5b5c70744cd80ea941c1d571141bd51715905ea067e4ee9c1e6988d9e809"}
Nov 01 03:34:33.989810 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:33.989736    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0cb10b-5597fdc799-qmc8n"
Nov 01 03:34:34.104598 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:34.104525    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0cb10b-6cb84f4cf5-qkv5p" podStartSLOduration=2.104510957 podStartE2EDuration="2.104510957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:34:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:34:34.102781198 +0000 UTC m=+1678.625787786" watchObservedRunningTime="2026-11-01 03:34:34.104510957 +0000 UTC m=+1678.627517545"
Nov 01 03:34:34.147952 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:34.147919    4930 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:34:34.148115 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:34.147980    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll" podUID="0405e5bf-ff6e-467d-9f4c-9bd36005c3ee" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:34:34.231997 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:34.231938    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0cb10b-5597fdc799-qmc8n" podStartSLOduration=2.231920172 podStartE2EDuration="2.231920172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:34:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:34:34.217446028 +0000 UTC m=+1678.740452618" watchObservedRunningTime="2026-11-01 03:34:34.231920172 +0000 UTC m=+1678.754926761"
Nov 01 03:34:34.242729 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:34.242632    4930 patch_prober.go:29] interesting pod/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:34:34.242729 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:34.242700    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n" podUID="7f4df86a-a104-4c26-9b8e-9b556b23afcf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:34:35.010758 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.010722    4930 generic.go:356] "Generic (PLEG): container finished" podID="7f4df86a-a104-4c26-9b8e-9b556b23afcf" containerID="179e253cd8413cfc95c8c24785d42b0588a3e9eaa91dc1d70d189a7512de9171" exitCode=0
Nov 01 03:34:35.011176 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.010795    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n" event={"ID":"7f4df86a-a104-4c26-9b8e-9b556b23afcf","Type":"ContainerDied","Data":"179e253cd8413cfc95c8c24785d42b0588a3e9eaa91dc1d70d189a7512de9171"}
Nov 01 03:34:35.012813 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.012777    4930 generic.go:356] "Generic (PLEG): container finished" podID="f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8" containerID="dbfcbf6fc66767eb265f4df5b42d750a5c8403f2ed14262eef96a0a9a97f651c" exitCode=0
Nov 01 03:34:35.012932 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.012845    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr" event={"ID":"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8","Type":"ContainerDied","Data":"dbfcbf6fc66767eb265f4df5b42d750a5c8403f2ed14262eef96a0a9a97f651c"}
Nov 01 03:34:35.303621 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.302471    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n"
Nov 01 03:34:35.332637 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.332612    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll"
Nov 01 03:34:35.337848 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.337828    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr"
Nov 01 03:34:35.342497 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.342473    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"
Nov 01 03:34:35.443877 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.443834    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a115577-6d0a-4047-ab8b-8eb78f51acbe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a115577-6d0a-4047-ab8b-8eb78f51acbe-workload-certs\") pod \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\" (UID: \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\") "
Nov 01 03:34:35.444033 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.443888    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f4df86a-a104-4c26-9b8e-9b556b23afcf-istio-token\" (UniqueName: \"kubernetes.io/projected/7f4df86a-a104-4c26-9b8e-9b556b23afcf-istio-token\") pod \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\" (UID: \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\") "
Nov 01 03:34:35.444033 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.443921    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-istio-podinfo\") pod \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\" (UID: \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\") "
Nov 01 03:34:35.444033 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.443974    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f4df86a-a104-4c26-9b8e-9b556b23afcf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f4df86a-a104-4c26-9b8e-9b556b23afcf-workload-certs\") pod \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\" (UID: \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\") "
Nov 01 03:34:35.444033 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.444011    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f4df86a-a104-4c26-9b8e-9b556b23afcf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4df86a-a104-4c26-9b8e-9b556b23afcf-credential-socket\") pod \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\" (UID: \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\") "
Nov 01 03:34:35.444273 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.444068    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-credential-socket\") pod \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\" (UID: \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\") "
Nov 01 03:34:35.444273 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.444130    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a115577-6d0a-4047-ab8b-8eb78f51acbe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a115577-6d0a-4047-ab8b-8eb78f51acbe-istio-podinfo\") pod \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\" (UID: \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\") "
Nov 01 03:34:35.444273 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.444184    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a115577-6d0a-4047-ab8b-8eb78f51acbe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a115577-6d0a-4047-ab8b-8eb78f51acbe-istiod-ca-cert\") pod \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\" (UID: \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\") "
Nov 01 03:34:35.444273 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.444214    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-workload-socket\") pod \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\" (UID: \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\") "
Nov 01 03:34:35.444273 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.444259    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-credential-socket\") pod \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\" (UID: \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\") "
Nov 01 03:34:35.444534 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.444185    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a115577-6d0a-4047-ab8b-8eb78f51acbe-workload-certs" pod "5a115577-6d0a-4047-ab8b-8eb78f51acbe" (UID: "5a115577-6d0a-4047-ab8b-8eb78f51acbe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:35.444534 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.444310    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f4df86a-a104-4c26-9b8e-9b556b23afcf-kube-api-access-cxrnj\" (UniqueName: \"kubernetes.io/projected/7f4df86a-a104-4c26-9b8e-9b556b23afcf-kube-api-access-cxrnj\") pod \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\" (UID: \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\") "
Nov 01 03:34:35.444534 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.444344    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f4df86a-a104-4c26-9b8e-9b556b23afcf-credential-socket" pod "7f4df86a-a104-4c26-9b8e-9b556b23afcf" (UID: "7f4df86a-a104-4c26-9b8e-9b556b23afcf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:35.444534 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.444357    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f4df86a-a104-4c26-9b8e-9b556b23afcf-workload-certs" pod "7f4df86a-a104-4c26-9b8e-9b556b23afcf" (UID: "7f4df86a-a104-4c26-9b8e-9b556b23afcf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:35.444534 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.444397    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a115577-6d0a-4047-ab8b-8eb78f51acbe-kube-api-access-jmmsz\" (UniqueName: \"kubernetes.io/projected/5a115577-6d0a-4047-ab8b-8eb78f51acbe-kube-api-access-jmmsz\") pod \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\" (UID: \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\") "
Nov 01 03:34:35.444534 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.444441    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-istio-podinfo\") pod \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\" (UID: \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\") "
Nov 01 03:34:35.444534 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.444496    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-istio-data\") pod \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\" (UID: \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\") "
Nov 01 03:34:35.444534 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.444525    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-istio-envoy\") pod \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\" (UID: \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\") "
Nov 01 03:34:35.445165 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.444610    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a115577-6d0a-4047-ab8b-8eb78f51acbe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a115577-6d0a-4047-ab8b-8eb78f51acbe-credential-socket\") pod \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\" (UID: \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\") "
Nov 01 03:34:35.445165 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.444642    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f4df86a-a104-4c26-9b8e-9b556b23afcf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f4df86a-a104-4c26-9b8e-9b556b23afcf-istio-envoy\") pod \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\" (UID: \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\") "
Nov 01 03:34:35.445165 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.444649    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-workload-socket" pod "0405e5bf-ff6e-467d-9f4c-9bd36005c3ee" (UID: "0405e5bf-ff6e-467d-9f4c-9bd36005c3ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:35.445165 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.444691    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-istiod-ca-cert\") pod \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\" (UID: \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\") "
Nov 01 03:34:35.445165 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.444731    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a115577-6d0a-4047-ab8b-8eb78f51acbe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a115577-6d0a-4047-ab8b-8eb78f51acbe-istio-data\") pod \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\" (UID: \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\") "
Nov 01 03:34:35.445165 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.444767    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f4df86a-a104-4c26-9b8e-9b556b23afcf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4df86a-a104-4c26-9b8e-9b556b23afcf-workload-socket\") pod \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\" (UID: \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\") "
Nov 01 03:34:35.445165 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.444819    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-istio-token\" (UniqueName: \"kubernetes.io/projected/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-istio-token\") pod \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\" (UID: \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\") "
Nov 01 03:34:35.445165 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.444871    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-istio-data" pod "f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8" (UID: "f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:35.445165 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.444875    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-workload-certs\") pod \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\" (UID: \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\") "
Nov 01 03:34:35.445165 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.445006    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-workload-socket\") pod \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\" (UID: \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\") "
Nov 01 03:34:35.445165 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.445074    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-workload-certs\") pod \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\" (UID: \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\") "
Nov 01 03:34:35.445165 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.445119    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-credential-socket" pod "0405e5bf-ff6e-467d-9f4c-9bd36005c3ee" (UID: "0405e5bf-ff6e-467d-9f4c-9bd36005c3ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:35.445165 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.445139    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-kube-api-access-dnd8l\" (UniqueName: \"kubernetes.io/projected/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-kube-api-access-dnd8l\") pod \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\" (UID: \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\") "
Nov 01 03:34:35.445165 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.445171    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-istio-token\" (UniqueName: \"kubernetes.io/projected/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-istio-token\") pod \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\" (UID: \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\") "
Nov 01 03:34:35.446194 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.445215    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-istio-envoy\") pod \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\" (UID: \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\") "
Nov 01 03:34:35.446194 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.445246    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f4df86a-a104-4c26-9b8e-9b556b23afcf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f4df86a-a104-4c26-9b8e-9b556b23afcf-istio-podinfo\") pod \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\" (UID: \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\") "
Nov 01 03:34:35.446194 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.445285    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a115577-6d0a-4047-ab8b-8eb78f51acbe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a115577-6d0a-4047-ab8b-8eb78f51acbe-istio-envoy\") pod \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\" (UID: \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\") "
Nov 01 03:34:35.446194 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.445313    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f4df86a-a104-4c26-9b8e-9b556b23afcf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f4df86a-a104-4c26-9b8e-9b556b23afcf-istio-data\") pod \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\" (UID: \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\") "
Nov 01 03:34:35.446194 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.445351    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f4df86a-a104-4c26-9b8e-9b556b23afcf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f4df86a-a104-4c26-9b8e-9b556b23afcf-istiod-ca-cert\") pod \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\" (UID: \"7f4df86a-a104-4c26-9b8e-9b556b23afcf\") "
Nov 01 03:34:35.446194 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.445420    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-istio-data\") pod \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\" (UID: \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\") "
Nov 01 03:34:35.446194 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.445433    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-istiod-ca-cert" pod "0405e5bf-ff6e-467d-9f4c-9bd36005c3ee" (UID: "0405e5bf-ff6e-467d-9f4c-9bd36005c3ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:34:35.446194 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.445459    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-kube-api-access-xmdrs\" (UniqueName: \"kubernetes.io/projected/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-kube-api-access-xmdrs\") pod \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\" (UID: \"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee\") "
Nov 01 03:34:35.446194 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.445493    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a115577-6d0a-4047-ab8b-8eb78f51acbe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a115577-6d0a-4047-ab8b-8eb78f51acbe-workload-socket\") pod \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\" (UID: \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\") "
Nov 01 03:34:35.446194 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.445534    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-istiod-ca-cert\") pod \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\" (UID: \"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8\") "
Nov 01 03:34:35.446194 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.445597    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a115577-6d0a-4047-ab8b-8eb78f51acbe-istio-token\" (UniqueName: \"kubernetes.io/projected/5a115577-6d0a-4047-ab8b-8eb78f51acbe-istio-token\") pod \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\" (UID: \"5a115577-6d0a-4047-ab8b-8eb78f51acbe\") "
Nov 01 03:34:35.446194 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.445725    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-workload-certs" pod "f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8" (UID: "f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:35.446194 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.445938    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a115577-6d0a-4047-ab8b-8eb78f51acbe-istiod-ca-cert" pod "5a115577-6d0a-4047-ab8b-8eb78f51acbe" (UID: "5a115577-6d0a-4047-ab8b-8eb78f51acbe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:34:35.446194 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.446016    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a115577-6d0a-4047-ab8b-8eb78f51acbe-istio-data" pod "5a115577-6d0a-4047-ab8b-8eb78f51acbe" (UID: "5a115577-6d0a-4047-ab8b-8eb78f51acbe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:35.447002 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.446282    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f4df86a-a104-4c26-9b8e-9b556b23afcf-workload-socket" pod "7f4df86a-a104-4c26-9b8e-9b556b23afcf" (UID: "7f4df86a-a104-4c26-9b8e-9b556b23afcf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:35.447002 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.446495    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-credential-socket" pod "f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8" (UID: "f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:35.447002 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.446584    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-istio-podinfo" pod "f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8" (UID: "f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:34:35.447002 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.446668    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a115577-6d0a-4047-ab8b-8eb78f51acbe-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:35.447002 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.446692    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f4df86a-a104-4c26-9b8e-9b556b23afcf-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:35.447002 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.446692    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f4df86a-a104-4c26-9b8e-9b556b23afcf-istio-token" pod "7f4df86a-a104-4c26-9b8e-9b556b23afcf" (UID: "7f4df86a-a104-4c26-9b8e-9b556b23afcf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:34:35.447002 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.446707    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4df86a-a104-4c26-9b8e-9b556b23afcf-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:35.447002 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.446721    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:35.447002 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.446727    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a115577-6d0a-4047-ab8b-8eb78f51acbe-istio-podinfo" pod "5a115577-6d0a-4047-ab8b-8eb78f51acbe" (UID: "5a115577-6d0a-4047-ab8b-8eb78f51acbe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:34:35.447002 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.446736    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a115577-6d0a-4047-ab8b-8eb78f51acbe-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:35.447002 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.446769    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:35.447002 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.446788    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:35.447002 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.446805    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:35.447002 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.446820    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:35.447002 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.446858    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a115577-6d0a-4047-ab8b-8eb78f51acbe-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:35.447002 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.446872    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4df86a-a104-4c26-9b8e-9b556b23afcf-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:35.447002 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.446885    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:35.447002 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.446903    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a115577-6d0a-4047-ab8b-8eb78f51acbe-credential-socket" pod "5a115577-6d0a-4047-ab8b-8eb78f51acbe" (UID: "5a115577-6d0a-4047-ab8b-8eb78f51acbe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:35.451422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.447146    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f4df86a-a104-4c26-9b8e-9b556b23afcf-kube-api-access-cxrnj" pod "7f4df86a-a104-4c26-9b8e-9b556b23afcf" (UID: "7f4df86a-a104-4c26-9b8e-9b556b23afcf"). InnerVolumeSpecName "kube-api-access-cxrnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:34:35.451422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.447330    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a115577-6d0a-4047-ab8b-8eb78f51acbe-workload-socket" pod "5a115577-6d0a-4047-ab8b-8eb78f51acbe" (UID: "5a115577-6d0a-4047-ab8b-8eb78f51acbe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:35.451422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.447436    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-workload-certs" pod "0405e5bf-ff6e-467d-9f4c-9bd36005c3ee" (UID: "0405e5bf-ff6e-467d-9f4c-9bd36005c3ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:35.451422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.447540    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f4df86a-a104-4c26-9b8e-9b556b23afcf-istio-data" pod "7f4df86a-a104-4c26-9b8e-9b556b23afcf" (UID: "7f4df86a-a104-4c26-9b8e-9b556b23afcf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:35.451422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.447592    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f4df86a-a104-4c26-9b8e-9b556b23afcf-istiod-ca-cert" pod "7f4df86a-a104-4c26-9b8e-9b556b23afcf" (UID: "7f4df86a-a104-4c26-9b8e-9b556b23afcf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:34:35.451422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.447766    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-istio-data" pod "0405e5bf-ff6e-467d-9f4c-9bd36005c3ee" (UID: "0405e5bf-ff6e-467d-9f4c-9bd36005c3ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:35.451422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.448259    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-istiod-ca-cert" pod "f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8" (UID: "f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:34:35.451422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.448873    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a115577-6d0a-4047-ab8b-8eb78f51acbe-istio-token" pod "5a115577-6d0a-4047-ab8b-8eb78f51acbe" (UID: "5a115577-6d0a-4047-ab8b-8eb78f51acbe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:34:35.451422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.449309    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-workload-socket" pod "f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8" (UID: "f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:35.451422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.449630    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f4df86a-a104-4c26-9b8e-9b556b23afcf-istio-envoy" pod "7f4df86a-a104-4c26-9b8e-9b556b23afcf" (UID: "7f4df86a-a104-4c26-9b8e-9b556b23afcf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:35.451422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.449726    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-istio-token" pod "f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8" (UID: "f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:34:35.451422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.450221    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f4df86a-a104-4c26-9b8e-9b556b23afcf-istio-podinfo" pod "7f4df86a-a104-4c26-9b8e-9b556b23afcf" (UID: "7f4df86a-a104-4c26-9b8e-9b556b23afcf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:34:35.451422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.450829    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-istio-envoy" pod "0405e5bf-ff6e-467d-9f4c-9bd36005c3ee" (UID: "0405e5bf-ff6e-467d-9f4c-9bd36005c3ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:35.451422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.451292    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-istio-podinfo" pod "0405e5bf-ff6e-467d-9f4c-9bd36005c3ee" (UID: "0405e5bf-ff6e-467d-9f4c-9bd36005c3ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:34:35.452231 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.452152    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-kube-api-access-xmdrs" pod "0405e5bf-ff6e-467d-9f4c-9bd36005c3ee" (UID: "0405e5bf-ff6e-467d-9f4c-9bd36005c3ee"). InnerVolumeSpecName "kube-api-access-xmdrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:34:35.452231 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.452190    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a115577-6d0a-4047-ab8b-8eb78f51acbe-istio-envoy" pod "5a115577-6d0a-4047-ab8b-8eb78f51acbe" (UID: "5a115577-6d0a-4047-ab8b-8eb78f51acbe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:35.452231 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.452197    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a115577-6d0a-4047-ab8b-8eb78f51acbe-kube-api-access-jmmsz" pod "5a115577-6d0a-4047-ab8b-8eb78f51acbe" (UID: "5a115577-6d0a-4047-ab8b-8eb78f51acbe"). InnerVolumeSpecName "kube-api-access-jmmsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:34:35.452231 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.452220    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-kube-api-access-dnd8l" pod "f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8" (UID: "f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8"). InnerVolumeSpecName "kube-api-access-dnd8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:34:35.452453 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.452258    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-istio-token" pod "0405e5bf-ff6e-467d-9f4c-9bd36005c3ee" (UID: "0405e5bf-ff6e-467d-9f4c-9bd36005c3ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:34:35.453353 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.453332    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-istio-envoy" pod "f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8" (UID: "f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:34:35.548309 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.548265    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:35.548502 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.548481    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:35.548502 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.548501    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnd8l\" (UniqueName: \"kubernetes.io/projected/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-kube-api-access-dnd8l\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:35.548696 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.548511    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:35.548696 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.548521    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:35.548696 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.548533    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f4df86a-a104-4c26-9b8e-9b556b23afcf-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:35.548696 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.548542    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a115577-6d0a-4047-ab8b-8eb78f51acbe-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:35.548696 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.548550    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f4df86a-a104-4c26-9b8e-9b556b23afcf-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:35.548696 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.548577    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f4df86a-a104-4c26-9b8e-9b556b23afcf-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:35.548696 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.548585    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:35.548696 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.548596    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmdrs\" (UniqueName: \"kubernetes.io/projected/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-kube-api-access-xmdrs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:35.548696 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.548606    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a115577-6d0a-4047-ab8b-8eb78f51acbe-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:35.548696 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.548615    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:35.548696 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.548624    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a115577-6d0a-4047-ab8b-8eb78f51acbe-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:35.548696 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.548633    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f4df86a-a104-4c26-9b8e-9b556b23afcf-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:35.548696 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.548642    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:35.548696 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.548651    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a115577-6d0a-4047-ab8b-8eb78f51acbe-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:35.548696 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.548660    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxrnj\" (UniqueName: \"kubernetes.io/projected/7f4df86a-a104-4c26-9b8e-9b556b23afcf-kube-api-access-cxrnj\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:35.548696 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.548668    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmmsz\" (UniqueName: \"kubernetes.io/projected/5a115577-6d0a-4047-ab8b-8eb78f51acbe-kube-api-access-jmmsz\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:35.548696 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.548678    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:35.548696 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.548691    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:35.549226 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.548705    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a115577-6d0a-4047-ab8b-8eb78f51acbe-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:35.549226 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.548719    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f4df86a-a104-4c26-9b8e-9b556b23afcf-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:35.549226 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.548728    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:35.552404 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.552376    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:35.789195 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:35.789102    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:36.030324 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:36.030258    4930 generic.go:356] "Generic (PLEG): container finished" podID="0405e5bf-ff6e-467d-9f4c-9bd36005c3ee" containerID="6f0ca25fbe6521eb5afe7e04a2a8d2422747e1d80f90124ec2776a591e3a8351" exitCode=0
Nov 01 03:34:36.030927 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:36.030436    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll" event={"ID":"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee","Type":"ContainerDied","Data":"6f0ca25fbe6521eb5afe7e04a2a8d2422747e1d80f90124ec2776a591e3a8351"}
Nov 01 03:34:36.030927 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:36.030528    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll" event={"ID":"0405e5bf-ff6e-467d-9f4c-9bd36005c3ee","Type":"ContainerDied","Data":"5fd6986d23225df3408eee472e112643b562dd0c5cc246a3bf11a323ba520ece"}
Nov 01 03:34:36.030927 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:36.030528    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll"
Nov 01 03:34:36.030927 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:36.030616    4930 scope.go:122] "RemoveContainer" containerID="6f0ca25fbe6521eb5afe7e04a2a8d2422747e1d80f90124ec2776a591e3a8351"
Nov 01 03:34:36.033259 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:36.033234    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n"
Nov 01 03:34:36.033508 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:36.033249    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n" event={"ID":"7f4df86a-a104-4c26-9b8e-9b556b23afcf","Type":"ContainerDied","Data":"a18676800dd2d3bc0cdeee4ba49e92b1b101a1d3328d9d757898cf1fdd6769d9"}
Nov 01 03:34:36.035166 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:36.035144    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr" event={"ID":"f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8","Type":"ContainerDied","Data":"6ed3f3636286b90c34e710dc54ec90876ada3a97a455a76542cdf865617c1336"}
Nov 01 03:34:36.035274 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:36.035172    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr"
Nov 01 03:34:36.037377 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:36.037355    4930 generic.go:356] "Generic (PLEG): container finished" podID="5a115577-6d0a-4047-ab8b-8eb78f51acbe" containerID="20b525519fe9a48c409c20991b41e9576708c11064082a14690f7b6aa020727b" exitCode=0
Nov 01 03:34:36.037469 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:36.037415    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"
Nov 01 03:34:36.037469 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:36.037437    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r" event={"ID":"5a115577-6d0a-4047-ab8b-8eb78f51acbe","Type":"ContainerDied","Data":"20b525519fe9a48c409c20991b41e9576708c11064082a14690f7b6aa020727b"}
Nov 01 03:34:36.037590 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:36.037483    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r" event={"ID":"5a115577-6d0a-4047-ab8b-8eb78f51acbe","Type":"ContainerDied","Data":"b5814351a3c6aa46164405f9564bb6cde12ad22dc35004db84710fe560333bd6"}
Nov 01 03:34:36.046992 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:36.046970    4930 scope.go:122] "RemoveContainer" containerID="6f0ca25fbe6521eb5afe7e04a2a8d2422747e1d80f90124ec2776a591e3a8351"
Nov 01 03:34:36.047403 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:34:36.047382    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f0ca25fbe6521eb5afe7e04a2a8d2422747e1d80f90124ec2776a591e3a8351\": container with ID starting with 6f0ca25fbe6521eb5afe7e04a2a8d2422747e1d80f90124ec2776a591e3a8351 not found: ID does not exist" containerID="6f0ca25fbe6521eb5afe7e04a2a8d2422747e1d80f90124ec2776a591e3a8351"
Nov 01 03:34:36.047496 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:36.047408    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f0ca25fbe6521eb5afe7e04a2a8d2422747e1d80f90124ec2776a591e3a8351"} err="failed to get container status \"6f0ca25fbe6521eb5afe7e04a2a8d2422747e1d80f90124ec2776a591e3a8351\": rpc error: code = NotFound desc = could not find container \"6f0ca25fbe6521eb5afe7e04a2a8d2422747e1d80f90124ec2776a591e3a8351\": container with ID starting with 6f0ca25fbe6521eb5afe7e04a2a8d2422747e1d80f90124ec2776a591e3a8351 not found: ID does not exist"
Nov 01 03:34:36.047496 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:36.047427    4930 scope.go:122] "RemoveContainer" containerID="179e253cd8413cfc95c8c24785d42b0588a3e9eaa91dc1d70d189a7512de9171"
Nov 01 03:34:36.061111 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:36.061086    4930 scope.go:122] "RemoveContainer" containerID="dbfcbf6fc66767eb265f4df5b42d750a5c8403f2ed14262eef96a0a9a97f651c"
Nov 01 03:34:36.072348 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:36.072328    4930 scope.go:122] "RemoveContainer" containerID="20b525519fe9a48c409c20991b41e9576708c11064082a14690f7b6aa020727b"
Nov 01 03:34:36.085690 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:36.085662    4930 scope.go:122] "RemoveContainer" containerID="9094fc4b7319e4e96f85bb5a6d8e218344d2f40afb91f12c36ab6e1417424ca8"
Nov 01 03:34:36.096086 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:36.096069    4930 scope.go:122] "RemoveContainer" containerID="20b525519fe9a48c409c20991b41e9576708c11064082a14690f7b6aa020727b"
Nov 01 03:34:36.096392 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:34:36.096369    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20b525519fe9a48c409c20991b41e9576708c11064082a14690f7b6aa020727b\": container with ID starting with 20b525519fe9a48c409c20991b41e9576708c11064082a14690f7b6aa020727b not found: ID does not exist" containerID="20b525519fe9a48c409c20991b41e9576708c11064082a14690f7b6aa020727b"
Nov 01 03:34:36.096454 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:36.096404    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20b525519fe9a48c409c20991b41e9576708c11064082a14690f7b6aa020727b"} err="failed to get container status \"20b525519fe9a48c409c20991b41e9576708c11064082a14690f7b6aa020727b\": rpc error: code = NotFound desc = could not find container \"20b525519fe9a48c409c20991b41e9576708c11064082a14690f7b6aa020727b\": container with ID starting with 20b525519fe9a48c409c20991b41e9576708c11064082a14690f7b6aa020727b not found: ID does not exist"
Nov 01 03:34:36.096454 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:36.096429    4930 scope.go:122] "RemoveContainer" containerID="9094fc4b7319e4e96f85bb5a6d8e218344d2f40afb91f12c36ab6e1417424ca8"
Nov 01 03:34:36.096765 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:34:36.096744    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9094fc4b7319e4e96f85bb5a6d8e218344d2f40afb91f12c36ab6e1417424ca8\": container with ID starting with 9094fc4b7319e4e96f85bb5a6d8e218344d2f40afb91f12c36ab6e1417424ca8 not found: ID does not exist" containerID="9094fc4b7319e4e96f85bb5a6d8e218344d2f40afb91f12c36ab6e1417424ca8"
Nov 01 03:34:36.096834 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:36.096773    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9094fc4b7319e4e96f85bb5a6d8e218344d2f40afb91f12c36ab6e1417424ca8"} err="failed to get container status \"9094fc4b7319e4e96f85bb5a6d8e218344d2f40afb91f12c36ab6e1417424ca8\": rpc error: code = NotFound desc = could not find container \"9094fc4b7319e4e96f85bb5a6d8e218344d2f40afb91f12c36ab6e1417424ca8\": container with ID starting with 9094fc4b7319e4e96f85bb5a6d8e218344d2f40afb91f12c36ab6e1417424ca8 not found: ID does not exist"
Nov 01 03:34:36.207552 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:36.207516    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll"]
Nov 01 03:34:36.218048 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:36.218020    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-h7mll"]
Nov 01 03:34:36.345457 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:36.345424    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n"]
Nov 01 03:34:36.352393 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:36.352365    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-6x26n"]
Nov 01 03:34:36.469988 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:36.469950    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"]
Nov 01 03:34:36.473695 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:36.473667    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-lz66r"]
Nov 01 03:34:36.577027 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:36.576995    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr"]
Nov 01 03:34:36.582720 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:36.582695    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ded2d7-openshift-default-5f59b5bd9-zj5cr"]
Nov 01 03:34:36.816128 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:36.816052    4930 scope.go:122] "RemoveContainer" containerID="34d3ff014710376aa82f5101f2ba09f3e036e33b36cd963a24d1af2ef1524e16"
Nov 01 03:34:36.827677 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:36.827646    4930 scope.go:122] "RemoveContainer" containerID="377247bfcd6ad86da38fbd46356f84da00db77ffce2ab256ed6fa9c07c8a3a40"
Nov 01 03:34:36.839433 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:36.839410    4930 scope.go:122] "RemoveContainer" containerID="10b1639cadf8c0b2abc646e911d3038a647213e55265c226e86496ade4a58d52"
Nov 01 03:34:36.849154 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:36.849133    4930 scope.go:122] "RemoveContainer" containerID="3084384ef164cba47fcb5329747dafb6028a38b677778baf73d2537582b1a0d8"
Nov 01 03:34:36.861166 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:36.861144    4930 scope.go:122] "RemoveContainer" containerID="05502d3f861de4736ff8f75334354ea2734f2fca74bbf0547fd571a08b14c021"
Nov 01 03:34:36.872868 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:36.872847    4930 scope.go:122] "RemoveContainer" containerID="57f461a3d368806b07d30d1874fce81388c3d0c267e67df0f8b44dd42a15e219"
Nov 01 03:34:36.885013 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:36.884978    4930 scope.go:122] "RemoveContainer" containerID="e012cf7b52a7a0e003450fd4b0b745a6bfefe21319bcf0ceefe423df98e392e4"
Nov 01 03:34:36.900351 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:36.900327    4930 scope.go:122] "RemoveContainer" containerID="3d912421efa5cee727d9206633b34f73d81622c0e1a0f8f52946313ab0b3b83a"
Nov 01 03:34:36.910551 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:36.910530    4930 scope.go:122] "RemoveContainer" containerID="f535b409ee6a3d8ad78701a400ab7dd32ed34fd46dd733a41e3891a71990d5fd"
Nov 01 03:34:36.920617 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:36.920598    4930 scope.go:122] "RemoveContainer" containerID="53caacd8ae17df838ae25c003f702df8ff477df23952408adbb29225d4298599"
Nov 01 03:34:36.930135 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:36.930111    4930 scope.go:122] "RemoveContainer" containerID="d24c1319b99a481b37ea62b6c11af07f77c38deea31ee734b3acc3bfef240f0b"
Nov 01 03:34:36.940751 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:36.940729    4930 scope.go:122] "RemoveContainer" containerID="0c607acf23828e0b0a5f054c8b275a806b69a186aed3972a492d090900f8a4e5"
Nov 01 03:34:37.623056 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:37.623023    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0405e5bf-ff6e-467d-9f4c-9bd36005c3ee" path="/var/lib/kubelet/pods/0405e5bf-ff6e-467d-9f4c-9bd36005c3ee/volumes"
Nov 01 03:34:37.623590 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:37.623546    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a115577-6d0a-4047-ab8b-8eb78f51acbe" path="/var/lib/kubelet/pods/5a115577-6d0a-4047-ab8b-8eb78f51acbe/volumes"
Nov 01 03:34:37.624042 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:37.624028    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f4df86a-a104-4c26-9b8e-9b556b23afcf" path="/var/lib/kubelet/pods/7f4df86a-a104-4c26-9b8e-9b556b23afcf/volumes"
Nov 01 03:34:37.625095 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:37.625077    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8" path="/var/lib/kubelet/pods/f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8/volumes"
Nov 01 03:34:38.989844 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:38.989802    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:39.953522 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:39.953494    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e8c77d-75ffdb94b5-7klwh"
Nov 01 03:34:39.953722 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:39.953575    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e8c77d-68444c5d47-bj8bs"
Nov 01 03:34:40.709073 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:40.709038    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:40.907876 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:40.907835    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:41.202004 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:41.201965    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:41.741764 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:41.741713    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"]
Nov 01 03:34:41.746578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:41.742757    4930 cpu_manager.go:414] "containerMap: removing container" podUID="f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8" containerName="istio-proxy"
Nov 01 03:34:41.746578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:41.742782    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8" containerName="istio-proxy"
Nov 01 03:34:41.746578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:41.742805    4930 cpu_manager.go:414] "containerMap: removing container" podUID="5a115577-6d0a-4047-ab8b-8eb78f51acbe" containerName="istio-proxy"
Nov 01 03:34:41.746578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:41.742813    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a115577-6d0a-4047-ab8b-8eb78f51acbe" containerName="istio-proxy"
Nov 01 03:34:41.746578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:41.742829    4930 cpu_manager.go:414] "containerMap: removing container" podUID="7f4df86a-a104-4c26-9b8e-9b556b23afcf" containerName="istio-proxy"
Nov 01 03:34:41.746578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:41.742837    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f4df86a-a104-4c26-9b8e-9b556b23afcf" containerName="istio-proxy"
Nov 01 03:34:41.746578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:41.742871    4930 cpu_manager.go:414] "containerMap: removing container" podUID="5a115577-6d0a-4047-ab8b-8eb78f51acbe" containerName="istio-proxy"
Nov 01 03:34:41.746578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:41.742879    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a115577-6d0a-4047-ab8b-8eb78f51acbe" containerName="istio-proxy"
Nov 01 03:34:41.746578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:41.742909    4930 cpu_manager.go:414] "containerMap: removing container" podUID="0405e5bf-ff6e-467d-9f4c-9bd36005c3ee" containerName="istio-proxy"
Nov 01 03:34:41.746578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:41.742918    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0405e5bf-ff6e-467d-9f4c-9bd36005c3ee" containerName="istio-proxy"
Nov 01 03:34:41.746578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:41.743062    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="0405e5bf-ff6e-467d-9f4c-9bd36005c3ee" containerName="istio-proxy"
Nov 01 03:34:41.746578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:41.743079    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="f558d1dd-df4f-4aeb-a86f-f1c2f3e4c7a8" containerName="istio-proxy"
Nov 01 03:34:41.746578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:41.743091    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a115577-6d0a-4047-ab8b-8eb78f51acbe" containerName="istio-proxy"
Nov 01 03:34:41.746578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:41.743102    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f4df86a-a104-4c26-9b8e-9b556b23afcf" containerName="istio-proxy"
Nov 01 03:34:41.773582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:41.771186    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"
Nov 01 03:34:41.856585 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:41.827204    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b941e0fe-6b17-4e30-917c-4d3f4382c545-credential-socket\") pod \"e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8\" (UID: \"b941e0fe-6b17-4e30-917c-4d3f4382c545\") " pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"
Nov 01 03:34:41.857029 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:41.856995    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b941e0fe-6b17-4e30-917c-4d3f4382c545-istio-data\") pod \"e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8\" (UID: \"b941e0fe-6b17-4e30-917c-4d3f4382c545\") " pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"
Nov 01 03:34:41.857225 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:41.857203    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b941e0fe-6b17-4e30-917c-4d3f4382c545-istio-podinfo\") pod \"e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8\" (UID: \"b941e0fe-6b17-4e30-917c-4d3f4382c545\") " pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"
Nov 01 03:34:41.857388 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:41.857368    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b941e0fe-6b17-4e30-917c-4d3f4382c545-istio-token\") pod \"e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8\" (UID: \"b941e0fe-6b17-4e30-917c-4d3f4382c545\") " pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"
Nov 01 03:34:41.857753 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:41.857714    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b941e0fe-6b17-4e30-917c-4d3f4382c545-workload-socket\") pod \"e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8\" (UID: \"b941e0fe-6b17-4e30-917c-4d3f4382c545\") " pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"
Nov 01 03:34:41.871863 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:41.871828    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfvq5\" (UniqueName: \"kubernetes.io/projected/b941e0fe-6b17-4e30-917c-4d3f4382c545-kube-api-access-nfvq5\") pod \"e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8\" (UID: \"b941e0fe-6b17-4e30-917c-4d3f4382c545\") " pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"
Nov 01 03:34:41.872145 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:41.872115    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b941e0fe-6b17-4e30-917c-4d3f4382c545-istio-envoy\") pod \"e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8\" (UID: \"b941e0fe-6b17-4e30-917c-4d3f4382c545\") " pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"
Nov 01 03:34:41.882416 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:41.872338    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b941e0fe-6b17-4e30-917c-4d3f4382c545-workload-certs\") pod \"e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8\" (UID: \"b941e0fe-6b17-4e30-917c-4d3f4382c545\") " pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"
Nov 01 03:34:41.882792 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:41.882769    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b941e0fe-6b17-4e30-917c-4d3f4382c545-istiod-ca-cert\") pod \"e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8\" (UID: \"b941e0fe-6b17-4e30-917c-4d3f4382c545\") " pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"
Nov 01 03:34:41.939576 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:41.933853    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-686153-openshift-default-dockercfg-v8v7b\""
Nov 01 03:34:41.964311 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:41.964278    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:41.968627 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:41.967342    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"]
Nov 01 03:34:42.011437 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:42.011333    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b941e0fe-6b17-4e30-917c-4d3f4382c545-credential-socket\") pod \"e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8\" (UID: \"b941e0fe-6b17-4e30-917c-4d3f4382c545\") " pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"
Nov 01 03:34:42.011437 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:42.011412    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b941e0fe-6b17-4e30-917c-4d3f4382c545-istio-data\") pod \"e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8\" (UID: \"b941e0fe-6b17-4e30-917c-4d3f4382c545\") " pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"
Nov 01 03:34:42.011806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:42.011442    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b941e0fe-6b17-4e30-917c-4d3f4382c545-istio-podinfo\") pod \"e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8\" (UID: \"b941e0fe-6b17-4e30-917c-4d3f4382c545\") " pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"
Nov 01 03:34:42.011806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:42.011470    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b941e0fe-6b17-4e30-917c-4d3f4382c545-istio-token\") pod \"e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8\" (UID: \"b941e0fe-6b17-4e30-917c-4d3f4382c545\") " pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"
Nov 01 03:34:42.011806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:42.011609    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b941e0fe-6b17-4e30-917c-4d3f4382c545-workload-socket\") pod \"e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8\" (UID: \"b941e0fe-6b17-4e30-917c-4d3f4382c545\") " pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"
Nov 01 03:34:42.011806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:42.011665    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfvq5\" (UniqueName: \"kubernetes.io/projected/b941e0fe-6b17-4e30-917c-4d3f4382c545-kube-api-access-nfvq5\") pod \"e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8\" (UID: \"b941e0fe-6b17-4e30-917c-4d3f4382c545\") " pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"
Nov 01 03:34:42.011806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:42.011764    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b941e0fe-6b17-4e30-917c-4d3f4382c545-istio-envoy\") pod \"e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8\" (UID: \"b941e0fe-6b17-4e30-917c-4d3f4382c545\") " pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"
Nov 01 03:34:42.011806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:42.011803    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b941e0fe-6b17-4e30-917c-4d3f4382c545-workload-certs\") pod \"e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8\" (UID: \"b941e0fe-6b17-4e30-917c-4d3f4382c545\") " pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"
Nov 01 03:34:42.012155 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:42.011865    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b941e0fe-6b17-4e30-917c-4d3f4382c545-istiod-ca-cert\") pod \"e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8\" (UID: \"b941e0fe-6b17-4e30-917c-4d3f4382c545\") " pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"
Nov 01 03:34:42.012155 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:42.011915    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b941e0fe-6b17-4e30-917c-4d3f4382c545-credential-socket\") pod \"e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8\" (UID: \"b941e0fe-6b17-4e30-917c-4d3f4382c545\") " pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"
Nov 01 03:34:42.013253 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:42.013225    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b941e0fe-6b17-4e30-917c-4d3f4382c545-istiod-ca-cert\") pod \"e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8\" (UID: \"b941e0fe-6b17-4e30-917c-4d3f4382c545\") " pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"
Nov 01 03:34:42.013419 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:42.013324    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b941e0fe-6b17-4e30-917c-4d3f4382c545-workload-certs\") pod \"e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8\" (UID: \"b941e0fe-6b17-4e30-917c-4d3f4382c545\") " pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"
Nov 01 03:34:42.014928 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:42.014450    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b941e0fe-6b17-4e30-917c-4d3f4382c545-istio-data\") pod \"e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8\" (UID: \"b941e0fe-6b17-4e30-917c-4d3f4382c545\") " pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"
Nov 01 03:34:42.015156 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:42.014667    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b941e0fe-6b17-4e30-917c-4d3f4382c545-workload-socket\") pod \"e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8\" (UID: \"b941e0fe-6b17-4e30-917c-4d3f4382c545\") " pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"
Nov 01 03:34:42.017283 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:42.017258    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b941e0fe-6b17-4e30-917c-4d3f4382c545-istio-podinfo\") pod \"e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8\" (UID: \"b941e0fe-6b17-4e30-917c-4d3f4382c545\") " pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"
Nov 01 03:34:42.018528 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:42.018503    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b941e0fe-6b17-4e30-917c-4d3f4382c545-istio-envoy\") pod \"e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8\" (UID: \"b941e0fe-6b17-4e30-917c-4d3f4382c545\") " pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"
Nov 01 03:34:42.073885 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:42.073849    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b941e0fe-6b17-4e30-917c-4d3f4382c545-istio-token\") pod \"e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8\" (UID: \"b941e0fe-6b17-4e30-917c-4d3f4382c545\") " pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"
Nov 01 03:34:42.074382 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:42.074344    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfvq5\" (UniqueName: \"kubernetes.io/projected/b941e0fe-6b17-4e30-917c-4d3f4382c545-kube-api-access-nfvq5\") pod \"e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8\" (UID: \"b941e0fe-6b17-4e30-917c-4d3f4382c545\") " pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"
Nov 01 03:34:42.205480 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:42.205123    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:42.232061 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:42.231519    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"
Nov 01 03:34:42.720734 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:42.720708    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"]
Nov 01 03:34:42.722303 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:34:42.722265    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb941e0fe_6b17_4e30_917c_4d3f4382c545.slice/crio-a50bd657a5288886ee2fac8513badf68f831be4dd3e5737a760d637e9a29a96c WatchSource:0}: Error finding container a50bd657a5288886ee2fac8513badf68f831be4dd3e5737a760d637e9a29a96c: Status 404 returned error can't find the container with id a50bd657a5288886ee2fac8513badf68f831be4dd3e5737a760d637e9a29a96c
Nov 01 03:34:42.725186 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:42.725145    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:34:42.725291 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:42.725230    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:34:42.725291 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:42.725270    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:34:43.128007 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:43.127961    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8" event={"ID":"b941e0fe-6b17-4e30-917c-4d3f4382c545","Type":"ContainerStarted","Data":"a1681e86692725e01cda7745ec555ae99eb41f125053fd0d010bb18028603f3f"}
Nov 01 03:34:43.128525 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:43.128046    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8" event={"ID":"b941e0fe-6b17-4e30-917c-4d3f4382c545","Type":"ContainerStarted","Data":"a50bd657a5288886ee2fac8513badf68f831be4dd3e5737a760d637e9a29a96c"}
Nov 01 03:34:43.234350 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:43.234322    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"
Nov 01 03:34:43.236251 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:43.236221    4930 patch_prober.go:29] interesting pod/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Nov 01 03:34:43.236392 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:43.236279    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8" podUID="b941e0fe-6b17-4e30-917c-4d3f4382c545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Nov 01 03:34:43.260050 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:43.259997    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8" podStartSLOduration=2.259982136 podStartE2EDuration="2.259982136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:34:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:34:43.257833121 +0000 UTC m=+1687.780839710" watchObservedRunningTime="2026-11-01 03:34:43.259982136 +0000 UTC m=+1687.782988724"
Nov 01 03:34:44.238246 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:44.238217    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"
Nov 01 03:34:45.016067 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:45.016038    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0cb10b-6cb84f4cf5-qkv5p"
Nov 01 03:34:45.016243 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:45.016113    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0cb10b-5597fdc799-qmc8n"
Nov 01 03:34:45.144930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:45.144895    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"
Nov 01 03:34:45.145861 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:45.145838    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"
Nov 01 03:34:45.324016 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:45.323924    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:45.824524 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:45.824485    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:46.100510 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:46.100465    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:46.294269 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:46.294231    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:46.470043 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:46.469956    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:49.278667 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:34:49.278547    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:34:49.589503 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:49.589457    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:49.786078 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:49.785904    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:49.904678 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:49.904531    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb84ee16-5lcr4"]
Nov 01 03:34:49.906043 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:49.906010    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a115577-6d0a-4047-ab8b-8eb78f51acbe" containerName="istio-proxy"
Nov 01 03:34:49.910948 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:49.910925    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb84ee16-5lcr4"
Nov 01 03:34:49.946712 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:49.946678    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb84ee16-5lcr4"]
Nov 01 03:34:50.014581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:50.006455    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8516c639-ba17-4a14-9d2c-74a4b8325829-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb84ee16-5lcr4\" (UID: \"8516c639-ba17-4a14-9d2c-74a4b8325829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb84ee16-5lcr4"
Nov 01 03:34:50.014581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:50.006741    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8516c639-ba17-4a14-9d2c-74a4b8325829-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb84ee16-5lcr4\" (UID: \"8516c639-ba17-4a14-9d2c-74a4b8325829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb84ee16-5lcr4"
Nov 01 03:34:50.014581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:50.006912    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkkjg\" (UniqueName: \"kubernetes.io/projected/8516c639-ba17-4a14-9d2c-74a4b8325829-kube-api-access-zkkjg\") pod \"maas-api-revoke-sub-eb84ee16-5lcr4\" (UID: \"8516c639-ba17-4a14-9d2c-74a4b8325829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb84ee16-5lcr4"
Nov 01 03:34:50.019191 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:50.019166    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:34:50.020185 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:50.019489    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:34:50.029884 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:50.029828    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:34:50.030159 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:50.030141    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:34:50.123583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:50.118709    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8516c639-ba17-4a14-9d2c-74a4b8325829-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb84ee16-5lcr4\" (UID: \"8516c639-ba17-4a14-9d2c-74a4b8325829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb84ee16-5lcr4"
Nov 01 03:34:50.123583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:50.118801    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkkjg\" (UniqueName: \"kubernetes.io/projected/8516c639-ba17-4a14-9d2c-74a4b8325829-kube-api-access-zkkjg\") pod \"maas-api-revoke-sub-eb84ee16-5lcr4\" (UID: \"8516c639-ba17-4a14-9d2c-74a4b8325829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb84ee16-5lcr4"
Nov 01 03:34:50.123583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:50.119083    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8516c639-ba17-4a14-9d2c-74a4b8325829-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb84ee16-5lcr4\" (UID: \"8516c639-ba17-4a14-9d2c-74a4b8325829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb84ee16-5lcr4"
Nov 01 03:34:50.123583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:50.119926    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8516c639-ba17-4a14-9d2c-74a4b8325829-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb84ee16-5lcr4\" (UID: \"8516c639-ba17-4a14-9d2c-74a4b8325829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb84ee16-5lcr4"
Nov 01 03:34:50.153093 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:50.153060    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8516c639-ba17-4a14-9d2c-74a4b8325829-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb84ee16-5lcr4\" (UID: \"8516c639-ba17-4a14-9d2c-74a4b8325829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb84ee16-5lcr4"
Nov 01 03:34:50.191497 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:50.187317    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-686153-76d64ff869-c69fk"]
Nov 01 03:34:50.198403 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:50.197953    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkkjg\" (UniqueName: \"kubernetes.io/projected/8516c639-ba17-4a14-9d2c-74a4b8325829-kube-api-access-zkkjg\") pod \"maas-api-revoke-sub-eb84ee16-5lcr4\" (UID: \"8516c639-ba17-4a14-9d2c-74a4b8325829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb84ee16-5lcr4"
Nov 01 03:34:50.208578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:50.199547    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-686153-76d64ff869-c69fk"
Nov 01 03:34:50.242578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:50.239941    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb84ee16-5lcr4"
Nov 01 03:34:50.264605 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:50.262761    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-686153-dockercfg-6t8n6\""
Nov 01 03:34:50.264605 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:50.262927    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-686153\""
Nov 01 03:34:50.326872 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:50.321259    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-686153-76d64ff869-c69fk"]
Nov 01 03:34:50.326872 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:50.326436    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6bhv\" (UniqueName: \"kubernetes.io/projected/97b6f511-4996-4752-887c-75bee107d98b-kube-api-access-z6bhv\") pod \"payload-processing-e2e-shared-b-w4-686153-76d64ff869-c69fk\" (UID: \"97b6f511-4996-4752-887c-75bee107d98b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-686153-76d64ff869-c69fk"
Nov 01 03:34:50.326872 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:50.326773    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b6f511-4996-4752-887c-75bee107d98b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-686153-76d64ff869-c69fk\" (UID: \"97b6f511-4996-4752-887c-75bee107d98b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-686153-76d64ff869-c69fk"
Nov 01 03:34:50.417235 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:50.417172    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-686153-78c947b5ff-hcd4s"]
Nov 01 03:34:50.440074 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:50.439597    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b6f511-4996-4752-887c-75bee107d98b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-686153-76d64ff869-c69fk\" (UID: \"97b6f511-4996-4752-887c-75bee107d98b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-686153-76d64ff869-c69fk"
Nov 01 03:34:50.440074 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:50.439699    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6bhv\" (UniqueName: \"kubernetes.io/projected/97b6f511-4996-4752-887c-75bee107d98b-kube-api-access-z6bhv\") pod \"payload-processing-e2e-shared-b-w4-686153-76d64ff869-c69fk\" (UID: \"97b6f511-4996-4752-887c-75bee107d98b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-686153-76d64ff869-c69fk"
Nov 01 03:34:50.464152 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:50.464122    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-686153-78c947b5ff-hcd4s"
Nov 01 03:34:50.520190 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:50.473120    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b6f511-4996-4752-887c-75bee107d98b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-686153-76d64ff869-c69fk\" (UID: \"97b6f511-4996-4752-887c-75bee107d98b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-686153-76d64ff869-c69fk"
Nov 01 03:34:50.534155 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:50.533312    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-686153-78c947b5ff-hcd4s"]
Nov 01 03:34:50.546244 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:50.545866    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11d7b556-9fe7-4910-9e18-8f8db5995f80-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-686153-78c947b5ff-hcd4s\" (UID: \"11d7b556-9fe7-4910-9e18-8f8db5995f80\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-686153-78c947b5ff-hcd4s"
Nov 01 03:34:50.546244 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:50.545997    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qbn2\" (UniqueName: \"kubernetes.io/projected/11d7b556-9fe7-4910-9e18-8f8db5995f80-kube-api-access-5qbn2\") pod \"payload-pre-processing-e2e-shared-b-w4-686153-78c947b5ff-hcd4s\" (UID: \"11d7b556-9fe7-4910-9e18-8f8db5995f80\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-686153-78c947b5ff-hcd4s"
Nov 01 03:34:50.560656 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:50.559227    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6bhv\" (UniqueName: \"kubernetes.io/projected/97b6f511-4996-4752-887c-75bee107d98b-kube-api-access-z6bhv\") pod \"payload-processing-e2e-shared-b-w4-686153-76d64ff869-c69fk\" (UID: \"97b6f511-4996-4752-887c-75bee107d98b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-686153-76d64ff869-c69fk"
Nov 01 03:34:50.745002 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:50.744906    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qbn2\" (UniqueName: \"kubernetes.io/projected/11d7b556-9fe7-4910-9e18-8f8db5995f80-kube-api-access-5qbn2\") pod \"payload-pre-processing-e2e-shared-b-w4-686153-78c947b5ff-hcd4s\" (UID: \"11d7b556-9fe7-4910-9e18-8f8db5995f80\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-686153-78c947b5ff-hcd4s"
Nov 01 03:34:50.745185 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:50.745141    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11d7b556-9fe7-4910-9e18-8f8db5995f80-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-686153-78c947b5ff-hcd4s\" (UID: \"11d7b556-9fe7-4910-9e18-8f8db5995f80\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-686153-78c947b5ff-hcd4s"
Nov 01 03:34:50.746045 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:50.746015    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11d7b556-9fe7-4910-9e18-8f8db5995f80-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-686153-78c947b5ff-hcd4s\" (UID: \"11d7b556-9fe7-4910-9e18-8f8db5995f80\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-686153-78c947b5ff-hcd4s"
Nov 01 03:34:50.849576 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:50.840439    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-686153-76d64ff869-c69fk"
Nov 01 03:34:50.947612 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:50.917246    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qbn2\" (UniqueName: \"kubernetes.io/projected/11d7b556-9fe7-4910-9e18-8f8db5995f80-kube-api-access-5qbn2\") pod \"payload-pre-processing-e2e-shared-b-w4-686153-78c947b5ff-hcd4s\" (UID: \"11d7b556-9fe7-4910-9e18-8f8db5995f80\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-686153-78c947b5ff-hcd4s"
Nov 01 03:34:51.050028 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:51.049995    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb84ee16-5lcr4"]
Nov 01 03:34:51.136276 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:51.136251    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-686153-78c947b5ff-hcd4s"
Nov 01 03:34:51.770641 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:51.760350    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb84ee16-5lcr4" event={"ID":"8516c639-ba17-4a14-9d2c-74a4b8325829","Type":"ContainerStarted","Data":"f88836be17d399c3d2bd765e31bb9ef4407b840f597eb4af72441ceb30852ca2"}
Nov 01 03:34:51.983600 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:51.976221    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-686153-76d64ff869-c69fk"]
Nov 01 03:34:52.352584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:52.351878    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:34:52.375983 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:52.375943    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-686153-78c947b5ff-hcd4s"]
Nov 01 03:34:52.763592 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:52.762549    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-686153-78c947b5ff-hcd4s" event={"ID":"11d7b556-9fe7-4910-9e18-8f8db5995f80","Type":"ContainerStarted","Data":"38db2cd143d25e22ffbde90415d0d49a53447b755ca60e3d7e041a3c5ca61eb9"}
Nov 01 03:34:52.763592 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:52.762662    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-686153-78c947b5ff-hcd4s"
Nov 01 03:34:52.763592 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:52.762694    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-686153-78c947b5ff-hcd4s" event={"ID":"11d7b556-9fe7-4910-9e18-8f8db5995f80","Type":"ContainerStarted","Data":"874926fd8f2dba8c99f64ef51d589c5d197d5d235aabac750c98be47249e2c29"}
Nov 01 03:34:52.764482 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:52.764451    4930 generic.go:356] "Generic (PLEG): container finished" podID="8516c639-ba17-4a14-9d2c-74a4b8325829" containerID="770208bb7fb2c282f97ca9f490ee6b863372e8b9ad72db52b3e46f50eb7bf7f2" exitCode=0
Nov 01 03:34:52.764600 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:52.764494    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb84ee16-5lcr4" event={"ID":"8516c639-ba17-4a14-9d2c-74a4b8325829","Type":"ContainerDied","Data":"770208bb7fb2c282f97ca9f490ee6b863372e8b9ad72db52b3e46f50eb7bf7f2"}
Nov 01 03:34:52.766377 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:52.766353    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-686153-76d64ff869-c69fk" event={"ID":"97b6f511-4996-4752-887c-75bee107d98b","Type":"ContainerStarted","Data":"02add003872f691767f15c5f508aebd4144de22a899943be6e2727c2f64efbe8"}
Nov 01 03:34:52.766488 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:52.766400    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-686153-76d64ff869-c69fk" event={"ID":"97b6f511-4996-4752-887c-75bee107d98b","Type":"ContainerStarted","Data":"7e2eed8fec5671ea401ee02a17d60bd00e53fa740d6c7574cf8c61bf930c5674"}
Nov 01 03:34:52.766488 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:52.766454    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-686153-76d64ff869-c69fk"
Nov 01 03:34:52.882725 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:52.882627    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-686153-78c947b5ff-hcd4s" podStartSLOduration=2.882612389 podStartE2EDuration="2.882612389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:34:52.880039779 +0000 UTC m=+1697.403046370" watchObservedRunningTime="2026-11-01 03:34:52.882612389 +0000 UTC m=+1697.405618973"
Nov 01 03:34:53.005951 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:53.005832    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-686153-76d64ff869-c69fk" podStartSLOduration=3.005817276 podStartE2EDuration="3.005817276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:34:52.99848495 +0000 UTC m=+1697.521491539" watchObservedRunningTime="2026-11-01 03:34:53.005817276 +0000 UTC m=+1697.528823861"
Nov 01 03:34:53.954511 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:53.954482    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb84ee16-5lcr4"
Nov 01 03:34:54.118662 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:54.118625    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8516c639-ba17-4a14-9d2c-74a4b8325829-kube-api-access-zkkjg\" (UniqueName: \"kubernetes.io/projected/8516c639-ba17-4a14-9d2c-74a4b8325829-kube-api-access-zkkjg\") pod \"8516c639-ba17-4a14-9d2c-74a4b8325829\" (UID: \"8516c639-ba17-4a14-9d2c-74a4b8325829\") "
Nov 01 03:34:54.118956 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:54.118809    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8516c639-ba17-4a14-9d2c-74a4b8325829-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8516c639-ba17-4a14-9d2c-74a4b8325829-maas-api-xks-ca\") pod \"8516c639-ba17-4a14-9d2c-74a4b8325829\" (UID: \"8516c639-ba17-4a14-9d2c-74a4b8325829\") "
Nov 01 03:34:54.118956 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:54.118856    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8516c639-ba17-4a14-9d2c-74a4b8325829-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8516c639-ba17-4a14-9d2c-74a4b8325829-maas-api-openshift-ca\") pod \"8516c639-ba17-4a14-9d2c-74a4b8325829\" (UID: \"8516c639-ba17-4a14-9d2c-74a4b8325829\") "
Nov 01 03:34:54.119940 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:54.119911    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8516c639-ba17-4a14-9d2c-74a4b8325829-maas-api-openshift-ca" pod "8516c639-ba17-4a14-9d2c-74a4b8325829" (UID: "8516c639-ba17-4a14-9d2c-74a4b8325829"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:34:54.123398 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:54.123363    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8516c639-ba17-4a14-9d2c-74a4b8325829-kube-api-access-zkkjg" pod "8516c639-ba17-4a14-9d2c-74a4b8325829" (UID: "8516c639-ba17-4a14-9d2c-74a4b8325829"). InnerVolumeSpecName "kube-api-access-zkkjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:34:54.127666 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:54.127636    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8516c639-ba17-4a14-9d2c-74a4b8325829-maas-api-xks-ca" pod "8516c639-ba17-4a14-9d2c-74a4b8325829" (UID: "8516c639-ba17-4a14-9d2c-74a4b8325829"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:34:54.221495 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:54.221458    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8516c639-ba17-4a14-9d2c-74a4b8325829-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:54.221495 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:54.221488    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8516c639-ba17-4a14-9d2c-74a4b8325829-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:54.221696 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:54.221515    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkkjg\" (UniqueName: \"kubernetes.io/projected/8516c639-ba17-4a14-9d2c-74a4b8325829-kube-api-access-zkkjg\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:34:54.784689 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:54.784649    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb84ee16-5lcr4" event={"ID":"8516c639-ba17-4a14-9d2c-74a4b8325829","Type":"ContainerDied","Data":"f88836be17d399c3d2bd765e31bb9ef4407b840f597eb4af72441ceb30852ca2"}
Nov 01 03:34:54.784856 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:54.784719    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f88836be17d399c3d2bd765e31bb9ef4407b840f597eb4af72441ceb30852ca2"
Nov 01 03:34:54.784856 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:54.784666    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb84ee16-5lcr4"
Nov 01 03:34:57.366423 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:34:57.366388    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:35:00.568651 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:00.568609    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:35:02.196497 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:02.196460    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:35:03.778346 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:03.778315    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-686153-76d64ff869-c69fk"
Nov 01 03:35:03.778991 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:03.778968    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-686153-78c947b5ff-hcd4s"
Nov 01 03:35:07.043616 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:07.043548    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:35:07.720439 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:07.720408    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:35:11.797324 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:11.797283    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:35:16.703246 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:16.703214    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:35:16.894112 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:16.894052    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:35:19.137016 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:35:19.136977    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:35:21.784120 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:21.784070    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:35:21.989246 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:21.989208    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:35:26.908530 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:26.908493    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:35:31.750167 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:31.750132    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:35:36.225633 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:36.225515    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:35:36.372917 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:36.372869    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:35:36.617341 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:36.617307    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:35:37.187125 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:37.187073    4930 scope.go:122] "RemoveContainer" containerID="d56817ff89bde777468460b44f45665345a8e6069b0a9044463fb9349ae985d5"
Nov 01 03:35:37.216506 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:37.216441    4930 scope.go:122] "RemoveContainer" containerID="8e7db56a5440b1f328b63460da48e4ac3a0ac853b07292ecd5e53e1f0fa3f5aa"
Nov 01 03:35:37.230518 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:37.230489    4930 scope.go:122] "RemoveContainer" containerID="1b0576b8ca23994ea7e4544dedfd0ce9c3e3578b35d9606198628d10c12d7430"
Nov 01 03:35:37.243791 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:37.243765    4930 scope.go:122] "RemoveContainer" containerID="686ed25ff460d0ea4fd03edbc55e4f45996cc6d7df6b3296bb36d147e4dd0532"
Nov 01 03:35:37.254432 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:37.254412    4930 scope.go:122] "RemoveContainer" containerID="6b1a65ce0e89dc3548df9322c44ca33699c9510d5c02424326a17d4e26f5ccc3"
Nov 01 03:35:37.294341 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:37.294300    4930 scope.go:122] "RemoveContainer" containerID="931bac320b3dc51128321b2345535c08786c3efeeac9cae74efa2e481a242ae4"
Nov 01 03:35:37.312663 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:37.312641    4930 scope.go:122] "RemoveContainer" containerID="f9d7b6248d0877439d7cba0ea4a76669b8ac01471d5383a49a02527c784f5186"
Nov 01 03:35:39.171610 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:39.169616    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" podUID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded"
Nov 01 03:35:39.284589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:39.284023    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7" podUID="1ce834cc-d34d-46cc-a081-4d2cc261becb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:35:39.593251 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:39.592596    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm" podUID="b2637bc6-3828-4b75-9783-5415519403a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:35:39.601506 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:39.600298    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp" podUID="667240b0-64a7-45cd-84fe-9cd7e0d8a301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:35:39.799664 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:39.799615    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t" podUID="13930c1e-2200-4412-b8d3-ae3e4126a851" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 01 03:35:39.982809 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:39.982696    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj" podUID="b0e32cb3-ac18-4c1f-819c-f1139a88bfea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:35:40.020783 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:40.020639    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj" podUID="b0e32cb3-ac18-4c1f-819c-f1139a88bfea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:35:40.067749 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:40.066638    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm" podUID="75d142b3-7172-4b57-bc07-a2f404c4656e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:35:40.177824 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:40.177780    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf" podUID="40881383-61bb-4fc8-9c75-f8ffce4834c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:35:40.219729 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:40.219683    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" podUID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:35:40.220352 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:40.220327    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:35:40.294750 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:40.294647    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7" podUID="1ce834cc-d34d-46cc-a081-4d2cc261becb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:35:40.295738 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:40.295713    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"
Nov 01 03:35:40.387817 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:40.387771    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw" podUID="b6d688b8-b911-4a6a-a842-2b0fb7a75e83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:35:40.613581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:40.600929    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp" podUID="667240b0-64a7-45cd-84fe-9cd7e0d8a301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:35:40.613581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:40.609648    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp"
Nov 01 03:35:40.613581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:40.611759    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm" podUID="b2637bc6-3828-4b75-9783-5415519403a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:35:40.613581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:40.612193    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"
Nov 01 03:35:40.699581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:40.679599    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8" podUID="88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:35:40.748597 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:40.723610    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2" podUID="e2483a06-2f4b-4a1e-af4e-a63eb50d29bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:35:40.809819 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:40.807686    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps" podUID="d658772c-b072-4513-a639-d19a94c78525" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:35:40.809819 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:40.808534    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t" podUID="13930c1e-2200-4412-b8d3-ae3e4126a851" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 01 03:35:40.809819 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:40.808897    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:35:41.009642 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:41.008653    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj" podUID="b0e32cb3-ac18-4c1f-819c-f1139a88bfea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:35:41.038017 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:41.037965    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:35:41.224579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:41.221267    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" podUID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded"
Nov 01 03:35:41.247721 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:41.247659    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n" podUID="d2511772-66f9-40a0-9166-87c4345485c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Nov 01 03:35:41.251475 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:41.251439    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf" podUID="40881383-61bb-4fc8-9c75-f8ffce4834c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:35:41.271043 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:41.270974    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:35:41.393584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:41.272634    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm" podUID="75d142b3-7172-4b57-bc07-a2f404c4656e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": context deadline exceeded"
Nov 01 03:35:41.393584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:41.272657    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n" podUID="d2511772-66f9-40a0-9166-87c4345485c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": context deadline exceeded"
Nov 01 03:35:41.393584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:41.272863    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:35:41.393584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:41.296965    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7" podUID="1ce834cc-d34d-46cc-a081-4d2cc261becb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Nov 01 03:35:41.406899 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:41.391872    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw" podUID="b6d688b8-b911-4a6a-a842-2b0fb7a75e83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:35:41.406899 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:41.400704    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:35:41.633063 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:41.630451    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2" podUID="e2483a06-2f4b-4a1e-af4e-a63eb50d29bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:35:41.633063 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:41.632420    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm" podUID="b2637bc6-3828-4b75-9783-5415519403a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:35:41.633063 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:41.632511    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp" podUID="667240b0-64a7-45cd-84fe-9cd7e0d8a301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Nov 01 03:35:41.705629 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:41.684170    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8" podUID="88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Nov 01 03:35:41.742425 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:41.740826    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2" podUID="e2483a06-2f4b-4a1e-af4e-a63eb50d29bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": context deadline exceeded"
Nov 01 03:35:41.956648 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:41.941993    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:35:42.149069 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:42.149014    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t" podUID="13930c1e-2200-4412-b8d3-ae3e4126a851" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:35:42.149331 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:42.149314    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"
Nov 01 03:35:42.149784 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:42.063914    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:35:42.174719 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:42.174645    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps" podUID="d658772c-b072-4513-a639-d19a94c78525" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:35:42.194592 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:42.180480    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:35:42.194592 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:42.180999    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj" podUID="b0e32cb3-ac18-4c1f-819c-f1139a88bfea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:35:42.240028 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:42.239926    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" podUID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded"
Nov 01 03:35:42.252887 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:42.252827    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n" podUID="d2511772-66f9-40a0-9166-87c4345485c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:35:42.253487 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:42.253466    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"
Nov 01 03:35:42.272692 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:42.272654    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf" podUID="40881383-61bb-4fc8-9c75-f8ffce4834c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:35:42.273259 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:42.273229    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm" podUID="75d142b3-7172-4b57-bc07-a2f404c4656e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": context deadline exceeded"
Nov 01 03:35:42.298745 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:42.298708    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7" podUID="1ce834cc-d34d-46cc-a081-4d2cc261becb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:35:42.547463 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:42.547292    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw" podUID="b6d688b8-b911-4a6a-a842-2b0fb7a75e83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 01 03:35:42.633901 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:42.632648    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm" podUID="b2637bc6-3828-4b75-9783-5415519403a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:35:42.633901 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:42.633204    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp" podUID="667240b0-64a7-45cd-84fe-9cd7e0d8a301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:35:42.693654 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:42.693607    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8" podUID="88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:35:42.733891 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:42.733851    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:35:42.742754 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:42.742721    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2" podUID="e2483a06-2f4b-4a1e-af4e-a63eb50d29bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": context deadline exceeded"
Nov 01 03:35:42.839740 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:42.830669    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp"
Nov 01 03:35:42.839740 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:42.832093    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"
Nov 01 03:35:42.839740 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:42.833645    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:35:42.839740 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:42.834391    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"
Nov 01 03:35:42.839740 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:42.835210    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:35:42.839740 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:42.836098    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:35:42.853573 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:42.841579    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:35:42.869450 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:42.868173    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"
Nov 01 03:35:42.877070 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:42.876016    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:35:42.916590 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:42.914847    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:35:42.946917 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:42.945330    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:35:43.023162 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:43.023120    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"
Nov 01 03:35:45.631576 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:45.630826    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:35:46.270617 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:46.269659    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:35:46.866683 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:46.859947    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:35:47.018583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:47.017757    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:35:47.244669 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:47.244537    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:35:47.452542 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:47.452505    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:35:47.739461 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:47.738817    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:35:49.122591 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:35:49.122538    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:35:49.775540 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:49.775432    4930 generic.go:356] "Generic (PLEG): container finished" podID="cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20" containerID="c2d5829d14a12ad87679cd55ed38ee14bcc992364fd70ad5c1245017a0bd6315" exitCode=0
Nov 01 03:35:49.775767 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:49.775743    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9" event={"ID":"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20","Type":"ContainerDied","Data":"c2d5829d14a12ad87679cd55ed38ee14bcc992364fd70ad5c1245017a0bd6315"}
Nov 01 03:35:49.776304 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:49.776281    4930 scope.go:122] "RemoveContainer" containerID="c2d5829d14a12ad87679cd55ed38ee14bcc992364fd70ad5c1245017a0bd6315"
Nov 01 03:35:49.779349 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:49.779311    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:35:49.779485 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:49.779406    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:35:49.779568 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:49.779487    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:35:49.788240 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:49.787105    4930 generic.go:356] "Generic (PLEG): container finished" podID="a4cae562-15e5-4d0b-bcef-62ca774e2369" containerID="de5005bcce6c56f7eea7f9d783b30dcf09982aa9a56cc44a65a2306b807a57be" exitCode=0
Nov 01 03:35:49.788240 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:49.787159    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh" event={"ID":"a4cae562-15e5-4d0b-bcef-62ca774e2369","Type":"ContainerDied","Data":"de5005bcce6c56f7eea7f9d783b30dcf09982aa9a56cc44a65a2306b807a57be"}
Nov 01 03:35:49.788240 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:49.787728    4930 scope.go:122] "RemoveContainer" containerID="de5005bcce6c56f7eea7f9d783b30dcf09982aa9a56cc44a65a2306b807a57be"
Nov 01 03:35:49.792717 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:49.792677    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:35:49.792851 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:49.792774    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:35:49.792851 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:49.792820    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:35:50.801576 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:50.801509    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh" event={"ID":"a4cae562-15e5-4d0b-bcef-62ca774e2369","Type":"ContainerStarted","Data":"f94ccf4edcf71b1b502e4937f17e375337dffceb203a49f96738a5857c35eeb4"}
Nov 01 03:35:50.805595 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:50.804533    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9" event={"ID":"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20","Type":"ContainerStarted","Data":"71f46d6bf0f05d45d4c2965a70f7474e8cb6cd51eb63322d5531ca89d4ce2907"}
Nov 01 03:35:51.193579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:51.192659    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"
Nov 01 03:35:51.244739 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:51.244704    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:35:51.407582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:51.391808    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"
Nov 01 03:35:51.407582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:51.394770    4930 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Nov 01 03:35:51.407582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:51.394822    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh" podUID="a4cae562-15e5-4d0b-bcef-62ca774e2369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Nov 01 03:35:51.478897 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:51.478753    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"
Nov 01 03:35:51.505713 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:51.505672    4930 patch_prober.go:29] interesting pod/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Nov 01 03:35:51.505929 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:51.505737    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9" podUID="cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Nov 01 03:35:51.527458 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:51.527417    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:35:51.818644 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:51.818088    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"
Nov 01 03:35:52.119568 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:52.119522    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:35:52.393637 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:52.390080    4930 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Nov 01 03:35:52.393637 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:52.390159    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh" podUID="a4cae562-15e5-4d0b-bcef-62ca774e2369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Nov 01 03:35:52.468579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:52.462998    4930 patch_prober.go:29] interesting pod/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Nov 01 03:35:52.468579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:52.463075    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9" podUID="cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Nov 01 03:35:53.391358 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:53.391323    4930 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Nov 01 03:35:53.391999 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:53.391967    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh" podUID="a4cae562-15e5-4d0b-bcef-62ca774e2369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Nov 01 03:35:53.478783 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:53.478661    4930 patch_prober.go:29] interesting pod/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Nov 01 03:35:53.478783 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:53.478723    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9" podUID="cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Nov 01 03:35:54.054071 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:54.054040    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2b25020-m2ql9"]
Nov 01 03:35:54.124325 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:54.124274    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2b25020-m2ql9"]
Nov 01 03:35:54.391592 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:54.390932    4930 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Nov 01 03:35:54.391592 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:54.391000    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh" podUID="a4cae562-15e5-4d0b-bcef-62ca774e2369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Nov 01 03:35:54.498580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:54.480913    4930 patch_prober.go:29] interesting pod/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Nov 01 03:35:54.498580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:54.480983    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9" podUID="cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Nov 01 03:35:55.333580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:55.333020    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:35:55.424581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:55.424343    4930 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Nov 01 03:35:55.424581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:55.424404    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh" podUID="a4cae562-15e5-4d0b-bcef-62ca774e2369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Nov 01 03:35:55.559293 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:55.559255    4930 patch_prober.go:29] interesting pod/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Nov 01 03:35:55.579949 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:55.579909    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9" podUID="cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Nov 01 03:35:55.748467 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:55.748261    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:35:55.846029 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:55.845968    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebdbcdf4-a27e-4735-8d69-39d0d0df4b29" path="/var/lib/kubelet/pods/ebdbcdf4-a27e-4735-8d69-39d0d0df4b29/volumes"
Nov 01 03:35:55.846894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:55.846857    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f2383c-64bdcdb46c-2z655"]
Nov 01 03:35:55.848778 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:55.848751    4930 cpu_manager.go:414] "containerMap: removing container" podUID="8516c639-ba17-4a14-9d2c-74a4b8325829" containerName="revoke-keys"
Nov 01 03:35:55.848922 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:55.848906    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8516c639-ba17-4a14-9d2c-74a4b8325829" containerName="revoke-keys"
Nov 01 03:35:55.849288 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:55.849266    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="8516c639-ba17-4a14-9d2c-74a4b8325829" containerName="revoke-keys"
Nov 01 03:35:55.852755 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:55.852635    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f2383c-64bdcdb46c-2z655"]
Nov 01 03:35:55.853837 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:55.853219    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f2383c-64bdcdb46c-2z655"
Nov 01 03:35:55.940106 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:55.935610    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f2383c-7959b95765-bjksj"]
Nov 01 03:35:55.959777 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:55.956867    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f2383c\""
Nov 01 03:35:55.959777 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:55.957002    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f2383c-dockercfg-58cf4\""
Nov 01 03:35:56.010635 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:56.010532    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f2383c-7959b95765-bjksj"
Nov 01 03:35:56.037586 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:56.034734    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbfgh\" (UniqueName: \"kubernetes.io/projected/58a93bc7-186b-4bfc-970c-045241a5cfd8-kube-api-access-pbfgh\") pod \"payload-pre-processing-e2e-api-a-f2383c-64bdcdb46c-2z655\" (UID: \"58a93bc7-186b-4bfc-970c-045241a5cfd8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f2383c-64bdcdb46c-2z655"
Nov 01 03:35:56.037586 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:56.034798    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58a93bc7-186b-4bfc-970c-045241a5cfd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f2383c-64bdcdb46c-2z655\" (UID: \"58a93bc7-186b-4bfc-970c-045241a5cfd8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f2383c-64bdcdb46c-2z655"
Nov 01 03:35:56.111652 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:56.111614    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f2383c-7959b95765-bjksj"]
Nov 01 03:35:56.141623 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:56.141244    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c62951f-01c4-4c25-9267-d741fb752efd-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f2383c-7959b95765-bjksj\" (UID: \"2c62951f-01c4-4c25-9267-d741fb752efd\") " pod="openshift-ingress/payload-processing-e2e-api-a-f2383c-7959b95765-bjksj"
Nov 01 03:35:56.141623 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:56.141330    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbfgh\" (UniqueName: \"kubernetes.io/projected/58a93bc7-186b-4bfc-970c-045241a5cfd8-kube-api-access-pbfgh\") pod \"payload-pre-processing-e2e-api-a-f2383c-64bdcdb46c-2z655\" (UID: \"58a93bc7-186b-4bfc-970c-045241a5cfd8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f2383c-64bdcdb46c-2z655"
Nov 01 03:35:56.141623 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:56.141367    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp87g\" (UniqueName: \"kubernetes.io/projected/2c62951f-01c4-4c25-9267-d741fb752efd-kube-api-access-zp87g\") pod \"payload-processing-e2e-api-a-f2383c-7959b95765-bjksj\" (UID: \"2c62951f-01c4-4c25-9267-d741fb752efd\") " pod="openshift-ingress/payload-processing-e2e-api-a-f2383c-7959b95765-bjksj"
Nov 01 03:35:56.141623 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:56.141406    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58a93bc7-186b-4bfc-970c-045241a5cfd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f2383c-64bdcdb46c-2z655\" (UID: \"58a93bc7-186b-4bfc-970c-045241a5cfd8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f2383c-64bdcdb46c-2z655"
Nov 01 03:35:56.162764 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:56.142531    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58a93bc7-186b-4bfc-970c-045241a5cfd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f2383c-64bdcdb46c-2z655\" (UID: \"58a93bc7-186b-4bfc-970c-045241a5cfd8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f2383c-64bdcdb46c-2z655"
Nov 01 03:35:56.239618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:56.235945    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbfgh\" (UniqueName: \"kubernetes.io/projected/58a93bc7-186b-4bfc-970c-045241a5cfd8-kube-api-access-pbfgh\") pod \"payload-pre-processing-e2e-api-a-f2383c-64bdcdb46c-2z655\" (UID: \"58a93bc7-186b-4bfc-970c-045241a5cfd8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f2383c-64bdcdb46c-2z655"
Nov 01 03:35:56.252039 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:56.246974    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c62951f-01c4-4c25-9267-d741fb752efd-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f2383c-7959b95765-bjksj\" (UID: \"2c62951f-01c4-4c25-9267-d741fb752efd\") " pod="openshift-ingress/payload-processing-e2e-api-a-f2383c-7959b95765-bjksj"
Nov 01 03:35:56.252039 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:56.247037    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp87g\" (UniqueName: \"kubernetes.io/projected/2c62951f-01c4-4c25-9267-d741fb752efd-kube-api-access-zp87g\") pod \"payload-processing-e2e-api-a-f2383c-7959b95765-bjksj\" (UID: \"2c62951f-01c4-4c25-9267-d741fb752efd\") " pod="openshift-ingress/payload-processing-e2e-api-a-f2383c-7959b95765-bjksj"
Nov 01 03:35:56.252039 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:56.248136    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c62951f-01c4-4c25-9267-d741fb752efd-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f2383c-7959b95765-bjksj\" (UID: \"2c62951f-01c4-4c25-9267-d741fb752efd\") " pod="openshift-ingress/payload-processing-e2e-api-a-f2383c-7959b95765-bjksj"
Nov 01 03:35:56.286638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:56.285834    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f2383c-64bdcdb46c-2z655"
Nov 01 03:35:56.337957 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:56.337058    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp87g\" (UniqueName: \"kubernetes.io/projected/2c62951f-01c4-4c25-9267-d741fb752efd-kube-api-access-zp87g\") pod \"payload-processing-e2e-api-a-f2383c-7959b95765-bjksj\" (UID: \"2c62951f-01c4-4c25-9267-d741fb752efd\") " pod="openshift-ingress/payload-processing-e2e-api-a-f2383c-7959b95765-bjksj"
Nov 01 03:35:56.408450 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:56.408136    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f2383c-7959b95765-bjksj"
Nov 01 03:35:56.412125 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:56.411681    4930 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Nov 01 03:35:56.412125 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:56.411735    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh" podUID="a4cae562-15e5-4d0b-bcef-62ca774e2369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Nov 01 03:35:56.498627 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:56.496730    4930 patch_prober.go:29] interesting pod/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Nov 01 03:35:56.498627 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:56.496811    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9" podUID="cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Nov 01 03:35:57.055315 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:57.055282    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f2383c-7959b95765-bjksj"]
Nov 01 03:35:57.055441 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:35:57.055284    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c62951f_01c4_4c25_9267_d741fb752efd.slice/crio-b01ba786fa8bf8789683a7b137c4c6098d010ab5e8396d7e6258e63cf3311ce6 WatchSource:0}: Error finding container b01ba786fa8bf8789683a7b137c4c6098d010ab5e8396d7e6258e63cf3311ce6: Status 404 returned error can't find the container with id b01ba786fa8bf8789683a7b137c4c6098d010ab5e8396d7e6258e63cf3311ce6
Nov 01 03:35:57.111678 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:57.111648    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f2383c-64bdcdb46c-2z655"]
Nov 01 03:35:57.279256 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:57.279219    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:35:57.395495 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:57.395459    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"
Nov 01 03:35:57.396649 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:57.396624    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"
Nov 01 03:35:57.468180 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:57.468151    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"
Nov 01 03:35:57.469326 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:57.469303    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"
Nov 01 03:35:57.484146 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:57.484110    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:35:57.627214 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:57.627178    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:35:57.947001 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:57.946966    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f2383c-7959b95765-bjksj" event={"ID":"2c62951f-01c4-4c25-9267-d741fb752efd","Type":"ContainerStarted","Data":"f6de633e509a57caaf2d74d3d2ce96203f0cfba5fdf0169d0f636db4796f742b"}
Nov 01 03:35:57.947222 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:57.947029    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f2383c-7959b95765-bjksj" event={"ID":"2c62951f-01c4-4c25-9267-d741fb752efd","Type":"ContainerStarted","Data":"b01ba786fa8bf8789683a7b137c4c6098d010ab5e8396d7e6258e63cf3311ce6"}
Nov 01 03:35:57.947222 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:57.947063    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-f2383c-7959b95765-bjksj"
Nov 01 03:35:57.948898 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:57.948872    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f2383c-64bdcdb46c-2z655" event={"ID":"58a93bc7-186b-4bfc-970c-045241a5cfd8","Type":"ContainerStarted","Data":"d1015b6878c59d80bf7598e648a3f9dce92c8bc1d52937216c4e3f6eea8ed69d"}
Nov 01 03:35:57.949048 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:57.948912    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f2383c-64bdcdb46c-2z655" event={"ID":"58a93bc7-186b-4bfc-970c-045241a5cfd8","Type":"ContainerStarted","Data":"d4ae2380a26fec95adf02ea42c362abe81ea49f8a7b4371030fc033545260bca"}
Nov 01 03:35:57.949115 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:57.949050    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f2383c-64bdcdb46c-2z655"
Nov 01 03:35:58.071937 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:58.071881    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-f2383c-7959b95765-bjksj" podStartSLOduration=3.071862463 podStartE2EDuration="3.071862463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:35:58.070193212 +0000 UTC m=+1762.593199801" watchObservedRunningTime="2026-11-01 03:35:58.071862463 +0000 UTC m=+1762.594869053"
Nov 01 03:35:58.195638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:35:58.193311    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f2383c-64bdcdb46c-2z655" podStartSLOduration=3.193289912 podStartE2EDuration="3.193289912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:35:58.190492014 +0000 UTC m=+1762.713498602" watchObservedRunningTime="2026-11-01 03:35:58.193289912 +0000 UTC m=+1762.716296503"
Nov 01 03:36:00.915761 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:00.915729    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:36:01.965090 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:01.965048    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0d70ec1-2kwvl"]
Nov 01 03:36:01.968219 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:01.968200    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d70ec1-2kwvl"
Nov 01 03:36:02.023072 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:02.023050    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:36:02.067723 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:02.067690    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d15cd3ef-c4ab-4b6a-bb53-61d3557b92de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0d70ec1-2kwvl\" (UID: \"d15cd3ef-c4ab-4b6a-bb53-61d3557b92de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d70ec1-2kwvl"
Nov 01 03:36:02.067902 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:02.067728    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d15cd3ef-c4ab-4b6a-bb53-61d3557b92de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0d70ec1-2kwvl\" (UID: \"d15cd3ef-c4ab-4b6a-bb53-61d3557b92de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d70ec1-2kwvl"
Nov 01 03:36:02.067948 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:02.067924    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gc8r\" (UniqueName: \"kubernetes.io/projected/d15cd3ef-c4ab-4b6a-bb53-61d3557b92de-kube-api-access-9gc8r\") pod \"maas-api-revoke-sub-e0d70ec1-2kwvl\" (UID: \"d15cd3ef-c4ab-4b6a-bb53-61d3557b92de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d70ec1-2kwvl"
Nov 01 03:36:02.072513 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:02.072492    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:36:02.072837 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:02.072814    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:36:02.072891 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:02.072856    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:36:02.076797 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:02.076779    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0d70ec1-2kwvl"]
Nov 01 03:36:02.169598 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:02.169526    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d15cd3ef-c4ab-4b6a-bb53-61d3557b92de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0d70ec1-2kwvl\" (UID: \"d15cd3ef-c4ab-4b6a-bb53-61d3557b92de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d70ec1-2kwvl"
Nov 01 03:36:02.169782 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:02.169619    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d15cd3ef-c4ab-4b6a-bb53-61d3557b92de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0d70ec1-2kwvl\" (UID: \"d15cd3ef-c4ab-4b6a-bb53-61d3557b92de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d70ec1-2kwvl"
Nov 01 03:36:02.169866 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:02.169785    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gc8r\" (UniqueName: \"kubernetes.io/projected/d15cd3ef-c4ab-4b6a-bb53-61d3557b92de-kube-api-access-9gc8r\") pod \"maas-api-revoke-sub-e0d70ec1-2kwvl\" (UID: \"d15cd3ef-c4ab-4b6a-bb53-61d3557b92de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d70ec1-2kwvl"
Nov 01 03:36:02.170228 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:02.170206    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d15cd3ef-c4ab-4b6a-bb53-61d3557b92de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0d70ec1-2kwvl\" (UID: \"d15cd3ef-c4ab-4b6a-bb53-61d3557b92de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d70ec1-2kwvl"
Nov 01 03:36:02.172140 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:02.172119    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d15cd3ef-c4ab-4b6a-bb53-61d3557b92de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0d70ec1-2kwvl\" (UID: \"d15cd3ef-c4ab-4b6a-bb53-61d3557b92de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d70ec1-2kwvl"
Nov 01 03:36:02.230413 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:02.230354    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gc8r\" (UniqueName: \"kubernetes.io/projected/d15cd3ef-c4ab-4b6a-bb53-61d3557b92de-kube-api-access-9gc8r\") pod \"maas-api-revoke-sub-e0d70ec1-2kwvl\" (UID: \"d15cd3ef-c4ab-4b6a-bb53-61d3557b92de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d70ec1-2kwvl"
Nov 01 03:36:02.281659 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:02.281604    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d70ec1-2kwvl"
Nov 01 03:36:02.664797 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:36:02.664768    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd15cd3ef_c4ab_4b6a_bb53_61d3557b92de.slice/crio-8183fce0ee88ee4eb75c2c9272909b12d416cbf541cbbe05c5ac513ed13b251f WatchSource:0}: Error finding container 8183fce0ee88ee4eb75c2c9272909b12d416cbf541cbbe05c5ac513ed13b251f: Status 404 returned error can't find the container with id 8183fce0ee88ee4eb75c2c9272909b12d416cbf541cbbe05c5ac513ed13b251f
Nov 01 03:36:02.665060 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:02.665035    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0d70ec1-2kwvl"]
Nov 01 03:36:02.909636 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:02.909600    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:36:03.008875 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:03.008841    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d70ec1-2kwvl" event={"ID":"d15cd3ef-c4ab-4b6a-bb53-61d3557b92de","Type":"ContainerStarted","Data":"244a5506ec20e7a6907fa7e303a54d16f6d4076b95af5826c92cb350092fe42d"}
Nov 01 03:36:03.009237 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:03.008904    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d70ec1-2kwvl" event={"ID":"d15cd3ef-c4ab-4b6a-bb53-61d3557b92de","Type":"ContainerStarted","Data":"8183fce0ee88ee4eb75c2c9272909b12d416cbf541cbbe05c5ac513ed13b251f"}
Nov 01 03:36:03.126356 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:03.126308    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d70ec1-2kwvl" podStartSLOduration=2.12629474 podStartE2EDuration="2.12629474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:36:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:36:03.123247327 +0000 UTC m=+1767.646253915" watchObservedRunningTime="2026-11-01 03:36:03.12629474 +0000 UTC m=+1767.649301327"
Nov 01 03:36:04.017549 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:04.017466    4930 generic.go:356] "Generic (PLEG): container finished" podID="d15cd3ef-c4ab-4b6a-bb53-61d3557b92de" containerID="244a5506ec20e7a6907fa7e303a54d16f6d4076b95af5826c92cb350092fe42d" exitCode=0
Nov 01 03:36:04.017549 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:04.017530    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d70ec1-2kwvl" event={"ID":"d15cd3ef-c4ab-4b6a-bb53-61d3557b92de","Type":"ContainerDied","Data":"244a5506ec20e7a6907fa7e303a54d16f6d4076b95af5826c92cb350092fe42d"}
Nov 01 03:36:05.209759 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:05.209731    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d70ec1-2kwvl"
Nov 01 03:36:05.312665 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:05.312601    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d15cd3ef-c4ab-4b6a-bb53-61d3557b92de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d15cd3ef-c4ab-4b6a-bb53-61d3557b92de-maas-api-xks-ca\") pod \"d15cd3ef-c4ab-4b6a-bb53-61d3557b92de\" (UID: \"d15cd3ef-c4ab-4b6a-bb53-61d3557b92de\") "
Nov 01 03:36:05.312799 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:05.312732    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d15cd3ef-c4ab-4b6a-bb53-61d3557b92de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d15cd3ef-c4ab-4b6a-bb53-61d3557b92de-maas-api-openshift-ca\") pod \"d15cd3ef-c4ab-4b6a-bb53-61d3557b92de\" (UID: \"d15cd3ef-c4ab-4b6a-bb53-61d3557b92de\") "
Nov 01 03:36:05.312903 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:05.312886    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d15cd3ef-c4ab-4b6a-bb53-61d3557b92de-kube-api-access-9gc8r\" (UniqueName: \"kubernetes.io/projected/d15cd3ef-c4ab-4b6a-bb53-61d3557b92de-kube-api-access-9gc8r\") pod \"d15cd3ef-c4ab-4b6a-bb53-61d3557b92de\" (UID: \"d15cd3ef-c4ab-4b6a-bb53-61d3557b92de\") "
Nov 01 03:36:05.313070 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:05.313048    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d15cd3ef-c4ab-4b6a-bb53-61d3557b92de-maas-api-openshift-ca" pod "d15cd3ef-c4ab-4b6a-bb53-61d3557b92de" (UID: "d15cd3ef-c4ab-4b6a-bb53-61d3557b92de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:36:05.313342 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:05.313324    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d15cd3ef-c4ab-4b6a-bb53-61d3557b92de-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:36:05.314740 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:05.314713    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d15cd3ef-c4ab-4b6a-bb53-61d3557b92de-maas-api-xks-ca" pod "d15cd3ef-c4ab-4b6a-bb53-61d3557b92de" (UID: "d15cd3ef-c4ab-4b6a-bb53-61d3557b92de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:36:05.315134 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:05.315110    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d15cd3ef-c4ab-4b6a-bb53-61d3557b92de-kube-api-access-9gc8r" pod "d15cd3ef-c4ab-4b6a-bb53-61d3557b92de" (UID: "d15cd3ef-c4ab-4b6a-bb53-61d3557b92de"). InnerVolumeSpecName "kube-api-access-9gc8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:36:05.414327 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:05.414294    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d15cd3ef-c4ab-4b6a-bb53-61d3557b92de-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:36:05.414486 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:05.414344    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gc8r\" (UniqueName: \"kubernetes.io/projected/d15cd3ef-c4ab-4b6a-bb53-61d3557b92de-kube-api-access-9gc8r\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:36:06.036259 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:06.036177    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d70ec1-2kwvl"
Nov 01 03:36:06.036259 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:06.036211    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d70ec1-2kwvl" event={"ID":"d15cd3ef-c4ab-4b6a-bb53-61d3557b92de","Type":"ContainerDied","Data":"8183fce0ee88ee4eb75c2c9272909b12d416cbf541cbbe05c5ac513ed13b251f"}
Nov 01 03:36:06.036452 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:06.036267    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8183fce0ee88ee4eb75c2c9272909b12d416cbf541cbbe05c5ac513ed13b251f"
Nov 01 03:36:07.632927 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:07.632894    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:36:08.970125 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:08.970098    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f2383c-64bdcdb46c-2z655"
Nov 01 03:36:08.970513 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:08.970166    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-f2383c-7959b95765-bjksj"
Nov 01 03:36:12.513893 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:12.513849    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:36:13.839294 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:13.839256    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:36:14.035355 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:14.035313    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:36:17.274209 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:17.274171    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:36:19.123771 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:36:19.123736    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:36:19.752952 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:19.752903    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:36:22.151534 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:22.151495    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:36:26.998431 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:26.998387    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:36:31.120083 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:31.120046    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:36:31.659366 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:31.659330    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:36:31.953701 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:31.953598    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:36:32.042010 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:32.041955    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"]
Nov 01 03:36:32.042740 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:32.042718    4930 cpu_manager.go:414] "containerMap: removing container" podUID="d15cd3ef-c4ab-4b6a-bb53-61d3557b92de" containerName="revoke-keys"
Nov 01 03:36:32.042740 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:32.042740    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d15cd3ef-c4ab-4b6a-bb53-61d3557b92de" containerName="revoke-keys"
Nov 01 03:36:32.042902 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:32.042870    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="d15cd3ef-c4ab-4b6a-bb53-61d3557b92de" containerName="revoke-keys"
Nov 01 03:36:32.045823 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:32.045803    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"
Nov 01 03:36:32.066304 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:32.066275    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baec5da4-406d-4923-aa69-69820f5e04c0-istiod-ca-cert\") pod \"e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8\" (UID: \"baec5da4-406d-4923-aa69-69820f5e04c0\") " pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"
Nov 01 03:36:32.066433 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:32.066378    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baec5da4-406d-4923-aa69-69820f5e04c0-istio-data\") pod \"e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8\" (UID: \"baec5da4-406d-4923-aa69-69820f5e04c0\") " pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"
Nov 01 03:36:32.066433 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:32.066417    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kqp8\" (UniqueName: \"kubernetes.io/projected/baec5da4-406d-4923-aa69-69820f5e04c0-kube-api-access-9kqp8\") pod \"e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8\" (UID: \"baec5da4-406d-4923-aa69-69820f5e04c0\") " pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"
Nov 01 03:36:32.066547 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:32.066453    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baec5da4-406d-4923-aa69-69820f5e04c0-istio-podinfo\") pod \"e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8\" (UID: \"baec5da4-406d-4923-aa69-69820f5e04c0\") " pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"
Nov 01 03:36:32.066547 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:32.066524    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baec5da4-406d-4923-aa69-69820f5e04c0-workload-socket\") pod \"e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8\" (UID: \"baec5da4-406d-4923-aa69-69820f5e04c0\") " pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"
Nov 01 03:36:32.066767 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:32.066746    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baec5da4-406d-4923-aa69-69820f5e04c0-workload-certs\") pod \"e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8\" (UID: \"baec5da4-406d-4923-aa69-69820f5e04c0\") " pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"
Nov 01 03:36:32.066830 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:32.066784    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baec5da4-406d-4923-aa69-69820f5e04c0-istio-envoy\") pod \"e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8\" (UID: \"baec5da4-406d-4923-aa69-69820f5e04c0\") " pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"
Nov 01 03:36:32.066893 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:32.066867    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baec5da4-406d-4923-aa69-69820f5e04c0-credential-socket\") pod \"e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8\" (UID: \"baec5da4-406d-4923-aa69-69820f5e04c0\") " pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"
Nov 01 03:36:32.066963 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:32.066943    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baec5da4-406d-4923-aa69-69820f5e04c0-istio-token\") pod \"e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8\" (UID: \"baec5da4-406d-4923-aa69-69820f5e04c0\") " pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"
Nov 01 03:36:32.116956 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:32.113350    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-7cd000-openshift-default-dockercfg-2pqxd\""
Nov 01 03:36:32.227674 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:32.205966    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baec5da4-406d-4923-aa69-69820f5e04c0-credential-socket\") pod \"e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8\" (UID: \"baec5da4-406d-4923-aa69-69820f5e04c0\") " pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"
Nov 01 03:36:32.227674 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:32.206053    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baec5da4-406d-4923-aa69-69820f5e04c0-istio-token\") pod \"e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8\" (UID: \"baec5da4-406d-4923-aa69-69820f5e04c0\") " pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"
Nov 01 03:36:32.227674 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:32.206189    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baec5da4-406d-4923-aa69-69820f5e04c0-istiod-ca-cert\") pod \"e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8\" (UID: \"baec5da4-406d-4923-aa69-69820f5e04c0\") " pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"
Nov 01 03:36:32.227674 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:32.206252    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baec5da4-406d-4923-aa69-69820f5e04c0-istio-data\") pod \"e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8\" (UID: \"baec5da4-406d-4923-aa69-69820f5e04c0\") " pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"
Nov 01 03:36:32.227674 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:32.206299    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kqp8\" (UniqueName: \"kubernetes.io/projected/baec5da4-406d-4923-aa69-69820f5e04c0-kube-api-access-9kqp8\") pod \"e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8\" (UID: \"baec5da4-406d-4923-aa69-69820f5e04c0\") " pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"
Nov 01 03:36:32.227674 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:32.206358    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baec5da4-406d-4923-aa69-69820f5e04c0-istio-podinfo\") pod \"e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8\" (UID: \"baec5da4-406d-4923-aa69-69820f5e04c0\") " pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"
Nov 01 03:36:32.227674 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:32.206391    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baec5da4-406d-4923-aa69-69820f5e04c0-workload-socket\") pod \"e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8\" (UID: \"baec5da4-406d-4923-aa69-69820f5e04c0\") " pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"
Nov 01 03:36:32.227674 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:32.206508    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baec5da4-406d-4923-aa69-69820f5e04c0-workload-certs\") pod \"e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8\" (UID: \"baec5da4-406d-4923-aa69-69820f5e04c0\") " pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"
Nov 01 03:36:32.227674 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:32.206534    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baec5da4-406d-4923-aa69-69820f5e04c0-istio-envoy\") pod \"e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8\" (UID: \"baec5da4-406d-4923-aa69-69820f5e04c0\") " pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"
Nov 01 03:36:32.227674 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:32.208662    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baec5da4-406d-4923-aa69-69820f5e04c0-istio-data\") pod \"e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8\" (UID: \"baec5da4-406d-4923-aa69-69820f5e04c0\") " pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"
Nov 01 03:36:32.227674 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:32.208949    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baec5da4-406d-4923-aa69-69820f5e04c0-credential-socket\") pod \"e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8\" (UID: \"baec5da4-406d-4923-aa69-69820f5e04c0\") " pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"
Nov 01 03:36:32.227674 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:32.209897    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baec5da4-406d-4923-aa69-69820f5e04c0-istiod-ca-cert\") pod \"e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8\" (UID: \"baec5da4-406d-4923-aa69-69820f5e04c0\") " pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"
Nov 01 03:36:32.227674 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:32.214107    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baec5da4-406d-4923-aa69-69820f5e04c0-workload-socket\") pod \"e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8\" (UID: \"baec5da4-406d-4923-aa69-69820f5e04c0\") " pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"
Nov 01 03:36:32.227674 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:32.214391    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baec5da4-406d-4923-aa69-69820f5e04c0-workload-certs\") pod \"e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8\" (UID: \"baec5da4-406d-4923-aa69-69820f5e04c0\") " pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"
Nov 01 03:36:32.227674 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:32.214832    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:36:32.247383 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:32.247312    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baec5da4-406d-4923-aa69-69820f5e04c0-istio-envoy\") pod \"e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8\" (UID: \"baec5da4-406d-4923-aa69-69820f5e04c0\") " pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"
Nov 01 03:36:32.249500 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:32.249470    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baec5da4-406d-4923-aa69-69820f5e04c0-istio-podinfo\") pod \"e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8\" (UID: \"baec5da4-406d-4923-aa69-69820f5e04c0\") " pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"
Nov 01 03:36:32.291581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:32.290650    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"]
Nov 01 03:36:32.327149 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:32.327050    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baec5da4-406d-4923-aa69-69820f5e04c0-istio-token\") pod \"e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8\" (UID: \"baec5da4-406d-4923-aa69-69820f5e04c0\") " pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"
Nov 01 03:36:32.334794 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:32.331158    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kqp8\" (UniqueName: \"kubernetes.io/projected/baec5da4-406d-4923-aa69-69820f5e04c0-kube-api-access-9kqp8\") pod \"e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8\" (UID: \"baec5da4-406d-4923-aa69-69820f5e04c0\") " pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"
Nov 01 03:36:32.404430 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:32.401016    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"
Nov 01 03:36:32.419038 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:32.419006    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:36:32.720543 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:32.717434    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:36:32.969902 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:36:32.969872    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaec5da4_406d_4923_aa69_69820f5e04c0.slice/crio-25e0d6224b7bc8c13ef531f577a8a77ccb853e0357982416d6d0ce32c85f1f53 WatchSource:0}: Error finding container 25e0d6224b7bc8c13ef531f577a8a77ccb853e0357982416d6d0ce32c85f1f53: Status 404 returned error can't find the container with id 25e0d6224b7bc8c13ef531f577a8a77ccb853e0357982416d6d0ce32c85f1f53
Nov 01 03:36:32.970710 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:32.970678    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"]
Nov 01 03:36:32.972725 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:32.972695    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:36:32.972808 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:32.972758    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:36:32.972808 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:32.972795    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:36:33.399578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:33.394513    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8" event={"ID":"baec5da4-406d-4923-aa69-69820f5e04c0","Type":"ContainerStarted","Data":"231cb1e1808fd5e653cc3939494ddc5f5c5e544420b1a5f88cc0daab42aa7d2e"}
Nov 01 03:36:33.399578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:33.394621    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8" event={"ID":"baec5da4-406d-4923-aa69-69820f5e04c0","Type":"ContainerStarted","Data":"25e0d6224b7bc8c13ef531f577a8a77ccb853e0357982416d6d0ce32c85f1f53"}
Nov 01 03:36:33.402587 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:33.401625    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"
Nov 01 03:36:34.206262 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:34.206170    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:36:34.290580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:34.289915    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8" podStartSLOduration=3.289891372 podStartE2EDuration="3.289891372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:36:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:36:33.568058276 +0000 UTC m=+1798.091064882" watchObservedRunningTime="2026-11-01 03:36:34.289891372 +0000 UTC m=+1798.812897961"
Nov 01 03:36:34.305278 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:34.304233    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"]
Nov 01 03:36:34.308584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:34.308521    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"
Nov 01 03:36:34.332588 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:34.330288    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-istio-data\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8\" (UID: \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"
Nov 01 03:36:34.332588 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:34.330387    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-workload-certs\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8\" (UID: \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"
Nov 01 03:36:34.332588 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:34.330432    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-istio-podinfo\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8\" (UID: \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"
Nov 01 03:36:34.332588 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:34.330512    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-istio-token\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8\" (UID: \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"
Nov 01 03:36:34.332588 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:34.330539    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-credential-socket\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8\" (UID: \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"
Nov 01 03:36:34.332588 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:34.330627    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-workload-socket\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8\" (UID: \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"
Nov 01 03:36:34.332588 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:34.330677    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-istio-envoy\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8\" (UID: \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"
Nov 01 03:36:34.332588 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:34.330786    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cc4j\" (UniqueName: \"kubernetes.io/projected/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-kube-api-access-8cc4j\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8\" (UID: \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"
Nov 01 03:36:34.332588 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:34.330938    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-istiod-ca-cert\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8\" (UID: \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"
Nov 01 03:36:34.438621 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:34.433411    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-istio-token\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8\" (UID: \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"
Nov 01 03:36:34.438621 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:34.437904    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-credential-socket\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8\" (UID: \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"
Nov 01 03:36:34.438621 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:34.437990    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-workload-socket\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8\" (UID: \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"
Nov 01 03:36:34.438621 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:34.438073    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-istio-envoy\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8\" (UID: \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"
Nov 01 03:36:34.438621 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:34.438219    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cc4j\" (UniqueName: \"kubernetes.io/projected/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-kube-api-access-8cc4j\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8\" (UID: \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"
Nov 01 03:36:34.438621 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:34.438329    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-istiod-ca-cert\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8\" (UID: \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"
Nov 01 03:36:34.438621 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:34.438429    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-istio-data\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8\" (UID: \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"
Nov 01 03:36:34.438621 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:34.438515    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-workload-certs\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8\" (UID: \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"
Nov 01 03:36:34.438621 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:34.438545    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-istio-podinfo\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8\" (UID: \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"
Nov 01 03:36:34.443911 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:34.443281    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-istiod-ca-cert\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8\" (UID: \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"
Nov 01 03:36:34.444515 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:34.444493    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-438641-openshift-default-dockercfg-4hlvf\""
Nov 01 03:36:34.445017 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:34.444998    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-credential-socket\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8\" (UID: \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"
Nov 01 03:36:34.445154 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:34.434148    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:36:34.445759 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:34.445740    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-istio-data\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8\" (UID: \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"
Nov 01 03:36:34.446176 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:34.446152    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-workload-certs\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8\" (UID: \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"
Nov 01 03:36:34.446309 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:34.436823    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"]
Nov 01 03:36:34.447138 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:34.447111    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-workload-socket\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8\" (UID: \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"
Nov 01 03:36:34.450088 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:34.450063    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"
Nov 01 03:36:34.451579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:34.451418    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-istio-envoy\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8\" (UID: \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"
Nov 01 03:36:34.453134 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:34.453076    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-istio-podinfo\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8\" (UID: \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"
Nov 01 03:36:34.622587 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:34.617467    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-istio-token\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8\" (UID: \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"
Nov 01 03:36:34.622587 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:34.620161    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cc4j\" (UniqueName: \"kubernetes.io/projected/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-kube-api-access-8cc4j\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8\" (UID: \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"
Nov 01 03:36:34.645583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:34.644987    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"
Nov 01 03:36:34.939636 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:34.934042    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:36:35.850868 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:35.849703    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw" podUID="b6d688b8-b911-4a6a-a842-2b0fb7a75e83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Nov 01 03:36:35.952901 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:35.952861    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2" podUID="e2483a06-2f4b-4a1e-af4e-a63eb50d29bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": context deadline exceeded"
Nov 01 03:36:35.994578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:35.972699    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8" podUID="88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Nov 01 03:36:36.111852 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:36.105296    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t" podUID="13930c1e-2200-4412-b8d3-ae3e4126a851" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": context deadline exceeded"
Nov 01 03:36:36.111852 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:36.105439    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps" podUID="d658772c-b072-4513-a639-d19a94c78525" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Nov 01 03:36:36.134599 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:36.115427    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf" podUID="40881383-61bb-4fc8-9c75-f8ffce4834c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Nov 01 03:36:36.260692 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:36.240764    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" podUID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:36:36.327593 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:36.323838    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm" podUID="75d142b3-7172-4b57-bc07-a2f404c4656e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/health\": context deadline exceeded"
Nov 01 03:36:36.327593 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:36.323784    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7" podUID="1ce834cc-d34d-46cc-a081-4d2cc261becb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:36:36.387807 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:36.387697    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t" podUID="13930c1e-2200-4412-b8d3-ae3e4126a851" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 01 03:36:36.415591 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:36.401966    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"
Nov 01 03:36:36.530758 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:36.530724    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"]
Nov 01 03:36:36.531079 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:36.531059    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"
Nov 01 03:36:36.559660 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:36.527093    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:36:36.585727 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:36.585684    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:36:36.586004 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:36.585976    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:36:36.586637 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:36.403809    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm" podUID="b2637bc6-3828-4b75-9783-5415519403a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": context deadline exceeded"
Nov 01 03:36:36.609582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:36.445710    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm" podUID="75d142b3-7172-4b57-bc07-a2f404c4656e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:36:36.699581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:36.698985    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm" podUID="b2637bc6-3828-4b75-9783-5415519403a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 01 03:36:36.852682 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:36.852635    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw" podUID="b6d688b8-b911-4a6a-a842-2b0fb7a75e83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:36:36.853825 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:36.853797    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:36:36.873809 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:36.873777    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:36:37.000018 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:36.999924    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2" podUID="e2483a06-2f4b-4a1e-af4e-a63eb50d29bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": context deadline exceeded"
Nov 01 03:36:37.000799 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:37.000772    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"
Nov 01 03:36:37.002333 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:37.002305    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8" podUID="88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Nov 01 03:36:37.141660 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:37.141610    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:36:37.145097 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:37.145042    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps" podUID="d658772c-b072-4513-a639-d19a94c78525" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:36:37.158093 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:37.158063    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:36:37.260626 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:37.256730    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:36:37.350586 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:37.334695    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7" podUID="1ce834cc-d34d-46cc-a081-4d2cc261becb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:36:37.350586 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:37.336263    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"
Nov 01 03:36:37.350586 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:37.346301    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" podUID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:36:37.350586 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:37.346870    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:36:37.456518 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:37.454827    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t" podUID="13930c1e-2200-4412-b8d3-ae3e4126a851" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 01 03:36:37.480582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:37.457200    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:36:37.537738 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:37.530634    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj" podUID="b0e32cb3-ac18-4c1f-819c-f1139a88bfea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:36:37.537738 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:37.532638    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf" podUID="40881383-61bb-4fc8-9c75-f8ffce4834c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Nov 01 03:36:37.537738 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:37.533146    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:36:37.580611 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:37.555368    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8" event={"ID":"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d","Type":"ContainerStarted","Data":"92e72d774936e9ce7b022fa14cdab8af4b94cd83a2f2c562795b051a50dee4f7"}
Nov 01 03:36:37.581346 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:37.555674    4930 scope.go:122] "RemoveContainer" containerID="5430b25a0e6a45da064e414f6a4a164a847485e26ff22c074a83dac8ca4b2a76"
Nov 01 03:36:37.591059 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:37.591028    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:36:37.753582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:37.749800    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm" podUID="75d142b3-7172-4b57-bc07-a2f404c4656e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": context deadline exceeded"
Nov 01 03:36:37.878054 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:37.846257    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp" podUID="667240b0-64a7-45cd-84fe-9cd7e0d8a301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:36:37.878054 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:37.857680    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw" podUID="b6d688b8-b911-4a6a-a842-2b0fb7a75e83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:36:37.933310 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:37.933079    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:36:37.941103 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:37.941075    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:36:37.941344 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:37.941326    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:36:38.019581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:38.000771    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2" podUID="e2483a06-2f4b-4a1e-af4e-a63eb50d29bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:36:38.019581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:38.001276    4930 scope.go:122] "RemoveContainer" containerID="62cea5625a1a4358640038d0340fa72508f1d86edf0982aac8e821fc35c8b9f2"
Nov 01 03:36:38.034939 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:38.034694    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8" podUID="88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:36:38.104333 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:38.104300    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:36:38.107292 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:38.107268    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jh955_f29ea61e-2a78-4c11-9566-f52497e135ff/console-operator/2.log"
Nov 01 03:36:38.140703 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:38.140615    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8" podUID="88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:36:38.154823 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:38.154796    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jh955_f29ea61e-2a78-4c11-9566-f52497e135ff/console-operator/2.log"
Nov 01 03:36:38.424888 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:38.383259    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" podUID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded"
Nov 01 03:36:38.542578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:38.542242    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:36:38.565574 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:38.555894    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t" podUID="13930c1e-2200-4412-b8d3-ae3e4126a851" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 01 03:36:38.565574 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:38.556530    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf" podUID="40881383-61bb-4fc8-9c75-f8ffce4834c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Nov 01 03:36:38.589614 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:38.589550    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:36:38.592029 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:38.591983    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7" podUID="1ce834cc-d34d-46cc-a081-4d2cc261becb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:36:38.602391 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:38.602341    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj" podUID="b0e32cb3-ac18-4c1f-819c-f1139a88bfea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:36:38.603096 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:38.603055    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:36:38.783633 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:38.783529    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8" event={"ID":"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d","Type":"ContainerStarted","Data":"190679fadf2457c52f69239758b5f78a2f5b923dfef156f24191b6a81729fe1b"}
Nov 01 03:36:38.934584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:38.854691    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm" podUID="75d142b3-7172-4b57-bc07-a2f404c4656e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.94:8000/ready\": context deadline exceeded"
Nov 01 03:36:38.934584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:38.910204    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw" podUID="b6d688b8-b911-4a6a-a842-2b0fb7a75e83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:36:38.974724 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:38.974250    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7" podUID="1ce834cc-d34d-46cc-a081-4d2cc261becb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": context deadline exceeded"
Nov 01 03:36:39.007171 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:39.006683    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2" podUID="e2483a06-2f4b-4a1e-af4e-a63eb50d29bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": context deadline exceeded"
Nov 01 03:36:39.016198 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:39.015808    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n" podUID="d2511772-66f9-40a0-9166-87c4345485c4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:36:39.145255 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:39.145210    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8" podUID="88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:36:39.354738 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:39.354700    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:36:39.420659 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:39.384894    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" podUID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:36:39.421616 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:39.421531    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8" podStartSLOduration=5.421508696 podStartE2EDuration="5.421508696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:36:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:36:38.994827671 +0000 UTC m=+1803.517834259" watchObservedRunningTime="2026-11-01 03:36:39.421508696 +0000 UTC m=+1803.944515285"
Nov 01 03:36:39.488584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:39.458798    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:36:39.584582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:39.559599    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:36:39.735677 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:39.707972    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"
Nov 01 03:36:39.735677 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:39.629010    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7" podUID="1ce834cc-d34d-46cc-a081-4d2cc261becb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:36:39.882648 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:39.864307    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:36:39.920124 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:39.920067    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw" podUID="b6d688b8-b911-4a6a-a842-2b0fb7a75e83" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:36:39.949984 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:39.949945    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp" podUID="667240b0-64a7-45cd-84fe-9cd7e0d8a301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:36:40.027751 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:40.015147    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:36:40.043476 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:40.043450    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"
Nov 01 03:36:40.043802 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:40.043788    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"
Nov 01 03:36:40.043976 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:40.043955    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"
Nov 01 03:36:40.044117 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:40.044107    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"
Nov 01 03:36:40.044266 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:40.041801    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2" podUID="e2483a06-2f4b-4a1e-af4e-a63eb50d29bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:36:40.050916 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:40.050834    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:36:40.140880 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:40.138844    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:36:40.140880 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:40.139760    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"
Nov 01 03:36:40.231177 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:40.231129    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:36:40.682577 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:40.657907    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:36:41.247269 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:41.247227    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-7cd000-54689dbf75-74hbs"]
Nov 01 03:36:41.315476 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:41.315439    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-7cd000-54689dbf75-74hbs"]
Nov 01 03:36:41.315822 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:41.315708    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-7cd000-54689dbf75-74hbs"
Nov 01 03:36:41.392625 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:41.382896    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-7cd000\""
Nov 01 03:36:41.392625 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:41.383593    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-7cd000-dockercfg-49zgx\""
Nov 01 03:36:41.450707 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:41.450606    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15dca7b4-1639-406e-a15e-67f7e85ba636-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-7cd000-54689dbf75-74hbs\" (UID: \"15dca7b4-1639-406e-a15e-67f7e85ba636\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-7cd000-54689dbf75-74hbs"
Nov 01 03:36:41.450866 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:41.450798    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvnkb\" (UniqueName: \"kubernetes.io/projected/15dca7b4-1639-406e-a15e-67f7e85ba636-kube-api-access-fvnkb\") pod \"payload-pre-processing-e2e-shared-a-w6-7cd000-54689dbf75-74hbs\" (UID: \"15dca7b4-1639-406e-a15e-67f7e85ba636\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-7cd000-54689dbf75-74hbs"
Nov 01 03:36:41.552633 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:41.552492    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvnkb\" (UniqueName: \"kubernetes.io/projected/15dca7b4-1639-406e-a15e-67f7e85ba636-kube-api-access-fvnkb\") pod \"payload-pre-processing-e2e-shared-a-w6-7cd000-54689dbf75-74hbs\" (UID: \"15dca7b4-1639-406e-a15e-67f7e85ba636\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-7cd000-54689dbf75-74hbs"
Nov 01 03:36:41.552808 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:41.552712    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15dca7b4-1639-406e-a15e-67f7e85ba636-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-7cd000-54689dbf75-74hbs\" (UID: \"15dca7b4-1639-406e-a15e-67f7e85ba636\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-7cd000-54689dbf75-74hbs"
Nov 01 03:36:41.553494 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:41.553465    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15dca7b4-1639-406e-a15e-67f7e85ba636-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-7cd000-54689dbf75-74hbs\" (UID: \"15dca7b4-1639-406e-a15e-67f7e85ba636\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-7cd000-54689dbf75-74hbs"
Nov 01 03:36:41.618656 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:41.618623    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvnkb\" (UniqueName: \"kubernetes.io/projected/15dca7b4-1639-406e-a15e-67f7e85ba636-kube-api-access-fvnkb\") pod \"payload-pre-processing-e2e-shared-a-w6-7cd000-54689dbf75-74hbs\" (UID: \"15dca7b4-1639-406e-a15e-67f7e85ba636\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-7cd000-54689dbf75-74hbs"
Nov 01 03:36:41.642858 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:41.642828    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-7cd000-54689dbf75-74hbs"
Nov 01 03:36:41.954264 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:41.954222    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:36:42.054497 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:42.054462    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-7cd000-54689dbf75-74hbs"]
Nov 01 03:36:42.055093 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:36:42.055067    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15dca7b4_1639_406e_a15e_67f7e85ba636.slice/crio-1cea0fee007568acaf3ba281858a4c9e66bd1dd004b31246fc9d5fd534c54694 WatchSource:0}: Error finding container 1cea0fee007568acaf3ba281858a4c9e66bd1dd004b31246fc9d5fd534c54694: Status 404 returned error can't find the container with id 1cea0fee007568acaf3ba281858a4c9e66bd1dd004b31246fc9d5fd534c54694
Nov 01 03:36:42.191856 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:42.191823    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:36:42.229717 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:42.229623    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-7cd000-54689dbf75-74hbs" event={"ID":"15dca7b4-1639-406e-a15e-67f7e85ba636","Type":"ContainerStarted","Data":"2cd937b8816e6ca3524cc7493d425a1854a70b2134dcb0b99ad992df9e243cd6"}
Nov 01 03:36:42.229717 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:42.229700    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-7cd000-54689dbf75-74hbs" event={"ID":"15dca7b4-1639-406e-a15e-67f7e85ba636","Type":"ContainerStarted","Data":"1cea0fee007568acaf3ba281858a4c9e66bd1dd004b31246fc9d5fd534c54694"}
Nov 01 03:36:42.229979 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:42.229955    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-7cd000-54689dbf75-74hbs"
Nov 01 03:36:42.393782 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:42.393718    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:36:42.921638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:42.921593    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:36:43.340113 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:43.338320    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:36:43.600391 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:43.600302    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:36:43.781145 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:43.781050    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:36:45.461822 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:45.461699    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:36:45.680884 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:45.668437    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:36:46.058690 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:46.053214    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-7cd000-54689dbf75-74hbs" podStartSLOduration=6.053191676 podStartE2EDuration="6.053191676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:36:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:36:42.38490472 +0000 UTC m=+1806.907911316" watchObservedRunningTime="2026-11-01 03:36:46.053191676 +0000 UTC m=+1810.576198268"
Nov 01 03:36:46.074086 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:46.063405    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:36:46.074086 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:46.070823    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-438641-57f787c447-mk9mn"]
Nov 01 03:36:46.143579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:46.112376    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-438641-57f787c447-mk9mn"
Nov 01 03:36:46.143579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:46.128467    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-438641-57f787c447-mk9mn"]
Nov 01 03:36:46.167978 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:46.167940    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-438641-dockercfg-zp68b\""
Nov 01 03:36:46.172520 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:46.169599    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-438641\""
Nov 01 03:36:46.200760 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:46.200696    4930 patch_prober.go:29] interesting pod/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 03:36:46.215613 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:46.201037    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8" podUID="b941e0fe-6b17-4e30-917c-4d3f4382c545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:36:46.226339 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:46.206101    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrff4\" (UniqueName: \"kubernetes.io/projected/463ab8cc-0751-4fcf-a387-4b85b9f09180-kube-api-access-qrff4\") pod \"payload-processing-e2e-shared-b-w0-438641-57f787c447-mk9mn\" (UID: \"463ab8cc-0751-4fcf-a387-4b85b9f09180\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-438641-57f787c447-mk9mn"
Nov 01 03:36:46.226339 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:46.216153    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/463ab8cc-0751-4fcf-a387-4b85b9f09180-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-438641-57f787c447-mk9mn\" (UID: \"463ab8cc-0751-4fcf-a387-4b85b9f09180\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-438641-57f787c447-mk9mn"
Nov 01 03:36:46.243621 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:46.243587    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-438641-668fd65b59-wmqpw"]
Nov 01 03:36:46.277908 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:46.277878    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-438641-668fd65b59-wmqpw"]
Nov 01 03:36:46.278320 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:46.278299    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-438641-668fd65b59-wmqpw"
Nov 01 03:36:46.323693 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:46.323492    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrff4\" (UniqueName: \"kubernetes.io/projected/463ab8cc-0751-4fcf-a387-4b85b9f09180-kube-api-access-qrff4\") pod \"payload-processing-e2e-shared-b-w0-438641-57f787c447-mk9mn\" (UID: \"463ab8cc-0751-4fcf-a387-4b85b9f09180\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-438641-57f787c447-mk9mn"
Nov 01 03:36:46.324137 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:46.324104    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/463ab8cc-0751-4fcf-a387-4b85b9f09180-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-438641-57f787c447-mk9mn\" (UID: \"463ab8cc-0751-4fcf-a387-4b85b9f09180\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-438641-57f787c447-mk9mn"
Nov 01 03:36:46.325447 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:46.325419    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/463ab8cc-0751-4fcf-a387-4b85b9f09180-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-438641-57f787c447-mk9mn\" (UID: \"463ab8cc-0751-4fcf-a387-4b85b9f09180\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-438641-57f787c447-mk9mn"
Nov 01 03:36:46.392228 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:46.392188    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrff4\" (UniqueName: \"kubernetes.io/projected/463ab8cc-0751-4fcf-a387-4b85b9f09180-kube-api-access-qrff4\") pod \"payload-processing-e2e-shared-b-w0-438641-57f787c447-mk9mn\" (UID: \"463ab8cc-0751-4fcf-a387-4b85b9f09180\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-438641-57f787c447-mk9mn"
Nov 01 03:36:46.427999 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:46.427964    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5974826d-4ad8-4b1e-b2ad-322b8d01359e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-438641-668fd65b59-wmqpw\" (UID: \"5974826d-4ad8-4b1e-b2ad-322b8d01359e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-438641-668fd65b59-wmqpw"
Nov 01 03:36:46.428167 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:46.428029    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h92m2\" (UniqueName: \"kubernetes.io/projected/5974826d-4ad8-4b1e-b2ad-322b8d01359e-kube-api-access-h92m2\") pod \"payload-pre-processing-e2e-shared-b-w0-438641-668fd65b59-wmqpw\" (UID: \"5974826d-4ad8-4b1e-b2ad-322b8d01359e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-438641-668fd65b59-wmqpw"
Nov 01 03:36:46.456072 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:46.456035    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-438641-57f787c447-mk9mn"
Nov 01 03:36:46.530811 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:46.530079    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5974826d-4ad8-4b1e-b2ad-322b8d01359e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-438641-668fd65b59-wmqpw\" (UID: \"5974826d-4ad8-4b1e-b2ad-322b8d01359e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-438641-668fd65b59-wmqpw"
Nov 01 03:36:46.530811 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:46.530124    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h92m2\" (UniqueName: \"kubernetes.io/projected/5974826d-4ad8-4b1e-b2ad-322b8d01359e-kube-api-access-h92m2\") pod \"payload-pre-processing-e2e-shared-b-w0-438641-668fd65b59-wmqpw\" (UID: \"5974826d-4ad8-4b1e-b2ad-322b8d01359e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-438641-668fd65b59-wmqpw"
Nov 01 03:36:46.534295 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:46.531016    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5974826d-4ad8-4b1e-b2ad-322b8d01359e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-438641-668fd65b59-wmqpw\" (UID: \"5974826d-4ad8-4b1e-b2ad-322b8d01359e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-438641-668fd65b59-wmqpw"
Nov 01 03:36:46.595401 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:46.595371    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h92m2\" (UniqueName: \"kubernetes.io/projected/5974826d-4ad8-4b1e-b2ad-322b8d01359e-kube-api-access-h92m2\") pod \"payload-pre-processing-e2e-shared-b-w0-438641-668fd65b59-wmqpw\" (UID: \"5974826d-4ad8-4b1e-b2ad-322b8d01359e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-438641-668fd65b59-wmqpw"
Nov 01 03:36:46.650154 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:46.650122    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-438641-668fd65b59-wmqpw"
Nov 01 03:36:46.878031 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:46.877997    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-438641-57f787c447-mk9mn"]
Nov 01 03:36:47.064535 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:36:47.064505    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5974826d_4ad8_4b1e_b2ad_322b8d01359e.slice/crio-d859cd74100fed0af0e6ebd21107a4e76820f406eec26735c36680946f608e9d WatchSource:0}: Error finding container d859cd74100fed0af0e6ebd21107a4e76820f406eec26735c36680946f608e9d: Status 404 returned error can't find the container with id d859cd74100fed0af0e6ebd21107a4e76820f406eec26735c36680946f608e9d
Nov 01 03:36:47.067053 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:47.067021    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-438641-668fd65b59-wmqpw"]
Nov 01 03:36:47.287101 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:47.287057    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-438641-57f787c447-mk9mn" event={"ID":"463ab8cc-0751-4fcf-a387-4b85b9f09180","Type":"ContainerStarted","Data":"08b9d41ffd65261958fce7198ed8a709790d38e4e5abc0e54252c5e96bfa531d"}
Nov 01 03:36:47.287293 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:47.287131    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-438641-57f787c447-mk9mn"
Nov 01 03:36:47.287293 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:47.287154    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-438641-57f787c447-mk9mn" event={"ID":"463ab8cc-0751-4fcf-a387-4b85b9f09180","Type":"ContainerStarted","Data":"78deddc28612dc709b47d5367deb7cdb007be44fa829697472339f86502a2bac"}
Nov 01 03:36:47.289089 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:47.289059    4930 generic.go:356] "Generic (PLEG): container finished" podID="ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6" containerID="34dc61e7d67b7c666d5691fecdf5db6b2f65960eaa7d26ce8c08582ae6e53bb6" exitCode=0
Nov 01 03:36:47.289210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:47.289133    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz" event={"ID":"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6","Type":"ContainerDied","Data":"34dc61e7d67b7c666d5691fecdf5db6b2f65960eaa7d26ce8c08582ae6e53bb6"}
Nov 01 03:36:47.289638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:47.289615    4930 scope.go:122] "RemoveContainer" containerID="34dc61e7d67b7c666d5691fecdf5db6b2f65960eaa7d26ce8c08582ae6e53bb6"
Nov 01 03:36:47.291207 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:47.291184    4930 generic.go:356] "Generic (PLEG): container finished" podID="b941e0fe-6b17-4e30-917c-4d3f4382c545" containerID="a1681e86692725e01cda7745ec555ae99eb41f125053fd0d010bb18028603f3f" exitCode=0
Nov 01 03:36:47.291318 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:47.291269    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8" event={"ID":"b941e0fe-6b17-4e30-917c-4d3f4382c545","Type":"ContainerDied","Data":"a1681e86692725e01cda7745ec555ae99eb41f125053fd0d010bb18028603f3f"}
Nov 01 03:36:47.291621 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:47.291584    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:36:47.291703 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:47.291667    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:36:47.291765 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:47.291709    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:36:47.291974 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:47.291955    4930 scope.go:122] "RemoveContainer" containerID="a1681e86692725e01cda7745ec555ae99eb41f125053fd0d010bb18028603f3f"
Nov 01 03:36:47.293828 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:47.293798    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:36:47.293897 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:47.293878    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:36:47.293951 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:47.293922    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:36:47.294173 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:47.294135    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-438641-668fd65b59-wmqpw" event={"ID":"5974826d-4ad8-4b1e-b2ad-322b8d01359e","Type":"ContainerStarted","Data":"41b6a0e5ca29458b9fcfd80c9de4d75951ea4990e6a5911e98ac6756753af89f"}
Nov 01 03:36:47.294279 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:47.294187    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-438641-668fd65b59-wmqpw" event={"ID":"5974826d-4ad8-4b1e-b2ad-322b8d01359e","Type":"ContainerStarted","Data":"d859cd74100fed0af0e6ebd21107a4e76820f406eec26735c36680946f608e9d"}
Nov 01 03:36:47.294279 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:47.294231    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-438641-668fd65b59-wmqpw"
Nov 01 03:36:47.535192 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:47.534156    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-438641-57f787c447-mk9mn" podStartSLOduration=1.534136045 podStartE2EDuration="1.534136045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:36:47.410628612 +0000 UTC m=+1811.933635203" watchObservedRunningTime="2026-11-01 03:36:47.534136045 +0000 UTC m=+1812.057142633"
Nov 01 03:36:48.127889 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:48.127851    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:36:48.403643 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:48.389955    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:36:48.418588 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:48.417046    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz" event={"ID":"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6","Type":"ContainerStarted","Data":"505ef1473dac97459c5dddc71051cbf6291897e8c261801c22a904306630ad34"}
Nov 01 03:36:48.430671 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:48.430625    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8" event={"ID":"b941e0fe-6b17-4e30-917c-4d3f4382c545","Type":"ContainerStarted","Data":"531d96c3fec86de40cc71a6cb46e25e1287e0820759da0190f74ec879ee8f3af"}
Nov 01 03:36:48.616579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:48.584053    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-438641-668fd65b59-wmqpw" podStartSLOduration=2.5726673780000002 podStartE2EDuration="2.572667378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:36:47.800620107 +0000 UTC m=+1812.323626696" watchObservedRunningTime="2026-11-01 03:36:48.572667378 +0000 UTC m=+1813.095673967"
Nov 01 03:36:49.058149 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:49.058112    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"
Nov 01 03:36:49.058743 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:49.058713    4930 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 01 03:36:49.058967 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:49.058933    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz" podUID="ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 01 03:36:49.222580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:49.221945    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:36:49.235612 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:49.235547    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"
Nov 01 03:36:49.236396 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:49.236360    4930 patch_prober.go:29] interesting pod/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Nov 01 03:36:49.236698 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:49.236667    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8" podUID="b941e0fe-6b17-4e30-917c-4d3f4382c545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Nov 01 03:36:49.275956 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:36:49.275875    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:36:50.057538 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:50.057507    4930 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 01 03:36:50.057948 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:50.057578    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz" podUID="ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 01 03:36:50.238234 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:50.238185    4930 patch_prober.go:29] interesting pod/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Nov 01 03:36:50.238234 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:50.238246    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8" podUID="b941e0fe-6b17-4e30-917c-4d3f4382c545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Nov 01 03:36:51.057846 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:51.057780    4930 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 01 03:36:51.057846 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:51.057829    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz" podUID="ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 01 03:36:51.233642 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:51.233599    4930 patch_prober.go:29] interesting pod/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Nov 01 03:36:51.233863 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:51.233684    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8" podUID="b941e0fe-6b17-4e30-917c-4d3f4382c545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Nov 01 03:36:52.057242 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:52.057206    4930 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 01 03:36:52.057421 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:52.057261    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz" podUID="ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 01 03:36:52.234454 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:52.234418    4930 patch_prober.go:29] interesting pod/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Nov 01 03:36:52.234891 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:52.234466    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8" podUID="b941e0fe-6b17-4e30-917c-4d3f4382c545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Nov 01 03:36:52.880582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:52.878646    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"
Nov 01 03:36:53.057604 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:53.057572    4930 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 01 03:36:53.057820 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:53.057621    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz" podUID="ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 01 03:36:53.233698 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:53.233599    4930 patch_prober.go:29] interesting pod/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Nov 01 03:36:53.233963 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:53.233925    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8" podUID="b941e0fe-6b17-4e30-917c-4d3f4382c545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Nov 01 03:36:53.246417 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:53.246392    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-7cd000-54689dbf75-74hbs"
Nov 01 03:36:53.985581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:53.981050    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:36:54.058046 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:54.058008    4930 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 01 03:36:54.058260 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:54.058121    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz" podUID="ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 01 03:36:54.364512 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:54.364478    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"
Nov 01 03:36:54.365227 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:54.365199    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"
Nov 01 03:36:54.366227 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:54.366207    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"
Nov 01 03:36:55.062278 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:55.062246    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"
Nov 01 03:36:55.063522 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:55.063503    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"
Nov 01 03:36:58.444697 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:58.444667    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-438641-57f787c447-mk9mn"
Nov 01 03:36:58.445183 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:58.444738    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-438641-668fd65b59-wmqpw"
Nov 01 03:36:58.823026 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:58.822939    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:36:59.030257 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:36:59.030221    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:37:04.042273 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:04.042224    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:37:04.477195 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:04.477140    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:37:04.698825 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:04.698767    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:37:05.787928 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:05.787887    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:37:06.688582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:06.685397    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"]
Nov 01 03:37:06.690172 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:06.690146    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"
Nov 01 03:37:06.753251 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:06.753220    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"]
Nov 01 03:37:06.791631 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:06.791600    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-7d7329-openshift-default-dockercfg-27mrl\""
Nov 01 03:37:06.838698 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:06.838503    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:37:06.876071 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:06.876039    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68863127-fc50-48b2-bf3e-c6c99fb4a29e-credential-socket\") pod \"e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66\" (UID: \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\") " pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"
Nov 01 03:37:06.876334 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:06.876310    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68863127-fc50-48b2-bf3e-c6c99fb4a29e-workload-certs\") pod \"e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66\" (UID: \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\") " pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"
Nov 01 03:37:06.876429 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:06.876398    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68863127-fc50-48b2-bf3e-c6c99fb4a29e-istio-envoy\") pod \"e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66\" (UID: \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\") " pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"
Nov 01 03:37:06.876490 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:06.876444    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68863127-fc50-48b2-bf3e-c6c99fb4a29e-istio-podinfo\") pod \"e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66\" (UID: \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\") " pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"
Nov 01 03:37:06.876546 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:06.876525    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrbd9\" (UniqueName: \"kubernetes.io/projected/68863127-fc50-48b2-bf3e-c6c99fb4a29e-kube-api-access-jrbd9\") pod \"e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66\" (UID: \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\") " pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"
Nov 01 03:37:06.876669 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:06.876653    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68863127-fc50-48b2-bf3e-c6c99fb4a29e-workload-socket\") pod \"e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66\" (UID: \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\") " pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"
Nov 01 03:37:06.876746 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:06.876700    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68863127-fc50-48b2-bf3e-c6c99fb4a29e-istiod-ca-cert\") pod \"e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66\" (UID: \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\") " pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"
Nov 01 03:37:06.876803 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:06.876772    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68863127-fc50-48b2-bf3e-c6c99fb4a29e-istio-data\") pod \"e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66\" (UID: \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\") " pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"
Nov 01 03:37:06.876877 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:06.876851    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68863127-fc50-48b2-bf3e-c6c99fb4a29e-istio-token\") pod \"e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66\" (UID: \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\") " pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"
Nov 01 03:37:06.978577 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:06.978437    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68863127-fc50-48b2-bf3e-c6c99fb4a29e-istio-data\") pod \"e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66\" (UID: \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\") " pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"
Nov 01 03:37:06.978577 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:06.978529    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68863127-fc50-48b2-bf3e-c6c99fb4a29e-istio-token\") pod \"e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66\" (UID: \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\") " pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"
Nov 01 03:37:06.978864 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:06.978626    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68863127-fc50-48b2-bf3e-c6c99fb4a29e-credential-socket\") pod \"e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66\" (UID: \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\") " pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"
Nov 01 03:37:06.978864 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:06.978669    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68863127-fc50-48b2-bf3e-c6c99fb4a29e-workload-certs\") pod \"e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66\" (UID: \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\") " pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"
Nov 01 03:37:06.978864 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:06.978730    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68863127-fc50-48b2-bf3e-c6c99fb4a29e-istio-envoy\") pod \"e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66\" (UID: \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\") " pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"
Nov 01 03:37:06.978864 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:06.978773    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68863127-fc50-48b2-bf3e-c6c99fb4a29e-istio-podinfo\") pod \"e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66\" (UID: \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\") " pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"
Nov 01 03:37:06.978864 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:06.978832    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrbd9\" (UniqueName: \"kubernetes.io/projected/68863127-fc50-48b2-bf3e-c6c99fb4a29e-kube-api-access-jrbd9\") pod \"e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66\" (UID: \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\") " pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"
Nov 01 03:37:06.979136 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:06.978908    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68863127-fc50-48b2-bf3e-c6c99fb4a29e-workload-socket\") pod \"e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66\" (UID: \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\") " pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"
Nov 01 03:37:06.979136 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:06.978941    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68863127-fc50-48b2-bf3e-c6c99fb4a29e-istiod-ca-cert\") pod \"e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66\" (UID: \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\") " pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"
Nov 01 03:37:06.979254 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:06.979128    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68863127-fc50-48b2-bf3e-c6c99fb4a29e-workload-certs\") pod \"e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66\" (UID: \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\") " pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"
Nov 01 03:37:06.979765 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:06.979737    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68863127-fc50-48b2-bf3e-c6c99fb4a29e-credential-socket\") pod \"e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66\" (UID: \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\") " pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"
Nov 01 03:37:06.979918 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:06.979771    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68863127-fc50-48b2-bf3e-c6c99fb4a29e-istiod-ca-cert\") pod \"e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66\" (UID: \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\") " pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"
Nov 01 03:37:06.980405 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:06.978904    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68863127-fc50-48b2-bf3e-c6c99fb4a29e-istio-data\") pod \"e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66\" (UID: \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\") " pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"
Nov 01 03:37:06.980405 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:06.980316    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68863127-fc50-48b2-bf3e-c6c99fb4a29e-workload-socket\") pod \"e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66\" (UID: \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\") " pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"
Nov 01 03:37:06.982990 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:06.982969    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68863127-fc50-48b2-bf3e-c6c99fb4a29e-istio-podinfo\") pod \"e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66\" (UID: \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\") " pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"
Nov 01 03:37:06.990899 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:06.990850    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68863127-fc50-48b2-bf3e-c6c99fb4a29e-istio-envoy\") pod \"e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66\" (UID: \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\") " pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"
Nov 01 03:37:07.048298 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:07.048265    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68863127-fc50-48b2-bf3e-c6c99fb4a29e-istio-token\") pod \"e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66\" (UID: \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\") " pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"
Nov 01 03:37:07.049086 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:07.049060    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrbd9\" (UniqueName: \"kubernetes.io/projected/68863127-fc50-48b2-bf3e-c6c99fb4a29e-kube-api-access-jrbd9\") pod \"e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66\" (UID: \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\") " pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"
Nov 01 03:37:07.183764 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:07.183732    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:37:07.311493 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:07.311413    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"
Nov 01 03:37:07.772067 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:07.772028    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"]
Nov 01 03:37:07.776519 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:07.776470    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:37:07.776648 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:07.776598    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:37:07.776718 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:07.776647    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:37:08.679950 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:08.679912    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66" event={"ID":"68863127-fc50-48b2-bf3e-c6c99fb4a29e","Type":"ContainerStarted","Data":"128cc4e340b3f0a09ae965072f66c97a64b1bca0f1657670db7c53f1454d9b75"}
Nov 01 03:37:08.680336 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:08.679975    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66" event={"ID":"68863127-fc50-48b2-bf3e-c6c99fb4a29e","Type":"ContainerStarted","Data":"9a7c3815409fbc9121f471b9f7bc46179c0c25dc92d8d6fbfc9bd0ed62e5815f"}
Nov 01 03:37:08.939595 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:08.939451    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66" podStartSLOduration=2.93943605 podStartE2EDuration="2.93943605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:37:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:37:08.88931377 +0000 UTC m=+1833.412320359" watchObservedRunningTime="2026-11-01 03:37:08.93943605 +0000 UTC m=+1833.462442693"
Nov 01 03:37:09.311764 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:09.311680    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"
Nov 01 03:37:09.317921 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:09.317891    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"
Nov 01 03:37:09.689240 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:09.689201    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"
Nov 01 03:37:09.690172 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:09.690155    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"
Nov 01 03:37:10.253533 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:10.253490    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:37:12.212300 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:12.212254    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:37:12.397702 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:12.397663    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:37:12.547810 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:12.547722    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:37:15.498676 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:15.498636    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:37:15.647769 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:15.647726    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:37:15.796594 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:15.796438    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:37:19.122895 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:37:19.122813    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:37:21.155428 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:21.155385    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:37:26.326236 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:26.326198    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:37:31.165053 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:31.165018    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:37:36.077756 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:36.077714    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:37:37.133334 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:37.133296    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:37:38.956723 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:38.956690    4930 scope.go:122] "RemoveContainer" containerID="b8a5a47d04cdc578425573fe9edeeee464ce95526463821910387637d62c24bb"
Nov 01 03:37:38.967730 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:38.967705    4930 scope.go:122] "RemoveContainer" containerID="9f53ee83f6113d88e02c7258ff41a9d3c7a0b71758de179a6774ce56d6d050cd"
Nov 01 03:37:38.977187 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:38.977164    4930 scope.go:122] "RemoveContainer" containerID="80efdc85614993a8b5aec7e8f614132212deb4db3b3454a5366283fb630b9e45"
Nov 01 03:37:40.958992 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:40.958958    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:37:42.810369 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:42.810333    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:37:42.972012 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:42.971978    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:37:45.942576 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:45.942522    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:37:46.425151 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:46.425117    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:37:49.122742 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:37:49.122706    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:37:50.764772 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:50.764718    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:37:50.925002 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:50.924924    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:37:51.134006 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:51.133968    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:37:53.758583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:53.742090    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:37:54.015131 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:54.015036    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:37:55.868712 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:37:55.868667    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:38:00.725794 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:00.725751    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:38:00.892712 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:00.892675    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:38:01.069633 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:01.069511    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:38:05.623135 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:05.623086    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:38:05.785367 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:05.785324    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:38:05.971736 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:05.971638    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:38:10.732097 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:10.732043    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:38:15.395313 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:15.395265    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:38:15.620149 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:15.620114    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:38:19.123697 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:38:19.123655    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:38:20.491350 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:20.491315    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:38:21.843943 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:21.843906    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:38:25.261360 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:25.261314    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:38:30.243068 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:30.243030    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:38:34.978189 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:34.978154    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:38:35.317982 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:35.317896    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:38:35.549607 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:35.549497    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:38:35.712437 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:35.712340    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:38:36.340231 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:36.339874    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"]
Nov 01 03:38:36.605667 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:36.564510    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:38:36.727588 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:36.727029    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"
Nov 01 03:38:36.807310 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:36.771625    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbc558d1-0eff-46fb-a780-7ebcc1939928-istio-token\") pod \"e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp\" (UID: \"cbc558d1-0eff-46fb-a780-7ebcc1939928\") " pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"
Nov 01 03:38:36.843733 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:36.843694    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhsz5\" (UniqueName: \"kubernetes.io/projected/cbc558d1-0eff-46fb-a780-7ebcc1939928-kube-api-access-nhsz5\") pod \"e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp\" (UID: \"cbc558d1-0eff-46fb-a780-7ebcc1939928\") " pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"
Nov 01 03:38:36.857759 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:36.857688    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc558d1-0eff-46fb-a780-7ebcc1939928-workload-certs\") pod \"e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp\" (UID: \"cbc558d1-0eff-46fb-a780-7ebcc1939928\") " pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"
Nov 01 03:38:36.882695 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:36.882655    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc558d1-0eff-46fb-a780-7ebcc1939928-istio-podinfo\") pod \"e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp\" (UID: \"cbc558d1-0eff-46fb-a780-7ebcc1939928\") " pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"
Nov 01 03:38:36.904585 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:36.883115    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc558d1-0eff-46fb-a780-7ebcc1939928-istio-data\") pod \"e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp\" (UID: \"cbc558d1-0eff-46fb-a780-7ebcc1939928\") " pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"
Nov 01 03:38:36.904879 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:36.904850    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc558d1-0eff-46fb-a780-7ebcc1939928-istiod-ca-cert\") pod \"e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp\" (UID: \"cbc558d1-0eff-46fb-a780-7ebcc1939928\") " pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"
Nov 01 03:38:36.905051 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:36.905034    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc558d1-0eff-46fb-a780-7ebcc1939928-istio-envoy\") pod \"e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp\" (UID: \"cbc558d1-0eff-46fb-a780-7ebcc1939928\") " pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"
Nov 01 03:38:36.905264 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:36.905246    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc558d1-0eff-46fb-a780-7ebcc1939928-credential-socket\") pod \"e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp\" (UID: \"cbc558d1-0eff-46fb-a780-7ebcc1939928\") " pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"
Nov 01 03:38:36.913730 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:36.905383    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc558d1-0eff-46fb-a780-7ebcc1939928-workload-socket\") pod \"e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp\" (UID: \"cbc558d1-0eff-46fb-a780-7ebcc1939928\") " pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"
Nov 01 03:38:36.921459 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:36.828472    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-da4f30-openshift-default-dockercfg-8kwqq\""
Nov 01 03:38:36.921459 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:36.847977    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"]
Nov 01 03:38:37.036584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:37.027186    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc558d1-0eff-46fb-a780-7ebcc1939928-istio-data\") pod \"e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp\" (UID: \"cbc558d1-0eff-46fb-a780-7ebcc1939928\") " pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"
Nov 01 03:38:37.036584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:37.027258    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc558d1-0eff-46fb-a780-7ebcc1939928-istiod-ca-cert\") pod \"e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp\" (UID: \"cbc558d1-0eff-46fb-a780-7ebcc1939928\") " pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"
Nov 01 03:38:37.036584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:37.027300    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc558d1-0eff-46fb-a780-7ebcc1939928-istio-envoy\") pod \"e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp\" (UID: \"cbc558d1-0eff-46fb-a780-7ebcc1939928\") " pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"
Nov 01 03:38:37.036584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:37.027403    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc558d1-0eff-46fb-a780-7ebcc1939928-credential-socket\") pod \"e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp\" (UID: \"cbc558d1-0eff-46fb-a780-7ebcc1939928\") " pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"
Nov 01 03:38:37.036584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:37.027440    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc558d1-0eff-46fb-a780-7ebcc1939928-workload-socket\") pod \"e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp\" (UID: \"cbc558d1-0eff-46fb-a780-7ebcc1939928\") " pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"
Nov 01 03:38:37.036584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:37.027535    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbc558d1-0eff-46fb-a780-7ebcc1939928-istio-token\") pod \"e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp\" (UID: \"cbc558d1-0eff-46fb-a780-7ebcc1939928\") " pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"
Nov 01 03:38:37.036584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:37.027589    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhsz5\" (UniqueName: \"kubernetes.io/projected/cbc558d1-0eff-46fb-a780-7ebcc1939928-kube-api-access-nhsz5\") pod \"e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp\" (UID: \"cbc558d1-0eff-46fb-a780-7ebcc1939928\") " pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"
Nov 01 03:38:37.036584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:37.027649    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc558d1-0eff-46fb-a780-7ebcc1939928-workload-certs\") pod \"e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp\" (UID: \"cbc558d1-0eff-46fb-a780-7ebcc1939928\") " pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"
Nov 01 03:38:37.036584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:37.027693    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc558d1-0eff-46fb-a780-7ebcc1939928-istio-podinfo\") pod \"e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp\" (UID: \"cbc558d1-0eff-46fb-a780-7ebcc1939928\") " pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"
Nov 01 03:38:37.036584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:37.028502    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc558d1-0eff-46fb-a780-7ebcc1939928-credential-socket\") pod \"e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp\" (UID: \"cbc558d1-0eff-46fb-a780-7ebcc1939928\") " pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"
Nov 01 03:38:37.036584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:37.028827    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc558d1-0eff-46fb-a780-7ebcc1939928-istio-data\") pod \"e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp\" (UID: \"cbc558d1-0eff-46fb-a780-7ebcc1939928\") " pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"
Nov 01 03:38:37.036584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:37.029439    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc558d1-0eff-46fb-a780-7ebcc1939928-istiod-ca-cert\") pod \"e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp\" (UID: \"cbc558d1-0eff-46fb-a780-7ebcc1939928\") " pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"
Nov 01 03:38:37.036584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:37.031157    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc558d1-0eff-46fb-a780-7ebcc1939928-workload-socket\") pod \"e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp\" (UID: \"cbc558d1-0eff-46fb-a780-7ebcc1939928\") " pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"
Nov 01 03:38:37.036584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:37.032230    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc558d1-0eff-46fb-a780-7ebcc1939928-workload-certs\") pod \"e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp\" (UID: \"cbc558d1-0eff-46fb-a780-7ebcc1939928\") " pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"
Nov 01 03:38:37.036584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:37.032667    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc558d1-0eff-46fb-a780-7ebcc1939928-istio-envoy\") pod \"e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp\" (UID: \"cbc558d1-0eff-46fb-a780-7ebcc1939928\") " pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"
Nov 01 03:38:37.039885 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:37.039827    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc558d1-0eff-46fb-a780-7ebcc1939928-istio-podinfo\") pod \"e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp\" (UID: \"cbc558d1-0eff-46fb-a780-7ebcc1939928\") " pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"
Nov 01 03:38:37.044045 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:37.044003    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:38:37.133796 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:37.127982    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbc558d1-0eff-46fb-a780-7ebcc1939928-istio-token\") pod \"e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp\" (UID: \"cbc558d1-0eff-46fb-a780-7ebcc1939928\") " pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"
Nov 01 03:38:37.133796 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:37.133281    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhsz5\" (UniqueName: \"kubernetes.io/projected/cbc558d1-0eff-46fb-a780-7ebcc1939928-kube-api-access-nhsz5\") pod \"e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp\" (UID: \"cbc558d1-0eff-46fb-a780-7ebcc1939928\") " pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"
Nov 01 03:38:37.162837 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:37.162803    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"
Nov 01 03:38:37.516899 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:37.515784    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:38:37.722899 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:38:37.722481    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbc558d1_0eff_46fb_a780_7ebcc1939928.slice/crio-731f50264c3ac9dfe2dccae7e1687eb149d36f0e501ce78fba96a0df6e202aef WatchSource:0}: Error finding container 731f50264c3ac9dfe2dccae7e1687eb149d36f0e501ce78fba96a0df6e202aef: Status 404 returned error can't find the container with id 731f50264c3ac9dfe2dccae7e1687eb149d36f0e501ce78fba96a0df6e202aef
Nov 01 03:38:37.724639 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:37.724608    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"]
Nov 01 03:38:37.725472 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:37.725441    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:38:37.725529 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:37.725507    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:38:37.725603 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:37.725536    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:38:38.659827 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:38.659608    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp" event={"ID":"cbc558d1-0eff-46fb-a780-7ebcc1939928","Type":"ContainerStarted","Data":"9081f00c6971d46de11917b10aa3ec83e72de04f3eea18d8d3d871a37934e8af"}
Nov 01 03:38:38.659827 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:38.659681    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp" event={"ID":"cbc558d1-0eff-46fb-a780-7ebcc1939928","Type":"ContainerStarted","Data":"731f50264c3ac9dfe2dccae7e1687eb149d36f0e501ce78fba96a0df6e202aef"}
Nov 01 03:38:38.890472 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:38.890404    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp" podStartSLOduration=2.890386015 podStartE2EDuration="2.890386015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:38:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:38:38.888832797 +0000 UTC m=+1923.411839386" watchObservedRunningTime="2026-11-01 03:38:38.890386015 +0000 UTC m=+1923.413392607"
Nov 01 03:38:39.189708 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:39.187382    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"
Nov 01 03:38:39.193434 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:39.193411    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"
Nov 01 03:38:39.671714 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:39.671674    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"
Nov 01 03:38:39.672780 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:39.672751    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"
Nov 01 03:38:40.876963 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:40.876917    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:38:40.992273 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:40.988766    4930 patch_prober.go:29] interesting pod/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 03:38:40.992273 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:40.988844    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8" podUID="a30e9e3f-be06-449e-84a9-e1e1b8be4d1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:38:42.245472 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:42.245430    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:38:42.415851 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:42.415792    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:38:42.590727 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:42.590690    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:38:45.732909 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:45.732828    4930 generic.go:356] "Generic (PLEG): container finished" podID="baec5da4-406d-4923-aa69-69820f5e04c0" containerID="231cb1e1808fd5e653cc3939494ddc5f5c5e544420b1a5f88cc0daab42aa7d2e" exitCode=0
Nov 01 03:38:45.732909 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:45.732897    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8" event={"ID":"baec5da4-406d-4923-aa69-69820f5e04c0","Type":"ContainerDied","Data":"231cb1e1808fd5e653cc3939494ddc5f5c5e544420b1a5f88cc0daab42aa7d2e"}
Nov 01 03:38:45.733383 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:45.733353    4930 scope.go:122] "RemoveContainer" containerID="231cb1e1808fd5e653cc3939494ddc5f5c5e544420b1a5f88cc0daab42aa7d2e"
Nov 01 03:38:45.735060 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:45.735030    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:38:45.735145 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:45.735105    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:38:45.735191 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:45.735157    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:38:46.120443 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:46.120401    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:38:46.321713 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:46.321620    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:38:46.540391 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:46.540354    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-da4f30-647ffb7887-skpkm"]
Nov 01 03:38:46.662785 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:46.662750    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-da4f30-647ffb7887-skpkm"]
Nov 01 03:38:46.662972 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:46.662885    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da4f30-647ffb7887-skpkm"
Nov 01 03:38:46.702621 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:46.699290    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-da4f30-646b9f5d65-qhxwx"]
Nov 01 03:38:46.705524 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:46.705497    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-da4f30-646b9f5d65-qhxwx"
Nov 01 03:38:46.734579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:46.725002    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-da4f30-dockercfg-nxl82\""
Nov 01 03:38:46.734579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:46.725127    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-da4f30\""
Nov 01 03:38:46.734579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:46.730010    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:38:46.791631 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:46.772145    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3077cb58-8c2b-48fd-a777-dafaecbce99e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-da4f30-647ffb7887-skpkm\" (UID: \"3077cb58-8c2b-48fd-a777-dafaecbce99e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-da4f30-647ffb7887-skpkm"
Nov 01 03:38:46.791631 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:46.772448    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsmsc\" (UniqueName: \"kubernetes.io/projected/3077cb58-8c2b-48fd-a777-dafaecbce99e-kube-api-access-rsmsc\") pod \"payload-pre-processing-e2e-api-b-da4f30-647ffb7887-skpkm\" (UID: \"3077cb58-8c2b-48fd-a777-dafaecbce99e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-da4f30-647ffb7887-skpkm"
Nov 01 03:38:46.791631 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:46.784627    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8" event={"ID":"baec5da4-406d-4923-aa69-69820f5e04c0","Type":"ContainerStarted","Data":"7e5397471b831ab3918ce97d528cff62e209f15466024e7a6e29760e8ab7ce3a"}
Nov 01 03:38:46.884230 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:46.884190    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-da4f30-646b9f5d65-qhxwx"]
Nov 01 03:38:46.932587 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:46.898270    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsmsc\" (UniqueName: \"kubernetes.io/projected/3077cb58-8c2b-48fd-a777-dafaecbce99e-kube-api-access-rsmsc\") pod \"payload-pre-processing-e2e-api-b-da4f30-647ffb7887-skpkm\" (UID: \"3077cb58-8c2b-48fd-a777-dafaecbce99e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-da4f30-647ffb7887-skpkm"
Nov 01 03:38:46.932587 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:46.898333    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwnxn\" (UniqueName: \"kubernetes.io/projected/08c53d46-b65c-4799-98f7-9bd64e561ebc-kube-api-access-zwnxn\") pod \"payload-processing-e2e-api-b-da4f30-646b9f5d65-qhxwx\" (UID: \"08c53d46-b65c-4799-98f7-9bd64e561ebc\") " pod="openshift-ingress/payload-processing-e2e-api-b-da4f30-646b9f5d65-qhxwx"
Nov 01 03:38:46.932587 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:46.898682    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3077cb58-8c2b-48fd-a777-dafaecbce99e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-da4f30-647ffb7887-skpkm\" (UID: \"3077cb58-8c2b-48fd-a777-dafaecbce99e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-da4f30-647ffb7887-skpkm"
Nov 01 03:38:46.932587 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:46.898827    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08c53d46-b65c-4799-98f7-9bd64e561ebc-plugins-config-volume\") pod \"payload-processing-e2e-api-b-da4f30-646b9f5d65-qhxwx\" (UID: \"08c53d46-b65c-4799-98f7-9bd64e561ebc\") " pod="openshift-ingress/payload-processing-e2e-api-b-da4f30-646b9f5d65-qhxwx"
Nov 01 03:38:46.932587 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:46.900634    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3077cb58-8c2b-48fd-a777-dafaecbce99e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-da4f30-647ffb7887-skpkm\" (UID: \"3077cb58-8c2b-48fd-a777-dafaecbce99e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-da4f30-647ffb7887-skpkm"
Nov 01 03:38:46.982584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:46.969660    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:38:46.982584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:46.972156    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsmsc\" (UniqueName: \"kubernetes.io/projected/3077cb58-8c2b-48fd-a777-dafaecbce99e-kube-api-access-rsmsc\") pod \"payload-pre-processing-e2e-api-b-da4f30-647ffb7887-skpkm\" (UID: \"3077cb58-8c2b-48fd-a777-dafaecbce99e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-da4f30-647ffb7887-skpkm"
Nov 01 03:38:47.086056 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:47.000590    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08c53d46-b65c-4799-98f7-9bd64e561ebc-plugins-config-volume\") pod \"payload-processing-e2e-api-b-da4f30-646b9f5d65-qhxwx\" (UID: \"08c53d46-b65c-4799-98f7-9bd64e561ebc\") " pod="openshift-ingress/payload-processing-e2e-api-b-da4f30-646b9f5d65-qhxwx"
Nov 01 03:38:47.086056 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:47.000715    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwnxn\" (UniqueName: \"kubernetes.io/projected/08c53d46-b65c-4799-98f7-9bd64e561ebc-kube-api-access-zwnxn\") pod \"payload-processing-e2e-api-b-da4f30-646b9f5d65-qhxwx\" (UID: \"08c53d46-b65c-4799-98f7-9bd64e561ebc\") " pod="openshift-ingress/payload-processing-e2e-api-b-da4f30-646b9f5d65-qhxwx"
Nov 01 03:38:47.086056 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:47.001982    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08c53d46-b65c-4799-98f7-9bd64e561ebc-plugins-config-volume\") pod \"payload-processing-e2e-api-b-da4f30-646b9f5d65-qhxwx\" (UID: \"08c53d46-b65c-4799-98f7-9bd64e561ebc\") " pod="openshift-ingress/payload-processing-e2e-api-b-da4f30-646b9f5d65-qhxwx"
Nov 01 03:38:47.086056 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:47.005344    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da4f30-647ffb7887-skpkm"
Nov 01 03:38:47.086056 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:47.069630    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwnxn\" (UniqueName: \"kubernetes.io/projected/08c53d46-b65c-4799-98f7-9bd64e561ebc-kube-api-access-zwnxn\") pod \"payload-processing-e2e-api-b-da4f30-646b9f5d65-qhxwx\" (UID: \"08c53d46-b65c-4799-98f7-9bd64e561ebc\") " pod="openshift-ingress/payload-processing-e2e-api-b-da4f30-646b9f5d65-qhxwx"
Nov 01 03:38:47.344349 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:47.344310    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-da4f30-646b9f5d65-qhxwx"
Nov 01 03:38:47.404702 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:47.403678    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"
Nov 01 03:38:47.404702 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:47.403893    4930 patch_prober.go:29] interesting pod/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Nov 01 03:38:47.404702 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:47.403997    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8" podUID="baec5da4-406d-4923-aa69-69820f5e04c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Nov 01 03:38:47.525604 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:47.520156    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-da4f30-647ffb7887-skpkm"]
Nov 01 03:38:47.529643 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:38:47.529598    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3077cb58_8c2b_48fd_a777_dafaecbce99e.slice/crio-bd4630d06284298bcce64e9c8d8b07c755094ae463c9ff569f4a6b3ac28ae8d3 WatchSource:0}: Error finding container bd4630d06284298bcce64e9c8d8b07c755094ae463c9ff569f4a6b3ac28ae8d3: Status 404 returned error can't find the container with id bd4630d06284298bcce64e9c8d8b07c755094ae463c9ff569f4a6b3ac28ae8d3
Nov 01 03:38:47.803483 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:47.802879    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da4f30-647ffb7887-skpkm" event={"ID":"3077cb58-8c2b-48fd-a777-dafaecbce99e","Type":"ContainerStarted","Data":"bb02cd4d37f900d46d6afe8a6bc10998fd656ca63a82422db6841282a80ce2d9"}
Nov 01 03:38:47.803483 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:47.802928    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da4f30-647ffb7887-skpkm" event={"ID":"3077cb58-8c2b-48fd-a777-dafaecbce99e","Type":"ContainerStarted","Data":"bd4630d06284298bcce64e9c8d8b07c755094ae463c9ff569f4a6b3ac28ae8d3"}
Nov 01 03:38:47.839426 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:38:47.839387    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08c53d46_b65c_4799_98f7_9bd64e561ebc.slice/crio-0f09ae6e899ecb55b9db6751dd6640924a1dc59915f3e1f92bb544ae688537a8 WatchSource:0}: Error finding container 0f09ae6e899ecb55b9db6751dd6640924a1dc59915f3e1f92bb544ae688537a8: Status 404 returned error can't find the container with id 0f09ae6e899ecb55b9db6751dd6640924a1dc59915f3e1f92bb544ae688537a8
Nov 01 03:38:47.840519 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:47.840496    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-da4f30-646b9f5d65-qhxwx"]
Nov 01 03:38:47.934173 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:47.934118    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da4f30-647ffb7887-skpkm" podStartSLOduration=1.9341010440000002 podStartE2EDuration="1.934101044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:38:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:38:47.928196831 +0000 UTC m=+1932.451203420" watchObservedRunningTime="2026-11-01 03:38:47.934101044 +0000 UTC m=+1932.457107632"
Nov 01 03:38:48.401450 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:48.401416    4930 patch_prober.go:29] interesting pod/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Nov 01 03:38:48.401642 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:48.401479    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8" podUID="baec5da4-406d-4923-aa69-69820f5e04c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Nov 01 03:38:48.815614 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:48.815530    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-da4f30-646b9f5d65-qhxwx" event={"ID":"08c53d46-b65c-4799-98f7-9bd64e561ebc","Type":"ContainerStarted","Data":"ac37a90661a61da657110f8ee03e900f4cb61a39587b955536ef5266da847f66"}
Nov 01 03:38:48.816012 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:48.815624    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-da4f30-646b9f5d65-qhxwx" event={"ID":"08c53d46-b65c-4799-98f7-9bd64e561ebc","Type":"ContainerStarted","Data":"0f09ae6e899ecb55b9db6751dd6640924a1dc59915f3e1f92bb544ae688537a8"}
Nov 01 03:38:48.816012 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:48.815676    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da4f30-647ffb7887-skpkm"
Nov 01 03:38:48.943386 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:48.943330    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-da4f30-646b9f5d65-qhxwx" podStartSLOduration=2.943312451 podStartE2EDuration="2.943312451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:38:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:38:48.941706388 +0000 UTC m=+1933.464712976" watchObservedRunningTime="2026-11-01 03:38:48.943312451 +0000 UTC m=+1933.466319040"
Nov 01 03:38:49.152329 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:38:49.152291    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:38:49.401871 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:49.401837    4930 patch_prober.go:29] interesting pod/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Nov 01 03:38:49.402046 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:49.401891    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8" podUID="baec5da4-406d-4923-aa69-69820f5e04c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Nov 01 03:38:49.825181 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:49.825143    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-da4f30-646b9f5d65-qhxwx"
Nov 01 03:38:50.402156 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:50.402118    4930 patch_prober.go:29] interesting pod/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Nov 01 03:38:50.402372 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:50.402181    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8" podUID="baec5da4-406d-4923-aa69-69820f5e04c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Nov 01 03:38:50.522364 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:50.522330    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh"]
Nov 01 03:38:50.541934 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:50.541903    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh"
Nov 01 03:38:50.587229 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:50.587184    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0d37273-4961-471d-a3e6-3b48c1cc0acb-istio-token\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh\" (UID: \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh"
Nov 01 03:38:50.587417 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:50.587254    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0d37273-4961-471d-a3e6-3b48c1cc0acb-istiod-ca-cert\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh\" (UID: \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh"
Nov 01 03:38:50.587417 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:50.587362    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0d37273-4961-471d-a3e6-3b48c1cc0acb-istio-data\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh\" (UID: \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh"
Nov 01 03:38:50.587417 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:50.587396    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d37273-4961-471d-a3e6-3b48c1cc0acb-workload-socket\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh\" (UID: \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh"
Nov 01 03:38:50.587708 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:50.587486    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f2dd\" (UniqueName: \"kubernetes.io/projected/a0d37273-4961-471d-a3e6-3b48c1cc0acb-kube-api-access-9f2dd\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh\" (UID: \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh"
Nov 01 03:38:50.587708 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:50.587547    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d37273-4961-471d-a3e6-3b48c1cc0acb-credential-socket\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh\" (UID: \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh"
Nov 01 03:38:50.587708 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:50.587621    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0d37273-4961-471d-a3e6-3b48c1cc0acb-istio-podinfo\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh\" (UID: \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh"
Nov 01 03:38:50.587708 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:50.587652    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0d37273-4961-471d-a3e6-3b48c1cc0acb-workload-certs\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh\" (UID: \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh"
Nov 01 03:38:50.587937 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:50.587732    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0d37273-4961-471d-a3e6-3b48c1cc0acb-istio-envoy\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh\" (UID: \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh"
Nov 01 03:38:50.690975 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:50.690891    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0d37273-4961-471d-a3e6-3b48c1cc0acb-istio-envoy\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh\" (UID: \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh"
Nov 01 03:38:50.692005 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:50.691979    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0d37273-4961-471d-a3e6-3b48c1cc0acb-istio-token\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh\" (UID: \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh"
Nov 01 03:38:50.692742 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:50.692721    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0d37273-4961-471d-a3e6-3b48c1cc0acb-istiod-ca-cert\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh\" (UID: \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh"
Nov 01 03:38:50.693662 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:50.693542    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0d37273-4961-471d-a3e6-3b48c1cc0acb-istiod-ca-cert\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh\" (UID: \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh"
Nov 01 03:38:50.693944 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:50.693928    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0d37273-4961-471d-a3e6-3b48c1cc0acb-istio-data\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh\" (UID: \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh"
Nov 01 03:38:50.694374 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:50.694286    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0d37273-4961-471d-a3e6-3b48c1cc0acb-istio-data\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh\" (UID: \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh"
Nov 01 03:38:50.694529 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:50.694512    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d37273-4961-471d-a3e6-3b48c1cc0acb-workload-socket\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh\" (UID: \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh"
Nov 01 03:38:50.694996 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:50.694884    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d37273-4961-471d-a3e6-3b48c1cc0acb-workload-socket\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh\" (UID: \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh"
Nov 01 03:38:50.695810 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:50.695790    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f2dd\" (UniqueName: \"kubernetes.io/projected/a0d37273-4961-471d-a3e6-3b48c1cc0acb-kube-api-access-9f2dd\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh\" (UID: \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh"
Nov 01 03:38:50.696017 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:50.696002    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d37273-4961-471d-a3e6-3b48c1cc0acb-credential-socket\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh\" (UID: \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh"
Nov 01 03:38:50.696427 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:50.696412    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0d37273-4961-471d-a3e6-3b48c1cc0acb-istio-podinfo\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh\" (UID: \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh"
Nov 01 03:38:50.696983 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:50.696956    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0d37273-4961-471d-a3e6-3b48c1cc0acb-workload-certs\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh\" (UID: \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh"
Nov 01 03:38:50.697522 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:50.697419    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0d37273-4961-471d-a3e6-3b48c1cc0acb-workload-certs\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh\" (UID: \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh"
Nov 01 03:38:50.697667 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:50.696352    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d37273-4961-471d-a3e6-3b48c1cc0acb-credential-socket\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh\" (UID: \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh"
Nov 01 03:38:50.699761 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:50.699733    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0d37273-4961-471d-a3e6-3b48c1cc0acb-istio-podinfo\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh\" (UID: \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh"
Nov 01 03:38:50.699984 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:50.699944    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0d37273-4961-471d-a3e6-3b48c1cc0acb-istio-envoy\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh\" (UID: \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh"
Nov 01 03:38:50.701957 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:50.701913    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh"]
Nov 01 03:38:50.758613 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:50.757596    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0d37273-4961-471d-a3e6-3b48c1cc0acb-istio-token\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh\" (UID: \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh"
Nov 01 03:38:50.759859 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:50.759827    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f2dd\" (UniqueName: \"kubernetes.io/projected/a0d37273-4961-471d-a3e6-3b48c1cc0acb-kube-api-access-9f2dd\") pod \"e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh\" (UID: \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\") " pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh"
Nov 01 03:38:50.885863 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:50.885825    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh"
Nov 01 03:38:51.197638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:51.197547    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"
Nov 01 03:38:51.403045 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:51.401869    4930 patch_prober.go:29] interesting pod/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Nov 01 03:38:51.403045 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:51.401935    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8" podUID="baec5da4-406d-4923-aa69-69820f5e04c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Nov 01 03:38:51.421244 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:38:51.421208    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0d37273_4961_471d_a3e6_3b48c1cc0acb.slice/crio-3094de6a14a30413d34d67451711d3cd511eaf434aff879dd33cb88ea3b239f2 WatchSource:0}: Error finding container 3094de6a14a30413d34d67451711d3cd511eaf434aff879dd33cb88ea3b239f2: Status 404 returned error can't find the container with id 3094de6a14a30413d34d67451711d3cd511eaf434aff879dd33cb88ea3b239f2
Nov 01 03:38:51.421534 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:51.421514    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh"]
Nov 01 03:38:51.424173 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:51.424143    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:38:51.424279 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:51.424204    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:38:51.424279 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:51.424232    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:38:51.843419 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:51.843319    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh" event={"ID":"a0d37273-4961-471d-a3e6-3b48c1cc0acb","Type":"ContainerStarted","Data":"26c389b331289fee223f580d31fa222247d670b55724104f98aabb7abe0f91ec"}
Nov 01 03:38:51.843419 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:51.843381    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh" event={"ID":"a0d37273-4961-471d-a3e6-3b48c1cc0acb","Type":"ContainerStarted","Data":"3094de6a14a30413d34d67451711d3cd511eaf434aff879dd33cb88ea3b239f2"}
Nov 01 03:38:51.893581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:51.887035    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh"
Nov 01 03:38:51.980914 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:51.966454    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh" podStartSLOduration=1.966436238 podStartE2EDuration="1.966436238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:38:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:38:51.965522726 +0000 UTC m=+1936.488529315" watchObservedRunningTime="2026-11-01 03:38:51.966436238 +0000 UTC m=+1936.489442827"
Nov 01 03:38:52.358593 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:52.354765    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:38:52.455580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:52.454750    4930 patch_prober.go:29] interesting pod/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body=
Nov 01 03:38:52.455580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:52.454807    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8" podUID="baec5da4-406d-4923-aa69-69820f5e04c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused"
Nov 01 03:38:52.896765 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:52.896720    4930 patch_prober.go:29] interesting pod/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 01 03:38:52.898085 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:52.896786    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh" podUID="a0d37273-4961-471d-a3e6-3b48c1cc0acb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 01 03:38:53.406874 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:53.406843    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"
Nov 01 03:38:53.408087 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:53.408062    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"
Nov 01 03:38:53.886576 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:53.885230    4930 generic.go:356] "Generic (PLEG): container finished" podID="a30e9e3f-be06-449e-84a9-e1e1b8be4d1d" containerID="190679fadf2457c52f69239758b5f78a2f5b923dfef156f24191b6a81729fe1b" exitCode=0
Nov 01 03:38:53.886576 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:53.886114    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8" event={"ID":"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d","Type":"ContainerDied","Data":"190679fadf2457c52f69239758b5f78a2f5b923dfef156f24191b6a81729fe1b"}
Nov 01 03:38:53.886576 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:53.886471    4930 scope.go:122] "RemoveContainer" containerID="190679fadf2457c52f69239758b5f78a2f5b923dfef156f24191b6a81729fe1b"
Nov 01 03:38:53.891392 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:53.891351    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:38:53.891497 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:53.891406    4930 patch_prober.go:29] interesting pod/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 01 03:38:53.891497 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:53.891447    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:38:53.891698 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:53.891494    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:38:53.891912 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:53.891444    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh" podUID="a0d37273-4961-471d-a3e6-3b48c1cc0acb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 01 03:38:54.927120 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:54.927076    4930 patch_prober.go:29] interesting pod/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 01 03:38:54.927799 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:54.927764    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh" podUID="a0d37273-4961-471d-a3e6-3b48c1cc0acb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 01 03:38:54.930739 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:54.930703    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8" event={"ID":"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d","Type":"ContainerStarted","Data":"6c690034d6f66d2aa5b9954df542badb601e62dc8f3ff86c35bdff923f8753af"}
Nov 01 03:38:54.931301 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:54.931275    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"
Nov 01 03:38:55.645979 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:55.645913    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"
Nov 01 03:38:55.647007 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:55.646985    4930 patch_prober.go:29] interesting pod/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 01 03:38:55.647075 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:55.647031    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8" podUID="a30e9e3f-be06-449e-84a9-e1e1b8be4d1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 01 03:38:55.890362 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:55.889905    4930 patch_prober.go:29] interesting pod/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 01 03:38:55.890362 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:55.889967    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh" podUID="a0d37273-4961-471d-a3e6-3b48c1cc0acb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 01 03:38:56.646085 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:56.646050    4930 patch_prober.go:29] interesting pod/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 01 03:38:56.646597 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:56.646109    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8" podUID="a30e9e3f-be06-449e-84a9-e1e1b8be4d1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 01 03:38:56.890349 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:56.889899    4930 patch_prober.go:29] interesting pod/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 01 03:38:56.890349 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:56.889962    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh" podUID="a0d37273-4961-471d-a3e6-3b48c1cc0acb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 01 03:38:57.424065 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:57.423303    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:38:57.645825 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:57.645787    4930 patch_prober.go:29] interesting pod/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 01 03:38:57.646024 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:57.645845    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8" podUID="a30e9e3f-be06-449e-84a9-e1e1b8be4d1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 01 03:38:57.887493 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:57.887456    4930 patch_prober.go:29] interesting pod/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 01 03:38:57.888109 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:57.888077    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh" podUID="a0d37273-4961-471d-a3e6-3b48c1cc0acb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 01 03:38:58.646144 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:58.646108    4930 patch_prober.go:29] interesting pod/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 01 03:38:58.646325 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:58.646159    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8" podUID="a30e9e3f-be06-449e-84a9-e1e1b8be4d1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 01 03:38:58.897893 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:58.897802    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh"
Nov 01 03:38:58.898693 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:58.898669    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh"
Nov 01 03:38:58.899463 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:58.899442    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh"
Nov 01 03:38:59.646405 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:59.646376    4930 patch_prober.go:29] interesting pod/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body=
Nov 01 03:38:59.646627 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:59.646433    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8" podUID="a30e9e3f-be06-449e-84a9-e1e1b8be4d1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused"
Nov 01 03:38:59.828898 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:38:59.828865    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da4f30-647ffb7887-skpkm"
Nov 01 03:39:00.650718 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:00.650689    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"
Nov 01 03:39:00.651988 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:00.651964    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"
Nov 01 03:39:00.837258 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:00.837226    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-da4f30-646b9f5d65-qhxwx"
Nov 01 03:39:01.289014 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:01.288971    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fdbdfefd-ttbq5"]
Nov 01 03:39:01.292372 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:01.292349    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fdbdfefd-ttbq5"
Nov 01 03:39:01.345673 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:01.345641    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:39:01.353768 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:01.353734    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fdbdfefd-ttbq5"]
Nov 01 03:39:01.354887 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:01.354844    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92ed3fc3-dd13-42e8-a20e-84e0ec2de11a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-fdbdfefd-ttbq5\" (UID: \"92ed3fc3-dd13-42e8-a20e-84e0ec2de11a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fdbdfefd-ttbq5"
Nov 01 03:39:01.355078 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:01.355050    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqtgx\" (UniqueName: \"kubernetes.io/projected/92ed3fc3-dd13-42e8-a20e-84e0ec2de11a-kube-api-access-pqtgx\") pod \"maas-api-revoke-keys-e2e-mt-fdbdfefd-ttbq5\" (UID: \"92ed3fc3-dd13-42e8-a20e-84e0ec2de11a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fdbdfefd-ttbq5"
Nov 01 03:39:01.355169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:01.355150    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92ed3fc3-dd13-42e8-a20e-84e0ec2de11a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-fdbdfefd-ttbq5\" (UID: \"92ed3fc3-dd13-42e8-a20e-84e0ec2de11a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fdbdfefd-ttbq5"
Nov 01 03:39:01.359998 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:01.359974    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:39:01.360124 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:01.360067    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:39:01.446861 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:01.446831    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:39:01.456803 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:01.456771    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92ed3fc3-dd13-42e8-a20e-84e0ec2de11a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-fdbdfefd-ttbq5\" (UID: \"92ed3fc3-dd13-42e8-a20e-84e0ec2de11a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fdbdfefd-ttbq5"
Nov 01 03:39:01.456943 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:01.456906    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqtgx\" (UniqueName: \"kubernetes.io/projected/92ed3fc3-dd13-42e8-a20e-84e0ec2de11a-kube-api-access-pqtgx\") pod \"maas-api-revoke-keys-e2e-mt-fdbdfefd-ttbq5\" (UID: \"92ed3fc3-dd13-42e8-a20e-84e0ec2de11a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fdbdfefd-ttbq5"
Nov 01 03:39:01.456985 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:01.456968    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92ed3fc3-dd13-42e8-a20e-84e0ec2de11a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-fdbdfefd-ttbq5\" (UID: \"92ed3fc3-dd13-42e8-a20e-84e0ec2de11a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fdbdfefd-ttbq5"
Nov 01 03:39:01.457643 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:01.457614    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92ed3fc3-dd13-42e8-a20e-84e0ec2de11a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-fdbdfefd-ttbq5\" (UID: \"92ed3fc3-dd13-42e8-a20e-84e0ec2de11a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fdbdfefd-ttbq5"
Nov 01 03:39:01.459439 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:01.459413    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92ed3fc3-dd13-42e8-a20e-84e0ec2de11a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-fdbdfefd-ttbq5\" (UID: \"92ed3fc3-dd13-42e8-a20e-84e0ec2de11a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fdbdfefd-ttbq5"
Nov 01 03:39:01.523973 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:01.523943    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqtgx\" (UniqueName: \"kubernetes.io/projected/92ed3fc3-dd13-42e8-a20e-84e0ec2de11a-kube-api-access-pqtgx\") pod \"maas-api-revoke-keys-e2e-mt-fdbdfefd-ttbq5\" (UID: \"92ed3fc3-dd13-42e8-a20e-84e0ec2de11a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fdbdfefd-ttbq5"
Nov 01 03:39:01.607129 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:01.607097    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fdbdfefd-ttbq5"
Nov 01 03:39:01.993367 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:39:01.993317    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92ed3fc3_dd13_42e8_a20e_84e0ec2de11a.slice/crio-49d35464bf5a48e28857789db11fbf878ec34269ce582c59f4c39d21bd603f37 WatchSource:0}: Error finding container 49d35464bf5a48e28857789db11fbf878ec34269ce582c59f4c39d21bd603f37: Status 404 returned error can't find the container with id 49d35464bf5a48e28857789db11fbf878ec34269ce582c59f4c39d21bd603f37
Nov 01 03:39:01.994225 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:01.994127    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fdbdfefd-ttbq5"]
Nov 01 03:39:02.039653 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:02.039628    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fdbdfefd-ttbq5" event={"ID":"92ed3fc3-dd13-42e8-a20e-84e0ec2de11a","Type":"ContainerStarted","Data":"49d35464bf5a48e28857789db11fbf878ec34269ce582c59f4c39d21bd603f37"}
Nov 01 03:39:02.483842 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:02.483806    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:39:03.050164 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:03.050130    4930 generic.go:356] "Generic (PLEG): container finished" podID="92ed3fc3-dd13-42e8-a20e-84e0ec2de11a" containerID="97b71e2a40920e80db9b74b783bd2ba54980c12507d6dc99fa8e079219a23366" exitCode=0
Nov 01 03:39:03.050614 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:03.050203    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fdbdfefd-ttbq5" event={"ID":"92ed3fc3-dd13-42e8-a20e-84e0ec2de11a","Type":"ContainerDied","Data":"97b71e2a40920e80db9b74b783bd2ba54980c12507d6dc99fa8e079219a23366"}
Nov 01 03:39:04.248517 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:04.248475    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fdbdfefd-ttbq5"
Nov 01 03:39:04.300467 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:04.300437    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92ed3fc3-dd13-42e8-a20e-84e0ec2de11a-kube-api-access-pqtgx\" (UniqueName: \"kubernetes.io/projected/92ed3fc3-dd13-42e8-a20e-84e0ec2de11a-kube-api-access-pqtgx\") pod \"92ed3fc3-dd13-42e8-a20e-84e0ec2de11a\" (UID: \"92ed3fc3-dd13-42e8-a20e-84e0ec2de11a\") "
Nov 01 03:39:04.300650 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:04.300599    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92ed3fc3-dd13-42e8-a20e-84e0ec2de11a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92ed3fc3-dd13-42e8-a20e-84e0ec2de11a-maas-api-xks-ca\") pod \"92ed3fc3-dd13-42e8-a20e-84e0ec2de11a\" (UID: \"92ed3fc3-dd13-42e8-a20e-84e0ec2de11a\") "
Nov 01 03:39:04.300650 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:04.300641    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92ed3fc3-dd13-42e8-a20e-84e0ec2de11a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92ed3fc3-dd13-42e8-a20e-84e0ec2de11a-maas-api-openshift-ca\") pod \"92ed3fc3-dd13-42e8-a20e-84e0ec2de11a\" (UID: \"92ed3fc3-dd13-42e8-a20e-84e0ec2de11a\") "
Nov 01 03:39:04.301184 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:04.301153    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92ed3fc3-dd13-42e8-a20e-84e0ec2de11a-maas-api-openshift-ca" pod "92ed3fc3-dd13-42e8-a20e-84e0ec2de11a" (UID: "92ed3fc3-dd13-42e8-a20e-84e0ec2de11a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:39:04.301581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:04.301518    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92ed3fc3-dd13-42e8-a20e-84e0ec2de11a-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:39:04.303001 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:04.302971    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92ed3fc3-dd13-42e8-a20e-84e0ec2de11a-kube-api-access-pqtgx" pod "92ed3fc3-dd13-42e8-a20e-84e0ec2de11a" (UID: "92ed3fc3-dd13-42e8-a20e-84e0ec2de11a"). InnerVolumeSpecName "kube-api-access-pqtgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:39:04.303094 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:04.302993    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92ed3fc3-dd13-42e8-a20e-84e0ec2de11a-maas-api-xks-ca" pod "92ed3fc3-dd13-42e8-a20e-84e0ec2de11a" (UID: "92ed3fc3-dd13-42e8-a20e-84e0ec2de11a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:39:04.402852 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:04.402813    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqtgx\" (UniqueName: \"kubernetes.io/projected/92ed3fc3-dd13-42e8-a20e-84e0ec2de11a-kube-api-access-pqtgx\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:39:04.402852 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:04.402855    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92ed3fc3-dd13-42e8-a20e-84e0ec2de11a-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:39:05.072320 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:05.072289    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fdbdfefd-ttbq5" event={"ID":"92ed3fc3-dd13-42e8-a20e-84e0ec2de11a","Type":"ContainerDied","Data":"49d35464bf5a48e28857789db11fbf878ec34269ce582c59f4c39d21bd603f37"}
Nov 01 03:39:05.072320 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:05.072314    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fdbdfefd-ttbq5"
Nov 01 03:39:05.072570 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:05.072337    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49d35464bf5a48e28857789db11fbf878ec34269ce582c59f4c39d21bd603f37"
Nov 01 03:39:07.797217 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:07.797172    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:39:13.092671 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:13.092632    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:39:13.410393 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:13.409424    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:39:13.617830 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:13.617655    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:39:17.870213 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:17.870049    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:39:19.123176 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:39:19.123142    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:39:19.241402 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:19.241317    4930 generic.go:356] "Generic (PLEG): container finished" podID="68863127-fc50-48b2-bf3e-c6c99fb4a29e" containerID="128cc4e340b3f0a09ae965072f66c97a64b1bca0f1657670db7c53f1454d9b75" exitCode=0
Nov 01 03:39:19.241538 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:19.241397    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66" event={"ID":"68863127-fc50-48b2-bf3e-c6c99fb4a29e","Type":"ContainerDied","Data":"128cc4e340b3f0a09ae965072f66c97a64b1bca0f1657670db7c53f1454d9b75"}
Nov 01 03:39:19.241886 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:19.241870    4930 scope.go:122] "RemoveContainer" containerID="128cc4e340b3f0a09ae965072f66c97a64b1bca0f1657670db7c53f1454d9b75"
Nov 01 03:39:19.243590 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:19.243537    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:39:19.243667 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:19.243619    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:39:19.243667 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:19.243650    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:39:20.251837 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:20.251804    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66" event={"ID":"68863127-fc50-48b2-bf3e-c6c99fb4a29e","Type":"ContainerStarted","Data":"ca9e4eebeb18dd2da1bb6784c3f5bb73a66ac98460eb42744c83599dbaa27a1d"}
Nov 01 03:39:20.311638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:20.311601    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"
Nov 01 03:39:20.312829 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:20.312799    4930 patch_prober.go:29] interesting pod/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Nov 01 03:39:20.312967 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:20.312872    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66" podUID="68863127-fc50-48b2-bf3e-c6c99fb4a29e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Nov 01 03:39:21.311872 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:21.311841    4930 patch_prober.go:29] interesting pod/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Nov 01 03:39:21.312275 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:21.311889    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66" podUID="68863127-fc50-48b2-bf3e-c6c99fb4a29e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Nov 01 03:39:22.312323 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:22.312288    4930 patch_prober.go:29] interesting pod/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Nov 01 03:39:22.312797 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:22.312346    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66" podUID="68863127-fc50-48b2-bf3e-c6c99fb4a29e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Nov 01 03:39:22.780541 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:22.780501    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:39:23.312036 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:23.311995    4930 patch_prober.go:29] interesting pod/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Nov 01 03:39:23.312249 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:23.312062    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66" podUID="68863127-fc50-48b2-bf3e-c6c99fb4a29e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Nov 01 03:39:24.311857 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:24.311821    4930 patch_prober.go:29] interesting pod/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Nov 01 03:39:24.312285 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:24.311876    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66" podUID="68863127-fc50-48b2-bf3e-c6c99fb4a29e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Nov 01 03:39:24.690580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:24.689973    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"
Nov 01 03:39:25.052828 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:25.052746    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afab651a-bvr2q"]
Nov 01 03:39:25.102275 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:25.102238    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afab651a-bvr2q"]
Nov 01 03:39:25.311913 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:25.311823    4930 patch_prober.go:29] interesting pod/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Nov 01 03:39:25.311913 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:25.311886    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66" podUID="68863127-fc50-48b2-bf3e-c6c99fb4a29e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Nov 01 03:39:25.622665 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:25.622632    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab67eed1-b327-4933-ac53-fd2b8c6e7a66" path="/var/lib/kubelet/pods/ab67eed1-b327-4933-ac53-fd2b8c6e7a66/volumes"
Nov 01 03:39:26.317249 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:26.317221    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"
Nov 01 03:39:26.318333 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:26.318311    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"
Nov 01 03:39:27.515973 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:27.515924    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:39:30.710925 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:30.710889    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:39:31.190550 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:31.190507    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:39:31.739739 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:31.739699    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"]
Nov 01 03:39:31.741415 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:31.741382    4930 cpu_manager.go:414] "containerMap: removing container" podUID="92ed3fc3-dd13-42e8-a20e-84e0ec2de11a" containerName="revoke-keys"
Nov 01 03:39:31.741415 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:31.741413    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92ed3fc3-dd13-42e8-a20e-84e0ec2de11a" containerName="revoke-keys"
Nov 01 03:39:31.741715 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:31.741665    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="92ed3fc3-dd13-42e8-a20e-84e0ec2de11a" containerName="revoke-keys"
Nov 01 03:39:31.749925 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:31.749898    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"
Nov 01 03:39:31.844579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:31.837846    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-211380-openshift-default-dockercfg-xsqqm\""
Nov 01 03:39:31.851303 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:31.851270    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:39:31.884346 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:31.884302    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b72d8427-c8a5-4d67-be83-57a2f54dddb4-istio-podinfo\") pod \"e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb\" (UID: \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\") " pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"
Nov 01 03:39:31.951818 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:31.951765    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b72d8427-c8a5-4d67-be83-57a2f54dddb4-istio-envoy\") pod \"e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb\" (UID: \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\") " pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"
Nov 01 03:39:31.990868 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:31.990766    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b72d8427-c8a5-4d67-be83-57a2f54dddb4-istiod-ca-cert\") pod \"e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb\" (UID: \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\") " pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"
Nov 01 03:39:31.991215 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:31.991190    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b72d8427-c8a5-4d67-be83-57a2f54dddb4-istio-data\") pod \"e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb\" (UID: \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\") " pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"
Nov 01 03:39:31.991445 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:31.991428    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b72d8427-c8a5-4d67-be83-57a2f54dddb4-istio-token\") pod \"e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb\" (UID: \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\") " pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"
Nov 01 03:39:31.994254 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:31.994225    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b72d8427-c8a5-4d67-be83-57a2f54dddb4-workload-certs\") pod \"e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb\" (UID: \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\") " pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"
Nov 01 03:39:31.994531 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:31.994510    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b72d8427-c8a5-4d67-be83-57a2f54dddb4-workload-socket\") pod \"e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb\" (UID: \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\") " pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"
Nov 01 03:39:31.994997 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:31.994969    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm68f\" (UniqueName: \"kubernetes.io/projected/b72d8427-c8a5-4d67-be83-57a2f54dddb4-kube-api-access-fm68f\") pod \"e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb\" (UID: \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\") " pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"
Nov 01 03:39:31.995182 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:31.995167    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b72d8427-c8a5-4d67-be83-57a2f54dddb4-credential-socket\") pod \"e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb\" (UID: \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\") " pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"
Nov 01 03:39:32.006939 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:32.006903    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"]
Nov 01 03:39:32.098957 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:32.098921    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b72d8427-c8a5-4d67-be83-57a2f54dddb4-istio-data\") pod \"e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb\" (UID: \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\") " pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"
Nov 01 03:39:32.099452 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:32.099395    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b72d8427-c8a5-4d67-be83-57a2f54dddb4-istio-token\") pod \"e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb\" (UID: \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\") " pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"
Nov 01 03:39:32.099669 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:32.099650    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b72d8427-c8a5-4d67-be83-57a2f54dddb4-workload-certs\") pod \"e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb\" (UID: \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\") " pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"
Nov 01 03:39:32.099830 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:32.099811    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b72d8427-c8a5-4d67-be83-57a2f54dddb4-workload-socket\") pod \"e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb\" (UID: \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\") " pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"
Nov 01 03:39:32.100038 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:32.100009    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm68f\" (UniqueName: \"kubernetes.io/projected/b72d8427-c8a5-4d67-be83-57a2f54dddb4-kube-api-access-fm68f\") pod \"e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb\" (UID: \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\") " pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"
Nov 01 03:39:32.100192 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:32.100176    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b72d8427-c8a5-4d67-be83-57a2f54dddb4-credential-socket\") pod \"e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb\" (UID: \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\") " pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"
Nov 01 03:39:32.100366 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:32.100351    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b72d8427-c8a5-4d67-be83-57a2f54dddb4-istio-podinfo\") pod \"e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb\" (UID: \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\") " pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"
Nov 01 03:39:32.100547 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:32.100532    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b72d8427-c8a5-4d67-be83-57a2f54dddb4-istio-envoy\") pod \"e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb\" (UID: \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\") " pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"
Nov 01 03:39:32.100718 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:32.100702    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b72d8427-c8a5-4d67-be83-57a2f54dddb4-istiod-ca-cert\") pod \"e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb\" (UID: \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\") " pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"
Nov 01 03:39:32.101694 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:32.101670    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b72d8427-c8a5-4d67-be83-57a2f54dddb4-istiod-ca-cert\") pod \"e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb\" (UID: \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\") " pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"
Nov 01 03:39:32.102254 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:32.102231    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b72d8427-c8a5-4d67-be83-57a2f54dddb4-istio-data\") pod \"e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb\" (UID: \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\") " pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"
Nov 01 03:39:32.102976 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:32.102954    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b72d8427-c8a5-4d67-be83-57a2f54dddb4-workload-certs\") pod \"e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb\" (UID: \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\") " pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"
Nov 01 03:39:32.103362 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:32.103339    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b72d8427-c8a5-4d67-be83-57a2f54dddb4-workload-socket\") pod \"e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb\" (UID: \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\") " pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"
Nov 01 03:39:32.103992 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:32.103970    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b72d8427-c8a5-4d67-be83-57a2f54dddb4-credential-socket\") pod \"e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb\" (UID: \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\") " pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"
Nov 01 03:39:32.120960 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:32.112771    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b72d8427-c8a5-4d67-be83-57a2f54dddb4-istio-envoy\") pod \"e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb\" (UID: \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\") " pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"
Nov 01 03:39:32.124146 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:32.124117    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b72d8427-c8a5-4d67-be83-57a2f54dddb4-istio-podinfo\") pod \"e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb\" (UID: \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\") " pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"
Nov 01 03:39:32.163268 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:32.163234    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b72d8427-c8a5-4d67-be83-57a2f54dddb4-istio-token\") pod \"e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb\" (UID: \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\") " pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"
Nov 01 03:39:32.163725 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:32.163695    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm68f\" (UniqueName: \"kubernetes.io/projected/b72d8427-c8a5-4d67-be83-57a2f54dddb4-kube-api-access-fm68f\") pod \"e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb\" (UID: \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\") " pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"
Nov 01 03:39:32.257614 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:32.257472    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:39:32.408605 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:32.408567    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"
Nov 01 03:39:32.849389 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:39:32.849361    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb72d8427_c8a5_4d67_be83_57a2f54dddb4.slice/crio-6801c32b51e4f8dbf1a085acfa6f7b4b166134cdcd1e0cb53e1ebb37ab454064 WatchSource:0}: Error finding container 6801c32b51e4f8dbf1a085acfa6f7b4b166134cdcd1e0cb53e1ebb37ab454064: Status 404 returned error can't find the container with id 6801c32b51e4f8dbf1a085acfa6f7b4b166134cdcd1e0cb53e1ebb37ab454064
Nov 01 03:39:32.849851 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:32.849828    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"]
Nov 01 03:39:32.852346 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:32.852313    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:39:32.852423 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:32.852382    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:39:32.852423 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:32.852412    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:39:32.894459 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:32.894430    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:39:33.401110 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:33.401061    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb" event={"ID":"b72d8427-c8a5-4d67-be83-57a2f54dddb4","Type":"ContainerStarted","Data":"6b1f11a10a7deb1b747125dcba38e6a0e7b6b526073041c52e6abadd79caeeac"}
Nov 01 03:39:33.401308 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:33.401142    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb" event={"ID":"b72d8427-c8a5-4d67-be83-57a2f54dddb4","Type":"ContainerStarted","Data":"6801c32b51e4f8dbf1a085acfa6f7b4b166134cdcd1e0cb53e1ebb37ab454064"}
Nov 01 03:39:33.409402 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:33.409373    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"
Nov 01 03:39:33.411065 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:33.411037    4930 patch_prober.go:29] interesting pod/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 01 03:39:33.411169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:33.411097    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb" podUID="b72d8427-c8a5-4d67-be83-57a2f54dddb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 01 03:39:33.524903 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:33.524847    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb" podStartSLOduration=2.524833487 podStartE2EDuration="2.524833487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:39:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:39:33.520771694 +0000 UTC m=+1978.043778281" watchObservedRunningTime="2026-11-01 03:39:33.524833487 +0000 UTC m=+1978.047840075"
Nov 01 03:39:33.837882 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:33.837853    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:39:33.981928 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:33.981876    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:39:34.413963 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:34.413932    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"
Nov 01 03:39:35.418168 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:35.418135    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"
Nov 01 03:39:35.419203 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:35.419179    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"
Nov 01 03:39:36.354927 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:36.354875    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:39:36.555467 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:36.555423    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:39:37.944717 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:37.944683    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:39:38.093144 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:38.093105    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:39:38.250099 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:38.250003    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:39:38.395497 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:38.395456    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:39:39.093294 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:39.093263    4930 scope.go:122] "RemoveContainer" containerID="2681dcc05077d544151263f69c09dca31383983aed088dbb68e80711be751c7d"
Nov 01 03:39:39.103911 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:39.103887    4930 scope.go:122] "RemoveContainer" containerID="5d73e91b053db311d1b537e81d2c3179bd6d0d6cb72e0ca7e66ade5cadaac9b6"
Nov 01 03:39:39.113708 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:39.113687    4930 scope.go:122] "RemoveContainer" containerID="25b6b1b773359bf82ee3bd9966bd600a22f9cf59454b59f548ec34f1c0afcb61"
Nov 01 03:39:41.297546 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:41.297507    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e8c77d-8cvvc"]
Nov 01 03:39:41.303186 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:41.303162    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e8c77d-8cvvc"
Nov 01 03:39:41.358152 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:41.358127    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e8c77d-8cvvc"]
Nov 01 03:39:41.401738 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:41.401711    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:39:41.401897 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:41.401756    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:39:41.401897 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:41.401811    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:39:41.401897 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:41.401853    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:39:41.452872 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:41.452838    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7ae0e93-00ce-4e61-907d-691e02b4220e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-e8c77d-8cvvc\" (UID: \"a7ae0e93-00ce-4e61-907d-691e02b4220e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e8c77d-8cvvc"
Nov 01 03:39:41.453037 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:41.453018    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7ae0e93-00ce-4e61-907d-691e02b4220e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-e8c77d-8cvvc\" (UID: \"a7ae0e93-00ce-4e61-907d-691e02b4220e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e8c77d-8cvvc"
Nov 01 03:39:41.453157 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:41.453140    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc95n\" (UniqueName: \"kubernetes.io/projected/a7ae0e93-00ce-4e61-907d-691e02b4220e-kube-api-access-mc95n\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-e8c77d-8cvvc\" (UID: \"a7ae0e93-00ce-4e61-907d-691e02b4220e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e8c77d-8cvvc"
Nov 01 03:39:41.555164 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:41.555095    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc95n\" (UniqueName: \"kubernetes.io/projected/a7ae0e93-00ce-4e61-907d-691e02b4220e-kube-api-access-mc95n\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-e8c77d-8cvvc\" (UID: \"a7ae0e93-00ce-4e61-907d-691e02b4220e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e8c77d-8cvvc"
Nov 01 03:39:41.555297 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:41.555279    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7ae0e93-00ce-4e61-907d-691e02b4220e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-e8c77d-8cvvc\" (UID: \"a7ae0e93-00ce-4e61-907d-691e02b4220e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e8c77d-8cvvc"
Nov 01 03:39:41.555424 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:41.555408    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7ae0e93-00ce-4e61-907d-691e02b4220e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-e8c77d-8cvvc\" (UID: \"a7ae0e93-00ce-4e61-907d-691e02b4220e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e8c77d-8cvvc"
Nov 01 03:39:41.556004 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:41.555980    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7ae0e93-00ce-4e61-907d-691e02b4220e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-e8c77d-8cvvc\" (UID: \"a7ae0e93-00ce-4e61-907d-691e02b4220e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e8c77d-8cvvc"
Nov 01 03:39:41.557998 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:41.557973    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7ae0e93-00ce-4e61-907d-691e02b4220e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-e8c77d-8cvvc\" (UID: \"a7ae0e93-00ce-4e61-907d-691e02b4220e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e8c77d-8cvvc"
Nov 01 03:39:41.615826 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:41.615796    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc95n\" (UniqueName: \"kubernetes.io/projected/a7ae0e93-00ce-4e61-907d-691e02b4220e-kube-api-access-mc95n\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-e8c77d-8cvvc\" (UID: \"a7ae0e93-00ce-4e61-907d-691e02b4220e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e8c77d-8cvvc"
Nov 01 03:39:41.617236 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:41.617217    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e8c77d-8cvvc"
Nov 01 03:39:42.009915 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:42.009884    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e8c77d-8cvvc"]
Nov 01 03:39:42.010521 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:39:42.010488    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7ae0e93_00ce_4e61_907d_691e02b4220e.slice/crio-a1f82cfb044c7aabbcb4c2b140da402f009731ce8266f616ded4ffd4d19a8f6c WatchSource:0}: Error finding container a1f82cfb044c7aabbcb4c2b140da402f009731ce8266f616ded4ffd4d19a8f6c: Status 404 returned error can't find the container with id a1f82cfb044c7aabbcb4c2b140da402f009731ce8266f616ded4ffd4d19a8f6c
Nov 01 03:39:42.504712 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:42.504670    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e8c77d-8cvvc" event={"ID":"a7ae0e93-00ce-4e61-907d-691e02b4220e","Type":"ContainerStarted","Data":"e1590c7dd6bf82b6c05759094e8165f92986e62abda9d6590863f6f311be92ad"}
Nov 01 03:39:42.505184 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:42.504742    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e8c77d-8cvvc" event={"ID":"a7ae0e93-00ce-4e61-907d-691e02b4220e","Type":"ContainerStarted","Data":"a1f82cfb044c7aabbcb4c2b140da402f009731ce8266f616ded4ffd4d19a8f6c"}
Nov 01 03:39:42.622399 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:42.622348    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e8c77d-8cvvc" podStartSLOduration=1.622335726 podStartE2EDuration="1.622335726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:39:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:39:42.619013519 +0000 UTC m=+1987.142020106" watchObservedRunningTime="2026-11-01 03:39:42.622335726 +0000 UTC m=+1987.145342314"
Nov 01 03:39:43.238850 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:43.238817    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:39:43.388148 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:43.388113    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:39:43.515435 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:43.515403    4930 generic.go:356] "Generic (PLEG): container finished" podID="a7ae0e93-00ce-4e61-907d-691e02b4220e" containerID="e1590c7dd6bf82b6c05759094e8165f92986e62abda9d6590863f6f311be92ad" exitCode=0
Nov 01 03:39:43.515972 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:43.515479    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e8c77d-8cvvc" event={"ID":"a7ae0e93-00ce-4e61-907d-691e02b4220e","Type":"ContainerDied","Data":"e1590c7dd6bf82b6c05759094e8165f92986e62abda9d6590863f6f311be92ad"}
Nov 01 03:39:43.762617 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:43.762500    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:39:44.692503 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:44.692479    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e8c77d-8cvvc"
Nov 01 03:39:44.822446 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:44.822303    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7ae0e93-00ce-4e61-907d-691e02b4220e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7ae0e93-00ce-4e61-907d-691e02b4220e-maas-api-openshift-ca\") pod \"a7ae0e93-00ce-4e61-907d-691e02b4220e\" (UID: \"a7ae0e93-00ce-4e61-907d-691e02b4220e\") "
Nov 01 03:39:44.822753 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:44.822715    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7ae0e93-00ce-4e61-907d-691e02b4220e-kube-api-access-mc95n\" (UniqueName: \"kubernetes.io/projected/a7ae0e93-00ce-4e61-907d-691e02b4220e-kube-api-access-mc95n\") pod \"a7ae0e93-00ce-4e61-907d-691e02b4220e\" (UID: \"a7ae0e93-00ce-4e61-907d-691e02b4220e\") "
Nov 01 03:39:44.822753 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:44.822736    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7ae0e93-00ce-4e61-907d-691e02b4220e-maas-api-openshift-ca" pod "a7ae0e93-00ce-4e61-907d-691e02b4220e" (UID: "a7ae0e93-00ce-4e61-907d-691e02b4220e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:39:44.822964 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:44.822777    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7ae0e93-00ce-4e61-907d-691e02b4220e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7ae0e93-00ce-4e61-907d-691e02b4220e-maas-api-xks-ca\") pod \"a7ae0e93-00ce-4e61-907d-691e02b4220e\" (UID: \"a7ae0e93-00ce-4e61-907d-691e02b4220e\") "
Nov 01 03:39:44.823593 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:44.823537    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7ae0e93-00ce-4e61-907d-691e02b4220e-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:39:44.825823 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:44.825797    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7ae0e93-00ce-4e61-907d-691e02b4220e-maas-api-xks-ca" pod "a7ae0e93-00ce-4e61-907d-691e02b4220e" (UID: "a7ae0e93-00ce-4e61-907d-691e02b4220e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:39:44.831472 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:44.831445    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7ae0e93-00ce-4e61-907d-691e02b4220e-kube-api-access-mc95n" pod "a7ae0e93-00ce-4e61-907d-691e02b4220e" (UID: "a7ae0e93-00ce-4e61-907d-691e02b4220e"). InnerVolumeSpecName "kube-api-access-mc95n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:39:44.925398 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:44.925368    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc95n\" (UniqueName: \"kubernetes.io/projected/a7ae0e93-00ce-4e61-907d-691e02b4220e-kube-api-access-mc95n\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:39:44.925398 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:44.925397    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7ae0e93-00ce-4e61-907d-691e02b4220e-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:39:45.532028 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:45.531989    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e8c77d-8cvvc"
Nov 01 03:39:45.532227 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:45.532018    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e8c77d-8cvvc" event={"ID":"a7ae0e93-00ce-4e61-907d-691e02b4220e","Type":"ContainerDied","Data":"a1f82cfb044c7aabbcb4c2b140da402f009731ce8266f616ded4ffd4d19a8f6c"}
Nov 01 03:39:45.532227 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:45.532077    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1f82cfb044c7aabbcb4c2b140da402f009731ce8266f616ded4ffd4d19a8f6c"
Nov 01 03:39:45.855076 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:45.855036    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:39:46.402707 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:46.402670    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"]
Nov 01 03:39:46.403474 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:46.403456    4930 cpu_manager.go:414] "containerMap: removing container" podUID="a7ae0e93-00ce-4e61-907d-691e02b4220e" containerName="revoke-keys"
Nov 01 03:39:46.403474 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:46.403472    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7ae0e93-00ce-4e61-907d-691e02b4220e" containerName="revoke-keys"
Nov 01 03:39:46.403786 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:46.403765    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7ae0e93-00ce-4e61-907d-691e02b4220e" containerName="revoke-keys"
Nov 01 03:39:46.406638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:46.406616    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"
Nov 01 03:39:46.465167 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:46.465137    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-22e11b-openshift-default-dockercfg-c2ctl\""
Nov 01 03:39:46.508580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:46.491625    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:39:46.663631 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:46.582159    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-istiod-ca-cert\") pod \"e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x\" (UID: \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\") " pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"
Nov 01 03:39:46.671653 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:46.664037    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6d6p\" (UniqueName: \"kubernetes.io/projected/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-kube-api-access-n6d6p\") pod \"e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x\" (UID: \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\") " pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"
Nov 01 03:39:46.798790 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:46.798752    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-workload-certs\") pod \"e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x\" (UID: \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\") " pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"
Nov 01 03:39:46.799278 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:46.799248    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-credential-socket\") pod \"e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x\" (UID: \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\") " pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"
Nov 01 03:39:46.799622 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:46.799599    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-istio-data\") pod \"e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x\" (UID: \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\") " pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"
Nov 01 03:39:46.799838 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:46.799815    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-istio-token\") pod \"e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x\" (UID: \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\") " pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"
Nov 01 03:39:46.800128 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:46.800106    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-workload-socket\") pod \"e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x\" (UID: \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\") " pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"
Nov 01 03:39:46.800413 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:46.800392    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-istio-podinfo\") pod \"e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x\" (UID: \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\") " pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"
Nov 01 03:39:46.800644 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:46.800626    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-istio-envoy\") pod \"e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x\" (UID: \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\") " pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"
Nov 01 03:39:46.800927 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:46.727190    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"]
Nov 01 03:39:46.902694 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:46.902662    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-istio-envoy\") pod \"e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x\" (UID: \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\") " pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"
Nov 01 03:39:46.903168 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:46.902722    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-istiod-ca-cert\") pod \"e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x\" (UID: \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\") " pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"
Nov 01 03:39:46.903168 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:46.902749    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6d6p\" (UniqueName: \"kubernetes.io/projected/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-kube-api-access-n6d6p\") pod \"e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x\" (UID: \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\") " pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"
Nov 01 03:39:46.903168 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:46.902816    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-workload-certs\") pod \"e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x\" (UID: \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\") " pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"
Nov 01 03:39:46.903168 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:46.902856    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-credential-socket\") pod \"e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x\" (UID: \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\") " pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"
Nov 01 03:39:46.903168 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:46.902927    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-istio-data\") pod \"e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x\" (UID: \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\") " pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"
Nov 01 03:39:46.903168 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:46.902963    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-istio-token\") pod \"e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x\" (UID: \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\") " pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"
Nov 01 03:39:46.903168 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:46.903065    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-workload-socket\") pod \"e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x\" (UID: \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\") " pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"
Nov 01 03:39:46.903168 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:46.903104    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-istio-podinfo\") pod \"e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x\" (UID: \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\") " pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"
Nov 01 03:39:46.903655 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:46.903248    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-workload-certs\") pod \"e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x\" (UID: \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\") " pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"
Nov 01 03:39:46.903655 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:46.903377    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-istio-data\") pod \"e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x\" (UID: \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\") " pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"
Nov 01 03:39:46.903655 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:46.903458    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-workload-socket\") pod \"e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x\" (UID: \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\") " pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"
Nov 01 03:39:46.903655 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:46.903584    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-credential-socket\") pod \"e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x\" (UID: \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\") " pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"
Nov 01 03:39:46.903655 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:46.903610    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-istiod-ca-cert\") pod \"e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x\" (UID: \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\") " pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"
Nov 01 03:39:46.904997 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:46.904971    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-istio-envoy\") pod \"e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x\" (UID: \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\") " pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"
Nov 01 03:39:46.905614 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:46.905594    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-istio-podinfo\") pod \"e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x\" (UID: \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\") " pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"
Nov 01 03:39:46.966665 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:46.966591    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-istio-token\") pod \"e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x\" (UID: \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\") " pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"
Nov 01 03:39:46.966886 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:46.966866    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6d6p\" (UniqueName: \"kubernetes.io/projected/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-kube-api-access-n6d6p\") pod \"e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x\" (UID: \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\") " pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"
Nov 01 03:39:47.086244 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:47.086204    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:39:47.090182 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:47.090157    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"
Nov 01 03:39:47.263287 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:47.263209    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:39:47.533847 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:39:47.533814    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d9844bf_a8fe_4904_87fa_fb427c29dbd6.slice/crio-a26d2c5d7e74cd33411111542a6472ad81ff758fbe33dca5ff69130d6114d711 WatchSource:0}: Error finding container a26d2c5d7e74cd33411111542a6472ad81ff758fbe33dca5ff69130d6114d711: Status 404 returned error can't find the container with id a26d2c5d7e74cd33411111542a6472ad81ff758fbe33dca5ff69130d6114d711
Nov 01 03:39:47.534084 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:47.534062    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"]
Nov 01 03:39:47.536460 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:47.536426    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:39:47.536579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:47.536488    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:39:47.536579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:47.536516    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:39:47.573609 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:47.573581    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0cb10b-sz879"]
Nov 01 03:39:47.579436 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:47.579409    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0cb10b-sz879"
Nov 01 03:39:47.614677 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:47.614623    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljg2l\" (UniqueName: \"kubernetes.io/projected/dbb22944-8ec6-44e6-b09b-d0d9478afb7a-kube-api-access-ljg2l\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0cb10b-sz879\" (UID: \"dbb22944-8ec6-44e6-b09b-d0d9478afb7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0cb10b-sz879"
Nov 01 03:39:47.614777 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:47.614760    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbb22944-8ec6-44e6-b09b-d0d9478afb7a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0cb10b-sz879\" (UID: \"dbb22944-8ec6-44e6-b09b-d0d9478afb7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0cb10b-sz879"
Nov 01 03:39:47.614906 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:47.614887    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbb22944-8ec6-44e6-b09b-d0d9478afb7a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0cb10b-sz879\" (UID: \"dbb22944-8ec6-44e6-b09b-d0d9478afb7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0cb10b-sz879"
Nov 01 03:39:47.632370 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:47.632328    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:39:47.633075 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:47.633051    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:39:47.633201 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:47.633183    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:39:47.633267 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:47.633247    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:39:47.637328 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:47.637289    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0cb10b-sz879"]
Nov 01 03:39:47.690613 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:47.686942    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x" event={"ID":"7d9844bf-a8fe-4904-87fa-fb427c29dbd6","Type":"ContainerStarted","Data":"cbb49232e7db24c7465743c1aecb38f7063b1da1d6f87afac56dbc6c2f4dce11"}
Nov 01 03:39:47.690613 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:47.687000    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x" event={"ID":"7d9844bf-a8fe-4904-87fa-fb427c29dbd6","Type":"ContainerStarted","Data":"a26d2c5d7e74cd33411111542a6472ad81ff758fbe33dca5ff69130d6114d711"}
Nov 01 03:39:47.716550 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:47.716506    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljg2l\" (UniqueName: \"kubernetes.io/projected/dbb22944-8ec6-44e6-b09b-d0d9478afb7a-kube-api-access-ljg2l\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0cb10b-sz879\" (UID: \"dbb22944-8ec6-44e6-b09b-d0d9478afb7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0cb10b-sz879"
Nov 01 03:39:47.717132 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:47.717074    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbb22944-8ec6-44e6-b09b-d0d9478afb7a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0cb10b-sz879\" (UID: \"dbb22944-8ec6-44e6-b09b-d0d9478afb7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0cb10b-sz879"
Nov 01 03:39:47.717291 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:47.717261    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbb22944-8ec6-44e6-b09b-d0d9478afb7a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0cb10b-sz879\" (UID: \"dbb22944-8ec6-44e6-b09b-d0d9478afb7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0cb10b-sz879"
Nov 01 03:39:47.717950 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:47.717919    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbb22944-8ec6-44e6-b09b-d0d9478afb7a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0cb10b-sz879\" (UID: \"dbb22944-8ec6-44e6-b09b-d0d9478afb7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0cb10b-sz879"
Nov 01 03:39:47.720208 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:47.720186    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbb22944-8ec6-44e6-b09b-d0d9478afb7a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0cb10b-sz879\" (UID: \"dbb22944-8ec6-44e6-b09b-d0d9478afb7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0cb10b-sz879"
Nov 01 03:39:47.779951 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:47.779923    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljg2l\" (UniqueName: \"kubernetes.io/projected/dbb22944-8ec6-44e6-b09b-d0d9478afb7a-kube-api-access-ljg2l\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-0cb10b-sz879\" (UID: \"dbb22944-8ec6-44e6-b09b-d0d9478afb7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0cb10b-sz879"
Nov 01 03:39:47.818313 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:47.818214    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x" podStartSLOduration=1.818199183 podStartE2EDuration="1.818199183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:39:47.814179493 +0000 UTC m=+1992.337186081" watchObservedRunningTime="2026-11-01 03:39:47.818199183 +0000 UTC m=+1992.341205772"
Nov 01 03:39:47.908318 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:47.908286    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0cb10b-sz879"
Nov 01 03:39:48.090397 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:48.090372    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"
Nov 01 03:39:48.091755 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:48.091720    4930 patch_prober.go:29] interesting pod/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 01 03:39:48.091883 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:48.091775    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x" podUID="7d9844bf-a8fe-4904-87fa-fb427c29dbd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 01 03:39:48.114280 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:48.114246    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj"]
Nov 01 03:39:48.114504 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:48.114476    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj" podUID="e9a70ec2-21fd-40ae-8d18-025a67248fce" containerName="istio-proxy" containerID="cri-o://477b9b9d8d539bd2cf241b2ffd19624dc8615362041762cf71cdf74597b51bf9" gracePeriod=30
Nov 01 03:39:48.367889 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:39:48.367847    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbb22944_8ec6_44e6_b09b_d0d9478afb7a.slice/crio-7b58bca020150ba193117eb9838c4320c85bf46e7b2cc4932911657ccac9c407 WatchSource:0}: Error finding container 7b58bca020150ba193117eb9838c4320c85bf46e7b2cc4932911657ccac9c407: Status 404 returned error can't find the container with id 7b58bca020150ba193117eb9838c4320c85bf46e7b2cc4932911657ccac9c407
Nov 01 03:39:48.368056 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:48.367981    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0cb10b-sz879"]
Nov 01 03:39:48.706858 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:48.706822    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0cb10b-sz879" event={"ID":"dbb22944-8ec6-44e6-b09b-d0d9478afb7a","Type":"ContainerStarted","Data":"7b58bca020150ba193117eb9838c4320c85bf46e7b2cc4932911657ccac9c407"}
Nov 01 03:39:48.790232 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:48.790198    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp"]
Nov 01 03:39:48.790464 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:48.790435    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp" podUID="3c321a5a-ae6b-49aa-bbc0-e02778d1b539" containerName="istio-proxy" containerID="cri-o://2f2e341e5a8e6c32912b60d0bc335b4b5c6f2620c873665171a4981e96b77973" gracePeriod=30
Nov 01 03:39:48.968480 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:48.968391    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:39:49.096407 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:49.096377    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"
Nov 01 03:39:49.127411 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:39:49.127374    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:39:49.716911 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:49.716879    4930 generic.go:356] "Generic (PLEG): container finished" podID="dbb22944-8ec6-44e6-b09b-d0d9478afb7a" containerID="5de91ad8e4e388fbe1e0613ed53765ff30a26f655ceb2390666f0d4c6c8de3da" exitCode=0
Nov 01 03:39:49.717098 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:49.716944    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0cb10b-sz879" event={"ID":"dbb22944-8ec6-44e6-b09b-d0d9478afb7a","Type":"ContainerDied","Data":"5de91ad8e4e388fbe1e0613ed53765ff30a26f655ceb2390666f0d4c6c8de3da"}
Nov 01 03:39:49.717271 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:49.717239    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"
Nov 01 03:39:49.718308 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:49.718287    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"
Nov 01 03:39:50.907818 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:50.907765    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:39:50.938465 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:50.938439    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0cb10b-sz879"
Nov 01 03:39:51.133575 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:51.133506    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbb22944-8ec6-44e6-b09b-d0d9478afb7a-kube-api-access-ljg2l\" (UniqueName: \"kubernetes.io/projected/dbb22944-8ec6-44e6-b09b-d0d9478afb7a-kube-api-access-ljg2l\") pod \"dbb22944-8ec6-44e6-b09b-d0d9478afb7a\" (UID: \"dbb22944-8ec6-44e6-b09b-d0d9478afb7a\") "
Nov 01 03:39:51.133782 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:51.133704    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbb22944-8ec6-44e6-b09b-d0d9478afb7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbb22944-8ec6-44e6-b09b-d0d9478afb7a-maas-api-openshift-ca\") pod \"dbb22944-8ec6-44e6-b09b-d0d9478afb7a\" (UID: \"dbb22944-8ec6-44e6-b09b-d0d9478afb7a\") "
Nov 01 03:39:51.145543 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:51.133877    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbb22944-8ec6-44e6-b09b-d0d9478afb7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbb22944-8ec6-44e6-b09b-d0d9478afb7a-maas-api-xks-ca\") pod \"dbb22944-8ec6-44e6-b09b-d0d9478afb7a\" (UID: \"dbb22944-8ec6-44e6-b09b-d0d9478afb7a\") "
Nov 01 03:39:51.153579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:51.150522    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbb22944-8ec6-44e6-b09b-d0d9478afb7a-maas-api-openshift-ca" pod "dbb22944-8ec6-44e6-b09b-d0d9478afb7a" (UID: "dbb22944-8ec6-44e6-b09b-d0d9478afb7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:39:51.177481 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:51.157429    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:39:51.177481 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:51.176357    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbb22944-8ec6-44e6-b09b-d0d9478afb7a-maas-api-xks-ca" pod "dbb22944-8ec6-44e6-b09b-d0d9478afb7a" (UID: "dbb22944-8ec6-44e6-b09b-d0d9478afb7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:39:51.177481 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:51.176513    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbb22944-8ec6-44e6-b09b-d0d9478afb7a-kube-api-access-ljg2l" pod "dbb22944-8ec6-44e6-b09b-d0d9478afb7a" (UID: "dbb22944-8ec6-44e6-b09b-d0d9478afb7a"). InnerVolumeSpecName "kube-api-access-ljg2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:39:51.267758 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:51.267717    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbb22944-8ec6-44e6-b09b-d0d9478afb7a-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:39:51.267758 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:51.267755    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljg2l\" (UniqueName: \"kubernetes.io/projected/dbb22944-8ec6-44e6-b09b-d0d9478afb7a-kube-api-access-ljg2l\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:39:51.268030 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:51.267771    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbb22944-8ec6-44e6-b09b-d0d9478afb7a-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:39:51.396685 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:51.396132    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:39:51.589255 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:51.587035    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:39:51.740965 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:51.740934    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0cb10b-sz879"
Nov 01 03:39:51.741254 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:51.740967    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0cb10b-sz879" event={"ID":"dbb22944-8ec6-44e6-b09b-d0d9478afb7a","Type":"ContainerDied","Data":"7b58bca020150ba193117eb9838c4320c85bf46e7b2cc4932911657ccac9c407"}
Nov 01 03:39:51.741333 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:51.741296    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b58bca020150ba193117eb9838c4320c85bf46e7b2cc4932911657ccac9c407"
Nov 01 03:39:51.790266 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:39:51.790225    4930 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbb22944_8ec6_44e6_b09b_d0d9478afb7a.slice\": RecentStats: unable to find data in memory cache]"
Nov 01 03:39:51.811255 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:51.811219    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:39:52.212418 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:52.212381    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-e8c77d-75ffdb94b5-7klwh"]
Nov 01 03:39:52.322103 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:52.322066    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e8c77d-68444c5d47-bj8bs"]
Nov 01 03:39:52.422724 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:52.422687    4930 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:39:52.422893 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:52.422759    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj" podUID="e9a70ec2-21fd-40ae-8d18-025a67248fce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:39:52.598683 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:52.598649    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:39:52.809550 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:52.809481    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:39:52.843539 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:52.843501    4930 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:39:52.843773 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:52.843574    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp" podUID="3c321a5a-ae6b-49aa-bbc0-e02778d1b539" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:39:53.419415 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:53.419389    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj"
Nov 01 03:39:53.495214 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:53.495183    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9a70ec2-21fd-40ae-8d18-025a67248fce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9a70ec2-21fd-40ae-8d18-025a67248fce-istio-data\") pod \"e9a70ec2-21fd-40ae-8d18-025a67248fce\" (UID: \"e9a70ec2-21fd-40ae-8d18-025a67248fce\") "
Nov 01 03:39:53.495380 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:53.495239    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9a70ec2-21fd-40ae-8d18-025a67248fce-istio-token\" (UniqueName: \"kubernetes.io/projected/e9a70ec2-21fd-40ae-8d18-025a67248fce-istio-token\") pod \"e9a70ec2-21fd-40ae-8d18-025a67248fce\" (UID: \"e9a70ec2-21fd-40ae-8d18-025a67248fce\") "
Nov 01 03:39:53.495380 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:53.495329    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9a70ec2-21fd-40ae-8d18-025a67248fce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9a70ec2-21fd-40ae-8d18-025a67248fce-istio-envoy\") pod \"e9a70ec2-21fd-40ae-8d18-025a67248fce\" (UID: \"e9a70ec2-21fd-40ae-8d18-025a67248fce\") "
Nov 01 03:39:53.495591 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:53.495382    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9a70ec2-21fd-40ae-8d18-025a67248fce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9a70ec2-21fd-40ae-8d18-025a67248fce-istiod-ca-cert\") pod \"e9a70ec2-21fd-40ae-8d18-025a67248fce\" (UID: \"e9a70ec2-21fd-40ae-8d18-025a67248fce\") "
Nov 01 03:39:53.495591 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:53.495442    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9a70ec2-21fd-40ae-8d18-025a67248fce-istio-data" pod "e9a70ec2-21fd-40ae-8d18-025a67248fce" (UID: "e9a70ec2-21fd-40ae-8d18-025a67248fce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:39:53.495591 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:53.495471    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9a70ec2-21fd-40ae-8d18-025a67248fce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9a70ec2-21fd-40ae-8d18-025a67248fce-istio-podinfo\") pod \"e9a70ec2-21fd-40ae-8d18-025a67248fce\" (UID: \"e9a70ec2-21fd-40ae-8d18-025a67248fce\") "
Nov 01 03:39:53.495852 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:53.495622    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9a70ec2-21fd-40ae-8d18-025a67248fce-kube-api-access-59tv7\" (UniqueName: \"kubernetes.io/projected/e9a70ec2-21fd-40ae-8d18-025a67248fce-kube-api-access-59tv7\") pod \"e9a70ec2-21fd-40ae-8d18-025a67248fce\" (UID: \"e9a70ec2-21fd-40ae-8d18-025a67248fce\") "
Nov 01 03:39:53.495852 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:53.495660    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9a70ec2-21fd-40ae-8d18-025a67248fce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a70ec2-21fd-40ae-8d18-025a67248fce-credential-socket\") pod \"e9a70ec2-21fd-40ae-8d18-025a67248fce\" (UID: \"e9a70ec2-21fd-40ae-8d18-025a67248fce\") "
Nov 01 03:39:53.495852 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:53.495813    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9a70ec2-21fd-40ae-8d18-025a67248fce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a70ec2-21fd-40ae-8d18-025a67248fce-workload-socket\") pod \"e9a70ec2-21fd-40ae-8d18-025a67248fce\" (UID: \"e9a70ec2-21fd-40ae-8d18-025a67248fce\") "
Nov 01 03:39:53.496022 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:53.495911    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9a70ec2-21fd-40ae-8d18-025a67248fce-credential-socket" pod "e9a70ec2-21fd-40ae-8d18-025a67248fce" (UID: "e9a70ec2-21fd-40ae-8d18-025a67248fce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:39:53.496022 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:53.495915    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9a70ec2-21fd-40ae-8d18-025a67248fce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9a70ec2-21fd-40ae-8d18-025a67248fce-workload-certs\") pod \"e9a70ec2-21fd-40ae-8d18-025a67248fce\" (UID: \"e9a70ec2-21fd-40ae-8d18-025a67248fce\") "
Nov 01 03:39:53.496130 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:53.496104    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9a70ec2-21fd-40ae-8d18-025a67248fce-workload-certs" pod "e9a70ec2-21fd-40ae-8d18-025a67248fce" (UID: "e9a70ec2-21fd-40ae-8d18-025a67248fce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:39:53.496224 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:53.496188    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9a70ec2-21fd-40ae-8d18-025a67248fce-workload-socket" pod "e9a70ec2-21fd-40ae-8d18-025a67248fce" (UID: "e9a70ec2-21fd-40ae-8d18-025a67248fce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:39:53.496724 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:53.496697    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9a70ec2-21fd-40ae-8d18-025a67248fce-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:39:53.496825 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:53.496727    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a70ec2-21fd-40ae-8d18-025a67248fce-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:39:53.496825 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:53.496744    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9a70ec2-21fd-40ae-8d18-025a67248fce-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:39:53.496825 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:53.496758    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9a70ec2-21fd-40ae-8d18-025a67248fce-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:39:53.497012 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:53.496923    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9a70ec2-21fd-40ae-8d18-025a67248fce-istiod-ca-cert" pod "e9a70ec2-21fd-40ae-8d18-025a67248fce" (UID: "e9a70ec2-21fd-40ae-8d18-025a67248fce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:39:53.497760 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:53.497734    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9a70ec2-21fd-40ae-8d18-025a67248fce-istio-token" pod "e9a70ec2-21fd-40ae-8d18-025a67248fce" (UID: "e9a70ec2-21fd-40ae-8d18-025a67248fce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:39:53.498048 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:53.498029    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9a70ec2-21fd-40ae-8d18-025a67248fce-istio-podinfo" pod "e9a70ec2-21fd-40ae-8d18-025a67248fce" (UID: "e9a70ec2-21fd-40ae-8d18-025a67248fce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:39:53.498224 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:53.498201    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9a70ec2-21fd-40ae-8d18-025a67248fce-istio-envoy" pod "e9a70ec2-21fd-40ae-8d18-025a67248fce" (UID: "e9a70ec2-21fd-40ae-8d18-025a67248fce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:39:53.498902 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:53.498878    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9a70ec2-21fd-40ae-8d18-025a67248fce-kube-api-access-59tv7" pod "e9a70ec2-21fd-40ae-8d18-025a67248fce" (UID: "e9a70ec2-21fd-40ae-8d18-025a67248fce"). InnerVolumeSpecName "kube-api-access-59tv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:39:53.598894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:53.598845    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9a70ec2-21fd-40ae-8d18-025a67248fce-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:39:53.598894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:53.598896    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9a70ec2-21fd-40ae-8d18-025a67248fce-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:39:53.599149 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:53.598914    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9a70ec2-21fd-40ae-8d18-025a67248fce-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:39:53.599149 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:53.598930    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59tv7\" (UniqueName: \"kubernetes.io/projected/e9a70ec2-21fd-40ae-8d18-025a67248fce-kube-api-access-59tv7\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:39:53.599149 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:53.598946    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9a70ec2-21fd-40ae-8d18-025a67248fce-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:39:53.849746 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:53.839512    4930 generic.go:356] "Generic (PLEG): container finished" podID="e9a70ec2-21fd-40ae-8d18-025a67248fce" containerID="477b9b9d8d539bd2cf241b2ffd19624dc8615362041762cf71cdf74597b51bf9" exitCode=0
Nov 01 03:39:53.849746 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:53.839826    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj" event={"ID":"e9a70ec2-21fd-40ae-8d18-025a67248fce","Type":"ContainerDied","Data":"477b9b9d8d539bd2cf241b2ffd19624dc8615362041762cf71cdf74597b51bf9"}
Nov 01 03:39:53.849746 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:53.839883    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj" event={"ID":"e9a70ec2-21fd-40ae-8d18-025a67248fce","Type":"ContainerDied","Data":"92374760c787bee288de1b1132abb43ba653dd3de0c90d033c5eb17232df8448"}
Nov 01 03:39:53.849746 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:53.839917    4930 scope.go:122] "RemoveContainer" containerID="477b9b9d8d539bd2cf241b2ffd19624dc8615362041762cf71cdf74597b51bf9"
Nov 01 03:39:53.849746 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:53.840102    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj"
Nov 01 03:39:54.079607 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:54.079178    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj"]
Nov 01 03:39:54.081919 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:54.081262    4930 scope.go:122] "RemoveContainer" containerID="477b9b9d8d539bd2cf241b2ffd19624dc8615362041762cf71cdf74597b51bf9"
Nov 01 03:39:54.083028 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:39:54.082907    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"477b9b9d8d539bd2cf241b2ffd19624dc8615362041762cf71cdf74597b51bf9\": container with ID starting with 477b9b9d8d539bd2cf241b2ffd19624dc8615362041762cf71cdf74597b51bf9 not found: ID does not exist" containerID="477b9b9d8d539bd2cf241b2ffd19624dc8615362041762cf71cdf74597b51bf9"
Nov 01 03:39:54.083028 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:54.082973    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"477b9b9d8d539bd2cf241b2ffd19624dc8615362041762cf71cdf74597b51bf9"} err="failed to get container status \"477b9b9d8d539bd2cf241b2ffd19624dc8615362041762cf71cdf74597b51bf9\": rpc error: code = NotFound desc = could not find container \"477b9b9d8d539bd2cf241b2ffd19624dc8615362041762cf71cdf74597b51bf9\": container with ID starting with 477b9b9d8d539bd2cf241b2ffd19624dc8615362041762cf71cdf74597b51bf9 not found: ID does not exist"
Nov 01 03:39:54.147583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:54.147481    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-qlhnj"]
Nov 01 03:39:54.157606 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:54.157521    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb84ee16-5lcr4"]
Nov 01 03:39:54.170734 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:54.170373    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb84ee16-5lcr4"]
Nov 01 03:39:54.317975 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:54.317949    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp"
Nov 01 03:39:54.406366 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:54.406285    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:39:54.452951 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:54.452914    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-istio-podinfo\") pod \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\" (UID: \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\") "
Nov 01 03:39:54.453683 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:54.452966    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-istiod-ca-cert\") pod \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\" (UID: \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\") "
Nov 01 03:39:54.453683 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:54.453024    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-istio-token\" (UniqueName: \"kubernetes.io/projected/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-istio-token\") pod \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\" (UID: \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\") "
Nov 01 03:39:54.453683 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:54.453053    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-workload-certs\") pod \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\" (UID: \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\") "
Nov 01 03:39:54.453683 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:54.453073    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-credential-socket\") pod \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\" (UID: \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\") "
Nov 01 03:39:54.453683 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:54.453153    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-workload-socket\") pod \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\" (UID: \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\") "
Nov 01 03:39:54.453683 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:54.453369    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-istio-data\") pod \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\" (UID: \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\") "
Nov 01 03:39:54.453683 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:54.453507    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-istio-envoy\") pod \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\" (UID: \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\") "
Nov 01 03:39:54.453683 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:54.453539    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-kube-api-access-kvtsj\" (UniqueName: \"kubernetes.io/projected/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-kube-api-access-kvtsj\") pod \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\" (UID: \"3c321a5a-ae6b-49aa-bbc0-e02778d1b539\") "
Nov 01 03:39:54.470087 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:54.469487    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-workload-socket" pod "3c321a5a-ae6b-49aa-bbc0-e02778d1b539" (UID: "3c321a5a-ae6b-49aa-bbc0-e02778d1b539"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:39:54.470087 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:54.469735    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-credential-socket" pod "3c321a5a-ae6b-49aa-bbc0-e02778d1b539" (UID: "3c321a5a-ae6b-49aa-bbc0-e02778d1b539"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:39:54.483957 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:54.483833    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-istiod-ca-cert" pod "3c321a5a-ae6b-49aa-bbc0-e02778d1b539" (UID: "3c321a5a-ae6b-49aa-bbc0-e02778d1b539"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:39:54.483957 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:54.483912    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-istio-data" pod "3c321a5a-ae6b-49aa-bbc0-e02778d1b539" (UID: "3c321a5a-ae6b-49aa-bbc0-e02778d1b539"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:39:54.491820 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:54.491792    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-workload-certs" pod "3c321a5a-ae6b-49aa-bbc0-e02778d1b539" (UID: "3c321a5a-ae6b-49aa-bbc0-e02778d1b539"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:39:54.510779 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:54.510710    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-kube-api-access-kvtsj" pod "3c321a5a-ae6b-49aa-bbc0-e02778d1b539" (UID: "3c321a5a-ae6b-49aa-bbc0-e02778d1b539"). InnerVolumeSpecName "kube-api-access-kvtsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:39:54.511754 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:54.511723    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-istio-envoy" pod "3c321a5a-ae6b-49aa-bbc0-e02778d1b539" (UID: "3c321a5a-ae6b-49aa-bbc0-e02778d1b539"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:39:54.512296 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:54.512270    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-istio-podinfo" pod "3c321a5a-ae6b-49aa-bbc0-e02778d1b539" (UID: "3c321a5a-ae6b-49aa-bbc0-e02778d1b539"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:39:54.513313 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:54.513279    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-istio-token" pod "3c321a5a-ae6b-49aa-bbc0-e02778d1b539" (UID: "3c321a5a-ae6b-49aa-bbc0-e02778d1b539"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:39:54.556797 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:54.556763    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:39:54.556797 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:54.556798    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:39:54.557027 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:54.556813    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvtsj\" (UniqueName: \"kubernetes.io/projected/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-kube-api-access-kvtsj\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:39:54.557027 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:54.556829    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:39:54.557027 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:54.556842    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:39:54.557027 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:54.556855    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:39:54.557027 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:54.556867    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:39:54.557027 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:54.556879    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:39:54.557027 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:54.556891    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c321a5a-ae6b-49aa-bbc0-e02778d1b539-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:39:54.851592 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:54.851542    4930 generic.go:356] "Generic (PLEG): container finished" podID="3c321a5a-ae6b-49aa-bbc0-e02778d1b539" containerID="2f2e341e5a8e6c32912b60d0bc335b4b5c6f2620c873665171a4981e96b77973" exitCode=0
Nov 01 03:39:54.851783 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:54.851610    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp" event={"ID":"3c321a5a-ae6b-49aa-bbc0-e02778d1b539","Type":"ContainerDied","Data":"2f2e341e5a8e6c32912b60d0bc335b4b5c6f2620c873665171a4981e96b77973"}
Nov 01 03:39:54.851783 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:54.851627    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp"
Nov 01 03:39:54.851783 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:54.851673    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp" event={"ID":"3c321a5a-ae6b-49aa-bbc0-e02778d1b539","Type":"ContainerDied","Data":"332eeb49ad462293c02802b5fc0a04aaeb2bd676eac8ddfe63e8af0f4775e3f7"}
Nov 01 03:39:54.851783 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:54.851704    4930 scope.go:122] "RemoveContainer" containerID="2f2e341e5a8e6c32912b60d0bc335b4b5c6f2620c873665171a4981e96b77973"
Nov 01 03:39:54.864535 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:54.864522    4930 scope.go:122] "RemoveContainer" containerID="2f2e341e5a8e6c32912b60d0bc335b4b5c6f2620c873665171a4981e96b77973"
Nov 01 03:39:54.864849 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:39:54.864822    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f2e341e5a8e6c32912b60d0bc335b4b5c6f2620c873665171a4981e96b77973\": container with ID starting with 2f2e341e5a8e6c32912b60d0bc335b4b5c6f2620c873665171a4981e96b77973 not found: ID does not exist" containerID="2f2e341e5a8e6c32912b60d0bc335b4b5c6f2620c873665171a4981e96b77973"
Nov 01 03:39:54.865023 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:54.864859    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f2e341e5a8e6c32912b60d0bc335b4b5c6f2620c873665171a4981e96b77973"} err="failed to get container status \"2f2e341e5a8e6c32912b60d0bc335b4b5c6f2620c873665171a4981e96b77973\": rpc error: code = NotFound desc = could not find container \"2f2e341e5a8e6c32912b60d0bc335b4b5c6f2620c873665171a4981e96b77973\": container with ID starting with 2f2e341e5a8e6c32912b60d0bc335b4b5c6f2620c873665171a4981e96b77973 not found: ID does not exist"
Nov 01 03:39:55.025243 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:55.025207    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp"]
Nov 01 03:39:55.035391 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:55.035354    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-rrzmp"]
Nov 01 03:39:55.625088 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:55.625054    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c321a5a-ae6b-49aa-bbc0-e02778d1b539" path="/var/lib/kubelet/pods/3c321a5a-ae6b-49aa-bbc0-e02778d1b539/volumes"
Nov 01 03:39:55.625595 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:55.625579    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8516c639-ba17-4a14-9d2c-74a4b8325829" path="/var/lib/kubelet/pods/8516c639-ba17-4a14-9d2c-74a4b8325829/volumes"
Nov 01 03:39:55.626076 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:55.626062    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9a70ec2-21fd-40ae-8d18-025a67248fce" path="/var/lib/kubelet/pods/e9a70ec2-21fd-40ae-8d18-025a67248fce/volumes"
Nov 01 03:39:57.213615 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:57.213577    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e8c77d-75ffdb94b5-7klwh" podUID="c4ef178c-04fb-43cf-a0e9-e6a42a087df5" containerName="payload-processing" containerID="cri-o://bfd1c81df946a3cc590cf2546b06cf760eef451e2a6f88f525db23fd432f9a48" gracePeriod=25
Nov 01 03:39:57.323042 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:57.322998    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e8c77d-68444c5d47-bj8bs" podUID="56f52541-2d41-4f11-a47f-489bfce8b284" containerName="payload-pre-processing" containerID="cri-o://08f24f5ab0b16965c447ba252ba9210546d2e5920e047780a2a4e3335ea58a8f" gracePeriod=25
Nov 01 03:39:57.524953 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:57.524580    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e8c77d-75ffdb94b5-7klwh"
Nov 01 03:39:57.643259 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:57.643230    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e8c77d-68444c5d47-bj8bs"
Nov 01 03:39:57.706030 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:57.705995    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4ef178c-04fb-43cf-a0e9-e6a42a087df5-kube-api-access-g5cc9\" (UniqueName: \"kubernetes.io/projected/c4ef178c-04fb-43cf-a0e9-e6a42a087df5-kube-api-access-g5cc9\") pod \"c4ef178c-04fb-43cf-a0e9-e6a42a087df5\" (UID: \"c4ef178c-04fb-43cf-a0e9-e6a42a087df5\") "
Nov 01 03:39:57.706228 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:57.706113    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4ef178c-04fb-43cf-a0e9-e6a42a087df5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4ef178c-04fb-43cf-a0e9-e6a42a087df5-plugins-config-volume\") pod \"c4ef178c-04fb-43cf-a0e9-e6a42a087df5\" (UID: \"c4ef178c-04fb-43cf-a0e9-e6a42a087df5\") "
Nov 01 03:39:57.706794 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:57.706690    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4ef178c-04fb-43cf-a0e9-e6a42a087df5-plugins-config-volume" pod "c4ef178c-04fb-43cf-a0e9-e6a42a087df5" (UID: "c4ef178c-04fb-43cf-a0e9-e6a42a087df5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:39:57.707806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:57.707783    4930 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4ef178c-04fb-43cf-a0e9-e6a42a087df5-plugins-config-volume\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:39:57.709155 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:57.709126    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4ef178c-04fb-43cf-a0e9-e6a42a087df5-kube-api-access-g5cc9" pod "c4ef178c-04fb-43cf-a0e9-e6a42a087df5" (UID: "c4ef178c-04fb-43cf-a0e9-e6a42a087df5"). InnerVolumeSpecName "kube-api-access-g5cc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:39:57.809075 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:57.808987    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56f52541-2d41-4f11-a47f-489bfce8b284-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56f52541-2d41-4f11-a47f-489bfce8b284-plugins-config-volume\") pod \"56f52541-2d41-4f11-a47f-489bfce8b284\" (UID: \"56f52541-2d41-4f11-a47f-489bfce8b284\") "
Nov 01 03:39:57.809250 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:57.809207    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56f52541-2d41-4f11-a47f-489bfce8b284-kube-api-access-97r6f\" (UniqueName: \"kubernetes.io/projected/56f52541-2d41-4f11-a47f-489bfce8b284-kube-api-access-97r6f\") pod \"56f52541-2d41-4f11-a47f-489bfce8b284\" (UID: \"56f52541-2d41-4f11-a47f-489bfce8b284\") "
Nov 01 03:39:57.809619 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:57.809456    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56f52541-2d41-4f11-a47f-489bfce8b284-plugins-config-volume" pod "56f52541-2d41-4f11-a47f-489bfce8b284" (UID: "56f52541-2d41-4f11-a47f-489bfce8b284"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:39:57.810088 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:57.810063    4930 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56f52541-2d41-4f11-a47f-489bfce8b284-plugins-config-volume\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:39:57.810208 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:57.810090    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5cc9\" (UniqueName: \"kubernetes.io/projected/c4ef178c-04fb-43cf-a0e9-e6a42a087df5-kube-api-access-g5cc9\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:39:57.812592 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:57.812543    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56f52541-2d41-4f11-a47f-489bfce8b284-kube-api-access-97r6f" pod "56f52541-2d41-4f11-a47f-489bfce8b284" (UID: "56f52541-2d41-4f11-a47f-489bfce8b284"). InnerVolumeSpecName "kube-api-access-97r6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:39:57.881246 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:57.881184    4930 generic.go:356] "Generic (PLEG): container finished" podID="56f52541-2d41-4f11-a47f-489bfce8b284" containerID="08f24f5ab0b16965c447ba252ba9210546d2e5920e047780a2a4e3335ea58a8f" exitCode=0
Nov 01 03:39:57.881246 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:57.881229    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e8c77d-68444c5d47-bj8bs"
Nov 01 03:39:57.881422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:57.881223    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e8c77d-68444c5d47-bj8bs" event={"ID":"56f52541-2d41-4f11-a47f-489bfce8b284","Type":"ContainerDied","Data":"08f24f5ab0b16965c447ba252ba9210546d2e5920e047780a2a4e3335ea58a8f"}
Nov 01 03:39:57.881422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:57.881404    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e8c77d-68444c5d47-bj8bs" event={"ID":"56f52541-2d41-4f11-a47f-489bfce8b284","Type":"ContainerDied","Data":"a7acc2fc3624ba5bf569cd658a04a102493d541aa1780b224a0df6a6acdd3453"}
Nov 01 03:39:57.881525 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:57.881446    4930 scope.go:122] "RemoveContainer" containerID="08f24f5ab0b16965c447ba252ba9210546d2e5920e047780a2a4e3335ea58a8f"
Nov 01 03:39:57.882999 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:57.882979    4930 generic.go:356] "Generic (PLEG): container finished" podID="c4ef178c-04fb-43cf-a0e9-e6a42a087df5" containerID="bfd1c81df946a3cc590cf2546b06cf760eef451e2a6f88f525db23fd432f9a48" exitCode=0
Nov 01 03:39:57.883106 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:57.883028    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e8c77d-75ffdb94b5-7klwh" event={"ID":"c4ef178c-04fb-43cf-a0e9-e6a42a087df5","Type":"ContainerDied","Data":"bfd1c81df946a3cc590cf2546b06cf760eef451e2a6f88f525db23fd432f9a48"}
Nov 01 03:39:57.883106 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:57.883068    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e8c77d-75ffdb94b5-7klwh" event={"ID":"c4ef178c-04fb-43cf-a0e9-e6a42a087df5","Type":"ContainerDied","Data":"67b2e946bb12892529fdeca56608134e7b2b4e20703eebe14e0a50c30b0d977a"}
Nov 01 03:39:57.883236 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:57.883129    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e8c77d-75ffdb94b5-7klwh"
Nov 01 03:39:57.902406 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:57.902387    4930 scope.go:122] "RemoveContainer" containerID="08f24f5ab0b16965c447ba252ba9210546d2e5920e047780a2a4e3335ea58a8f"
Nov 01 03:39:57.902732 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:39:57.902703    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08f24f5ab0b16965c447ba252ba9210546d2e5920e047780a2a4e3335ea58a8f\": container with ID starting with 08f24f5ab0b16965c447ba252ba9210546d2e5920e047780a2a4e3335ea58a8f not found: ID does not exist" containerID="08f24f5ab0b16965c447ba252ba9210546d2e5920e047780a2a4e3335ea58a8f"
Nov 01 03:39:57.902844 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:57.902738    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08f24f5ab0b16965c447ba252ba9210546d2e5920e047780a2a4e3335ea58a8f"} err="failed to get container status \"08f24f5ab0b16965c447ba252ba9210546d2e5920e047780a2a4e3335ea58a8f\": rpc error: code = NotFound desc = could not find container \"08f24f5ab0b16965c447ba252ba9210546d2e5920e047780a2a4e3335ea58a8f\": container with ID starting with 08f24f5ab0b16965c447ba252ba9210546d2e5920e047780a2a4e3335ea58a8f not found: ID does not exist"
Nov 01 03:39:57.902844 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:57.902763    4930 scope.go:122] "RemoveContainer" containerID="bfd1c81df946a3cc590cf2546b06cf760eef451e2a6f88f525db23fd432f9a48"
Nov 01 03:39:57.911258 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:57.911234    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97r6f\" (UniqueName: \"kubernetes.io/projected/56f52541-2d41-4f11-a47f-489bfce8b284-kube-api-access-97r6f\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:39:57.913365 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:57.913344    4930 scope.go:122] "RemoveContainer" containerID="bfd1c81df946a3cc590cf2546b06cf760eef451e2a6f88f525db23fd432f9a48"
Nov 01 03:39:57.913643 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:39:57.913619    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfd1c81df946a3cc590cf2546b06cf760eef451e2a6f88f525db23fd432f9a48\": container with ID starting with bfd1c81df946a3cc590cf2546b06cf760eef451e2a6f88f525db23fd432f9a48 not found: ID does not exist" containerID="bfd1c81df946a3cc590cf2546b06cf760eef451e2a6f88f525db23fd432f9a48"
Nov 01 03:39:57.913735 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:57.913656    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfd1c81df946a3cc590cf2546b06cf760eef451e2a6f88f525db23fd432f9a48"} err="failed to get container status \"bfd1c81df946a3cc590cf2546b06cf760eef451e2a6f88f525db23fd432f9a48\": rpc error: code = NotFound desc = could not find container \"bfd1c81df946a3cc590cf2546b06cf760eef451e2a6f88f525db23fd432f9a48\": container with ID starting with bfd1c81df946a3cc590cf2546b06cf760eef451e2a6f88f525db23fd432f9a48 not found: ID does not exist"
Nov 01 03:39:58.050757 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:58.050726    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e8c77d-68444c5d47-bj8bs"]
Nov 01 03:39:58.064120 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:58.064047    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e8c77d-68444c5d47-bj8bs"]
Nov 01 03:39:58.174771 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:58.174728    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-e8c77d-75ffdb94b5-7klwh"]
Nov 01 03:39:58.180487 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:58.180458    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-e8c77d-75ffdb94b5-7klwh"]
Nov 01 03:39:59.415988 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:59.415933    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0cb10b-6cb84f4cf5-qkv5p"]
Nov 01 03:39:59.530052 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:59.530005    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0cb10b-5597fdc799-qmc8n"]
Nov 01 03:39:59.581643 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:59.581612    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:39:59.622754 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:59.622723    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56f52541-2d41-4f11-a47f-489bfce8b284" path="/var/lib/kubelet/pods/56f52541-2d41-4f11-a47f-489bfce8b284/volumes"
Nov 01 03:39:59.623194 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:59.623180    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4ef178c-04fb-43cf-a0e9-e6a42a087df5" path="/var/lib/kubelet/pods/c4ef178c-04fb-43cf-a0e9-e6a42a087df5/volumes"
Nov 01 03:39:59.777179 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:59.777085    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:39:59.922965 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:39:59.922928    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:40:03.312677 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:03.312626    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:40:03.562084 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:03.556231    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:40:03.744656 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:03.744620    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-22e11b-5ff5d6f995-fkzzs"]
Nov 01 03:40:03.745688 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:03.745662    4930 cpu_manager.go:414] "containerMap: removing container" podUID="56f52541-2d41-4f11-a47f-489bfce8b284" containerName="payload-pre-processing"
Nov 01 03:40:03.745873 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:03.745855    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56f52541-2d41-4f11-a47f-489bfce8b284" containerName="payload-pre-processing"
Nov 01 03:40:03.745963 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:03.745955    4930 cpu_manager.go:414] "containerMap: removing container" podUID="3c321a5a-ae6b-49aa-bbc0-e02778d1b539" containerName="istio-proxy"
Nov 01 03:40:03.746025 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:03.746014    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c321a5a-ae6b-49aa-bbc0-e02778d1b539" containerName="istio-proxy"
Nov 01 03:40:03.746118 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:03.746106    4930 cpu_manager.go:414] "containerMap: removing container" podUID="dbb22944-8ec6-44e6-b09b-d0d9478afb7a" containerName="revoke-keys"
Nov 01 03:40:03.746203 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:03.746194    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbb22944-8ec6-44e6-b09b-d0d9478afb7a" containerName="revoke-keys"
Nov 01 03:40:03.746362 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:03.746351    4930 cpu_manager.go:414] "containerMap: removing container" podUID="e9a70ec2-21fd-40ae-8d18-025a67248fce" containerName="istio-proxy"
Nov 01 03:40:03.746778 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:03.746759    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9a70ec2-21fd-40ae-8d18-025a67248fce" containerName="istio-proxy"
Nov 01 03:40:03.747168 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:03.747148    4930 cpu_manager.go:414] "containerMap: removing container" podUID="c4ef178c-04fb-43cf-a0e9-e6a42a087df5" containerName="payload-processing"
Nov 01 03:40:03.747292 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:03.747279    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4ef178c-04fb-43cf-a0e9-e6a42a087df5" containerName="payload-processing"
Nov 01 03:40:03.747637 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:03.747613    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="56f52541-2d41-4f11-a47f-489bfce8b284" containerName="payload-pre-processing"
Nov 01 03:40:03.747637 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:03.747639    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c321a5a-ae6b-49aa-bbc0-e02778d1b539" containerName="istio-proxy"
Nov 01 03:40:03.747796 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:03.747653    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbb22944-8ec6-44e6-b09b-d0d9478afb7a" containerName="revoke-keys"
Nov 01 03:40:03.747796 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:03.747668    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9a70ec2-21fd-40ae-8d18-025a67248fce" containerName="istio-proxy"
Nov 01 03:40:03.747796 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:03.747680    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4ef178c-04fb-43cf-a0e9-e6a42a087df5" containerName="payload-processing"
Nov 01 03:40:03.753131 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:03.753111    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22e11b-5ff5d6f995-fkzzs"
Nov 01 03:40:03.806153 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:03.805322    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-22e11b\""
Nov 01 03:40:03.806280 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:03.805330    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-22e11b-dockercfg-bbt7v\""
Nov 01 03:40:03.809057 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:03.809034    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-22e11b-5ff5d6f995-fkzzs"]
Nov 01 03:40:04.009514 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.009425    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03b489d4-3a39-4830-8629-d2a41315182d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-22e11b-5ff5d6f995-fkzzs\" (UID: \"03b489d4-3a39-4830-8629-d2a41315182d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22e11b-5ff5d6f995-fkzzs"
Nov 01 03:40:04.010004 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.009976    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdlc5\" (UniqueName: \"kubernetes.io/projected/03b489d4-3a39-4830-8629-d2a41315182d-kube-api-access-kdlc5\") pod \"payload-pre-processing-e2e-inf-a-22e11b-5ff5d6f995-fkzzs\" (UID: \"03b489d4-3a39-4830-8629-d2a41315182d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22e11b-5ff5d6f995-fkzzs"
Nov 01 03:40:04.057581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.056834    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-22e11b-6b684b568c-jvmc9"]
Nov 01 03:40:04.063753 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.063707    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-22e11b-6b684b568c-jvmc9"
Nov 01 03:40:04.069276 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.069180    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-22e11b-6b684b568c-jvmc9"]
Nov 01 03:40:04.123804 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.123772    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdlc5\" (UniqueName: \"kubernetes.io/projected/03b489d4-3a39-4830-8629-d2a41315182d-kube-api-access-kdlc5\") pod \"payload-pre-processing-e2e-inf-a-22e11b-5ff5d6f995-fkzzs\" (UID: \"03b489d4-3a39-4830-8629-d2a41315182d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22e11b-5ff5d6f995-fkzzs"
Nov 01 03:40:04.124002 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.123897    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d91bba7f-304b-424e-a819-d78a9275f6be-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-22e11b-6b684b568c-jvmc9\" (UID: \"d91bba7f-304b-424e-a819-d78a9275f6be\") " pod="openshift-ingress/payload-processing-e2e-inf-a-22e11b-6b684b568c-jvmc9"
Nov 01 03:40:04.124002 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.123989    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03b489d4-3a39-4830-8629-d2a41315182d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-22e11b-5ff5d6f995-fkzzs\" (UID: \"03b489d4-3a39-4830-8629-d2a41315182d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22e11b-5ff5d6f995-fkzzs"
Nov 01 03:40:04.124192 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.124026    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlj2z\" (UniqueName: \"kubernetes.io/projected/d91bba7f-304b-424e-a819-d78a9275f6be-kube-api-access-nlj2z\") pod \"payload-processing-e2e-inf-a-22e11b-6b684b568c-jvmc9\" (UID: \"d91bba7f-304b-424e-a819-d78a9275f6be\") " pod="openshift-ingress/payload-processing-e2e-inf-a-22e11b-6b684b568c-jvmc9"
Nov 01 03:40:04.125140 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.125116    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03b489d4-3a39-4830-8629-d2a41315182d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-22e11b-5ff5d6f995-fkzzs\" (UID: \"03b489d4-3a39-4830-8629-d2a41315182d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22e11b-5ff5d6f995-fkzzs"
Nov 01 03:40:04.143256 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.143229    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d91612c9-75kzp"]
Nov 01 03:40:04.147847 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.147830    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d91612c9-75kzp"
Nov 01 03:40:04.185257 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.185228    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdlc5\" (UniqueName: \"kubernetes.io/projected/03b489d4-3a39-4830-8629-d2a41315182d-kube-api-access-kdlc5\") pod \"payload-pre-processing-e2e-inf-a-22e11b-5ff5d6f995-fkzzs\" (UID: \"03b489d4-3a39-4830-8629-d2a41315182d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22e11b-5ff5d6f995-fkzzs"
Nov 01 03:40:04.200939 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.200540    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:40:04.200939 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.200749    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:40:04.200939 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.200907    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:40:04.201159 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.200954    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:40:04.205433 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.205410    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d91612c9-75kzp"]
Nov 01 03:40:04.226531 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.226495    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d91bba7f-304b-424e-a819-d78a9275f6be-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-22e11b-6b684b568c-jvmc9\" (UID: \"d91bba7f-304b-424e-a819-d78a9275f6be\") " pod="openshift-ingress/payload-processing-e2e-inf-a-22e11b-6b684b568c-jvmc9"
Nov 01 03:40:04.226701 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.226610    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62aeb801-0607-477c-81b6-7a9f4f0d668b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d91612c9-75kzp\" (UID: \"62aeb801-0607-477c-81b6-7a9f4f0d668b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d91612c9-75kzp"
Nov 01 03:40:04.226701 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.226694    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq294\" (UniqueName: \"kubernetes.io/projected/62aeb801-0607-477c-81b6-7a9f4f0d668b-kube-api-access-rq294\") pod \"maas-api-revoke-sub-d91612c9-75kzp\" (UID: \"62aeb801-0607-477c-81b6-7a9f4f0d668b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d91612c9-75kzp"
Nov 01 03:40:04.226850 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.226820    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62aeb801-0607-477c-81b6-7a9f4f0d668b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d91612c9-75kzp\" (UID: \"62aeb801-0607-477c-81b6-7a9f4f0d668b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d91612c9-75kzp"
Nov 01 03:40:04.226919 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.226873    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlj2z\" (UniqueName: \"kubernetes.io/projected/d91bba7f-304b-424e-a819-d78a9275f6be-kube-api-access-nlj2z\") pod \"payload-processing-e2e-inf-a-22e11b-6b684b568c-jvmc9\" (UID: \"d91bba7f-304b-424e-a819-d78a9275f6be\") " pod="openshift-ingress/payload-processing-e2e-inf-a-22e11b-6b684b568c-jvmc9"
Nov 01 03:40:04.227339 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.227318    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d91bba7f-304b-424e-a819-d78a9275f6be-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-22e11b-6b684b568c-jvmc9\" (UID: \"d91bba7f-304b-424e-a819-d78a9275f6be\") " pod="openshift-ingress/payload-processing-e2e-inf-a-22e11b-6b684b568c-jvmc9"
Nov 01 03:40:04.287430 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.287371    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlj2z\" (UniqueName: \"kubernetes.io/projected/d91bba7f-304b-424e-a819-d78a9275f6be-kube-api-access-nlj2z\") pod \"payload-processing-e2e-inf-a-22e11b-6b684b568c-jvmc9\" (UID: \"d91bba7f-304b-424e-a819-d78a9275f6be\") " pod="openshift-ingress/payload-processing-e2e-inf-a-22e11b-6b684b568c-jvmc9"
Nov 01 03:40:04.328818 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.328789    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62aeb801-0607-477c-81b6-7a9f4f0d668b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d91612c9-75kzp\" (UID: \"62aeb801-0607-477c-81b6-7a9f4f0d668b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d91612c9-75kzp"
Nov 01 03:40:04.329229 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.328876    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq294\" (UniqueName: \"kubernetes.io/projected/62aeb801-0607-477c-81b6-7a9f4f0d668b-kube-api-access-rq294\") pod \"maas-api-revoke-sub-d91612c9-75kzp\" (UID: \"62aeb801-0607-477c-81b6-7a9f4f0d668b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d91612c9-75kzp"
Nov 01 03:40:04.329229 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.328951    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62aeb801-0607-477c-81b6-7a9f4f0d668b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d91612c9-75kzp\" (UID: \"62aeb801-0607-477c-81b6-7a9f4f0d668b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d91612c9-75kzp"
Nov 01 03:40:04.329605 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.329550    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62aeb801-0607-477c-81b6-7a9f4f0d668b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d91612c9-75kzp\" (UID: \"62aeb801-0607-477c-81b6-7a9f4f0d668b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d91612c9-75kzp"
Nov 01 03:40:04.331053 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.331032    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62aeb801-0607-477c-81b6-7a9f4f0d668b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d91612c9-75kzp\" (UID: \"62aeb801-0607-477c-81b6-7a9f4f0d668b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d91612c9-75kzp"
Nov 01 03:40:04.368367 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.368341    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22e11b-5ff5d6f995-fkzzs"
Nov 01 03:40:04.389830 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.389807    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq294\" (UniqueName: \"kubernetes.io/projected/62aeb801-0607-477c-81b6-7a9f4f0d668b-kube-api-access-rq294\") pod \"maas-api-revoke-sub-d91612c9-75kzp\" (UID: \"62aeb801-0607-477c-81b6-7a9f4f0d668b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d91612c9-75kzp"
Nov 01 03:40:04.416536 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.416499    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0cb10b-6cb84f4cf5-qkv5p" podUID="279f59c4-604c-4352-bc67-8f2bd727f105" containerName="payload-processing" containerID="cri-o://a466f818cf9e639733733f1c1b8a0dfe4f4715e975b56342f76a7eb0ff0505fd" gracePeriod=25
Nov 01 03:40:04.423209 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.423184    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-22e11b-6b684b568c-jvmc9"
Nov 01 03:40:04.481307 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.480543    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d91612c9-75kzp"
Nov 01 03:40:04.531127 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.530943    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0cb10b-5597fdc799-qmc8n" podUID="9788fdf0-ba01-4cd7-8afb-5cd53881ffbf" containerName="payload-pre-processing" containerID="cri-o://461e20ffc571b8a786ec44aba1082f0fb2f1b75446963723e57d380c7aad31ea" gracePeriod=25
Nov 01 03:40:04.754478 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.754356    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-22e11b-5ff5d6f995-fkzzs"]
Nov 01 03:40:04.799481 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.798982    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0cb10b-6cb84f4cf5-qkv5p"
Nov 01 03:40:04.838114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.836034    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/279f59c4-604c-4352-bc67-8f2bd727f105-kube-api-access-dwv6j\" (UniqueName: \"kubernetes.io/projected/279f59c4-604c-4352-bc67-8f2bd727f105-kube-api-access-dwv6j\") pod \"279f59c4-604c-4352-bc67-8f2bd727f105\" (UID: \"279f59c4-604c-4352-bc67-8f2bd727f105\") "
Nov 01 03:40:04.838114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.836216    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/279f59c4-604c-4352-bc67-8f2bd727f105-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/279f59c4-604c-4352-bc67-8f2bd727f105-plugins-config-volume\") pod \"279f59c4-604c-4352-bc67-8f2bd727f105\" (UID: \"279f59c4-604c-4352-bc67-8f2bd727f105\") "
Nov 01 03:40:04.845122 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.838458    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/279f59c4-604c-4352-bc67-8f2bd727f105-kube-api-access-dwv6j" pod "279f59c4-604c-4352-bc67-8f2bd727f105" (UID: "279f59c4-604c-4352-bc67-8f2bd727f105"). InnerVolumeSpecName "kube-api-access-dwv6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:40:04.845122 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.838632    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/279f59c4-604c-4352-bc67-8f2bd727f105-plugins-config-volume" pod "279f59c4-604c-4352-bc67-8f2bd727f105" (UID: "279f59c4-604c-4352-bc67-8f2bd727f105"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:40:04.868803 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.868772    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-22e11b-6b684b568c-jvmc9"]
Nov 01 03:40:04.901987 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.901956    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0cb10b-5597fdc799-qmc8n"
Nov 01 03:40:04.906365 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.906327    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:40:04.917233 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.917211    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d91612c9-75kzp"]
Nov 01 03:40:04.927020 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:40:04.926996    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62aeb801_0607_477c_81b6_7a9f4f0d668b.slice/crio-b5b68078f564f25e9c993d146396b8d75c7dd1af90099fbb258cae0d26e20e35 WatchSource:0}: Error finding container b5b68078f564f25e9c993d146396b8d75c7dd1af90099fbb258cae0d26e20e35: Status 404 returned error can't find the container with id b5b68078f564f25e9c993d146396b8d75c7dd1af90099fbb258cae0d26e20e35
Nov 01 03:40:04.937943 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.937919    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9788fdf0-ba01-4cd7-8afb-5cd53881ffbf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9788fdf0-ba01-4cd7-8afb-5cd53881ffbf-plugins-config-volume\") pod \"9788fdf0-ba01-4cd7-8afb-5cd53881ffbf\" (UID: \"9788fdf0-ba01-4cd7-8afb-5cd53881ffbf\") "
Nov 01 03:40:04.938237 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.938216    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9788fdf0-ba01-4cd7-8afb-5cd53881ffbf-kube-api-access-t96tv\" (UniqueName: \"kubernetes.io/projected/9788fdf0-ba01-4cd7-8afb-5cd53881ffbf-kube-api-access-t96tv\") pod \"9788fdf0-ba01-4cd7-8afb-5cd53881ffbf\" (UID: \"9788fdf0-ba01-4cd7-8afb-5cd53881ffbf\") "
Nov 01 03:40:04.950180 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.949806    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9788fdf0-ba01-4cd7-8afb-5cd53881ffbf-plugins-config-volume" pod "9788fdf0-ba01-4cd7-8afb-5cd53881ffbf" (UID: "9788fdf0-ba01-4cd7-8afb-5cd53881ffbf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:40:04.984580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.965414    4930 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9788fdf0-ba01-4cd7-8afb-5cd53881ffbf-plugins-config-volume\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:40:04.985001 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.984973    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwv6j\" (UniqueName: \"kubernetes.io/projected/279f59c4-604c-4352-bc67-8f2bd727f105-kube-api-access-dwv6j\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:40:04.985130 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.985117    4930 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/279f59c4-604c-4352-bc67-8f2bd727f105-plugins-config-volume\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:40:04.985220 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:04.981163    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9788fdf0-ba01-4cd7-8afb-5cd53881ffbf-kube-api-access-t96tv" pod "9788fdf0-ba01-4cd7-8afb-5cd53881ffbf" (UID: "9788fdf0-ba01-4cd7-8afb-5cd53881ffbf"). InnerVolumeSpecName "kube-api-access-t96tv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:40:05.029486 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:05.029453    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d91612c9-75kzp" event={"ID":"62aeb801-0607-477c-81b6-7a9f4f0d668b","Type":"ContainerStarted","Data":"b5b68078f564f25e9c993d146396b8d75c7dd1af90099fbb258cae0d26e20e35"}
Nov 01 03:40:05.031610 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:05.031584    4930 generic.go:356] "Generic (PLEG): container finished" podID="279f59c4-604c-4352-bc67-8f2bd727f105" containerID="a466f818cf9e639733733f1c1b8a0dfe4f4715e975b56342f76a7eb0ff0505fd" exitCode=0
Nov 01 03:40:05.031778 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:05.031760    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0cb10b-6cb84f4cf5-qkv5p" event={"ID":"279f59c4-604c-4352-bc67-8f2bd727f105","Type":"ContainerDied","Data":"a466f818cf9e639733733f1c1b8a0dfe4f4715e975b56342f76a7eb0ff0505fd"}
Nov 01 03:40:05.031853 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:05.031805    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0cb10b-6cb84f4cf5-qkv5p" event={"ID":"279f59c4-604c-4352-bc67-8f2bd727f105","Type":"ContainerDied","Data":"ee52978f03144abf9de27031cd8593961622fdfd69553e70c919d3e3b8f19932"}
Nov 01 03:40:05.031853 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:05.031828    4930 scope.go:122] "RemoveContainer" containerID="a466f818cf9e639733733f1c1b8a0dfe4f4715e975b56342f76a7eb0ff0505fd"
Nov 01 03:40:05.031974 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:05.031946    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0cb10b-6cb84f4cf5-qkv5p"
Nov 01 03:40:05.033928 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:05.033903    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22e11b-5ff5d6f995-fkzzs" event={"ID":"03b489d4-3a39-4830-8629-d2a41315182d","Type":"ContainerStarted","Data":"8e475a0c4374bd50c56b34d778bf910a563369859d9df2ded6999f3cc97a2bba"}
Nov 01 03:40:05.034050 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:05.033945    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22e11b-5ff5d6f995-fkzzs" event={"ID":"03b489d4-3a39-4830-8629-d2a41315182d","Type":"ContainerStarted","Data":"bd00e1a4ea172f667468bf44950d35789afb8161c92442842870fe7a2e4f74d4"}
Nov 01 03:40:05.034523 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:05.034505    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22e11b-5ff5d6f995-fkzzs"
Nov 01 03:40:05.035902 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:05.035881    4930 generic.go:356] "Generic (PLEG): container finished" podID="9788fdf0-ba01-4cd7-8afb-5cd53881ffbf" containerID="461e20ffc571b8a786ec44aba1082f0fb2f1b75446963723e57d380c7aad31ea" exitCode=0
Nov 01 03:40:05.036002 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:05.035969    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0cb10b-5597fdc799-qmc8n" event={"ID":"9788fdf0-ba01-4cd7-8afb-5cd53881ffbf","Type":"ContainerDied","Data":"461e20ffc571b8a786ec44aba1082f0fb2f1b75446963723e57d380c7aad31ea"}
Nov 01 03:40:05.036002 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:05.035996    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0cb10b-5597fdc799-qmc8n" event={"ID":"9788fdf0-ba01-4cd7-8afb-5cd53881ffbf","Type":"ContainerDied","Data":"dc4c5b5c70744cd80ea941c1d571141bd51715905ea067e4ee9c1e6988d9e809"}
Nov 01 03:40:05.036121 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:05.036048    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0cb10b-5597fdc799-qmc8n"
Nov 01 03:40:05.046870 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:05.046847    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-22e11b-6b684b568c-jvmc9" event={"ID":"d91bba7f-304b-424e-a819-d78a9275f6be","Type":"ContainerStarted","Data":"d9c41f0ca27b9bf7e42ed60391536d0c2084c6c02064fab21415e611689707a4"}
Nov 01 03:40:05.048412 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:05.048390    4930 scope.go:122] "RemoveContainer" containerID="a466f818cf9e639733733f1c1b8a0dfe4f4715e975b56342f76a7eb0ff0505fd"
Nov 01 03:40:05.048849 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:40:05.048824    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a466f818cf9e639733733f1c1b8a0dfe4f4715e975b56342f76a7eb0ff0505fd\": container with ID starting with a466f818cf9e639733733f1c1b8a0dfe4f4715e975b56342f76a7eb0ff0505fd not found: ID does not exist" containerID="a466f818cf9e639733733f1c1b8a0dfe4f4715e975b56342f76a7eb0ff0505fd"
Nov 01 03:40:05.049037 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:05.049002    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a466f818cf9e639733733f1c1b8a0dfe4f4715e975b56342f76a7eb0ff0505fd"} err="failed to get container status \"a466f818cf9e639733733f1c1b8a0dfe4f4715e975b56342f76a7eb0ff0505fd\": rpc error: code = NotFound desc = could not find container \"a466f818cf9e639733733f1c1b8a0dfe4f4715e975b56342f76a7eb0ff0505fd\": container with ID starting with a466f818cf9e639733733f1c1b8a0dfe4f4715e975b56342f76a7eb0ff0505fd not found: ID does not exist"
Nov 01 03:40:05.049174 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:05.049037    4930 scope.go:122] "RemoveContainer" containerID="461e20ffc571b8a786ec44aba1082f0fb2f1b75446963723e57d380c7aad31ea"
Nov 01 03:40:05.061921 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:05.061900    4930 scope.go:122] "RemoveContainer" containerID="461e20ffc571b8a786ec44aba1082f0fb2f1b75446963723e57d380c7aad31ea"
Nov 01 03:40:05.062242 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:40:05.062221    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"461e20ffc571b8a786ec44aba1082f0fb2f1b75446963723e57d380c7aad31ea\": container with ID starting with 461e20ffc571b8a786ec44aba1082f0fb2f1b75446963723e57d380c7aad31ea not found: ID does not exist" containerID="461e20ffc571b8a786ec44aba1082f0fb2f1b75446963723e57d380c7aad31ea"
Nov 01 03:40:05.062320 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:05.062249    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"461e20ffc571b8a786ec44aba1082f0fb2f1b75446963723e57d380c7aad31ea"} err="failed to get container status \"461e20ffc571b8a786ec44aba1082f0fb2f1b75446963723e57d380c7aad31ea\": rpc error: code = NotFound desc = could not find container \"461e20ffc571b8a786ec44aba1082f0fb2f1b75446963723e57d380c7aad31ea\": container with ID starting with 461e20ffc571b8a786ec44aba1082f0fb2f1b75446963723e57d380c7aad31ea not found: ID does not exist"
Nov 01 03:40:05.086338 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:05.086304    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t96tv\" (UniqueName: \"kubernetes.io/projected/9788fdf0-ba01-4cd7-8afb-5cd53881ffbf-kube-api-access-t96tv\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:40:05.271074 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:05.271020    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22e11b-5ff5d6f995-fkzzs" podStartSLOduration=2.2710054729999998 podStartE2EDuration="2.271005473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:40:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:40:05.269961153 +0000 UTC m=+2009.792967742" watchObservedRunningTime="2026-11-01 03:40:05.271005473 +0000 UTC m=+2009.794012061"
Nov 01 03:40:05.399768 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:05.399629    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0cb10b-6cb84f4cf5-qkv5p"]
Nov 01 03:40:05.408701 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:05.408659    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0cb10b-6cb84f4cf5-qkv5p"]
Nov 01 03:40:05.508660 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:05.508625    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0cb10b-5597fdc799-qmc8n"]
Nov 01 03:40:05.510057 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:05.510035    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0cb10b-5597fdc799-qmc8n"]
Nov 01 03:40:05.622754 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:05.622725    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="279f59c4-604c-4352-bc67-8f2bd727f105" path="/var/lib/kubelet/pods/279f59c4-604c-4352-bc67-8f2bd727f105/volumes"
Nov 01 03:40:05.623201 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:05.623185    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9788fdf0-ba01-4cd7-8afb-5cd53881ffbf" path="/var/lib/kubelet/pods/9788fdf0-ba01-4cd7-8afb-5cd53881ffbf/volumes"
Nov 01 03:40:06.057103 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:06.057069    4930 generic.go:356] "Generic (PLEG): container finished" podID="62aeb801-0607-477c-81b6-7a9f4f0d668b" containerID="b3e97344120d9e39da6dccc6c71483f63018238994907b311bb8d4918341c1d3" exitCode=0
Nov 01 03:40:06.057275 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:06.057153    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d91612c9-75kzp" event={"ID":"62aeb801-0607-477c-81b6-7a9f4f0d668b","Type":"ContainerDied","Data":"b3e97344120d9e39da6dccc6c71483f63018238994907b311bb8d4918341c1d3"}
Nov 01 03:40:06.060951 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:06.060922    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-22e11b-6b684b568c-jvmc9" event={"ID":"d91bba7f-304b-424e-a819-d78a9275f6be","Type":"ContainerStarted","Data":"b1b926ecb93a81542e99b31821d658d34eca6e07c62d5ad26f68656f96488a59"}
Nov 01 03:40:06.061118 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:06.061089    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-22e11b-6b684b568c-jvmc9"
Nov 01 03:40:06.291238 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:06.291136    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-22e11b-6b684b568c-jvmc9" podStartSLOduration=3.291123822 podStartE2EDuration="3.291123822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:40:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:40:06.287964266 +0000 UTC m=+2010.810970855" watchObservedRunningTime="2026-11-01 03:40:06.291123822 +0000 UTC m=+2010.814130411"
Nov 01 03:40:07.258610 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:07.258572    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d91612c9-75kzp"
Nov 01 03:40:07.425736 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:07.425699    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62aeb801-0607-477c-81b6-7a9f4f0d668b-kube-api-access-rq294\" (UniqueName: \"kubernetes.io/projected/62aeb801-0607-477c-81b6-7a9f4f0d668b-kube-api-access-rq294\") pod \"62aeb801-0607-477c-81b6-7a9f4f0d668b\" (UID: \"62aeb801-0607-477c-81b6-7a9f4f0d668b\") "
Nov 01 03:40:07.425945 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:07.425910    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62aeb801-0607-477c-81b6-7a9f4f0d668b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62aeb801-0607-477c-81b6-7a9f4f0d668b-maas-api-xks-ca\") pod \"62aeb801-0607-477c-81b6-7a9f4f0d668b\" (UID: \"62aeb801-0607-477c-81b6-7a9f4f0d668b\") "
Nov 01 03:40:07.426030 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:07.425957    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62aeb801-0607-477c-81b6-7a9f4f0d668b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62aeb801-0607-477c-81b6-7a9f4f0d668b-maas-api-openshift-ca\") pod \"62aeb801-0607-477c-81b6-7a9f4f0d668b\" (UID: \"62aeb801-0607-477c-81b6-7a9f4f0d668b\") "
Nov 01 03:40:07.427115 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:07.427085    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62aeb801-0607-477c-81b6-7a9f4f0d668b-maas-api-openshift-ca" pod "62aeb801-0607-477c-81b6-7a9f4f0d668b" (UID: "62aeb801-0607-477c-81b6-7a9f4f0d668b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:40:07.428410 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:07.428362    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62aeb801-0607-477c-81b6-7a9f4f0d668b-kube-api-access-rq294" pod "62aeb801-0607-477c-81b6-7a9f4f0d668b" (UID: "62aeb801-0607-477c-81b6-7a9f4f0d668b"). InnerVolumeSpecName "kube-api-access-rq294". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:40:07.429889 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:07.429862    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62aeb801-0607-477c-81b6-7a9f4f0d668b-maas-api-xks-ca" pod "62aeb801-0607-477c-81b6-7a9f4f0d668b" (UID: "62aeb801-0607-477c-81b6-7a9f4f0d668b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:40:07.528355 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:07.528325    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq294\" (UniqueName: \"kubernetes.io/projected/62aeb801-0607-477c-81b6-7a9f4f0d668b-kube-api-access-rq294\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:40:07.528355 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:07.528353    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62aeb801-0607-477c-81b6-7a9f4f0d668b-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:40:07.528581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:07.528364    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62aeb801-0607-477c-81b6-7a9f4f0d668b-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:40:08.080797 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:08.080769    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d91612c9-75kzp"
Nov 01 03:40:08.080972 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:08.080801    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d91612c9-75kzp" event={"ID":"62aeb801-0607-477c-81b6-7a9f4f0d668b","Type":"ContainerDied","Data":"b5b68078f564f25e9c993d146396b8d75c7dd1af90099fbb258cae0d26e20e35"}
Nov 01 03:40:08.080972 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:08.080851    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5b68078f564f25e9c993d146396b8d75c7dd1af90099fbb258cae0d26e20e35"
Nov 01 03:40:10.124806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:10.124766    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:40:10.280295 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:10.280250    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:40:10.426013 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:10.425895    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:40:11.384391 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:11.384086    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d7329-h5bnz"]
Nov 01 03:40:11.385165 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:11.385133    4930 cpu_manager.go:414] "containerMap: removing container" podUID="9788fdf0-ba01-4cd7-8afb-5cd53881ffbf" containerName="payload-pre-processing"
Nov 01 03:40:11.385165 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:11.385158    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9788fdf0-ba01-4cd7-8afb-5cd53881ffbf" containerName="payload-pre-processing"
Nov 01 03:40:11.385276 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:11.385221    4930 cpu_manager.go:414] "containerMap: removing container" podUID="279f59c4-604c-4352-bc67-8f2bd727f105" containerName="payload-processing"
Nov 01 03:40:11.385276 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:11.385229    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="279f59c4-604c-4352-bc67-8f2bd727f105" containerName="payload-processing"
Nov 01 03:40:11.385356 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:11.385280    4930 cpu_manager.go:414] "containerMap: removing container" podUID="62aeb801-0607-477c-81b6-7a9f4f0d668b" containerName="revoke-keys"
Nov 01 03:40:11.385356 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:11.385288    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62aeb801-0607-477c-81b6-7a9f4f0d668b" containerName="revoke-keys"
Nov 01 03:40:11.385459 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:11.385446    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="279f59c4-604c-4352-bc67-8f2bd727f105" containerName="payload-processing"
Nov 01 03:40:11.385502 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:11.385471    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="62aeb801-0607-477c-81b6-7a9f4f0d668b" containerName="revoke-keys"
Nov 01 03:40:11.385502 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:11.385485    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="9788fdf0-ba01-4cd7-8afb-5cd53881ffbf" containerName="payload-pre-processing"
Nov 01 03:40:11.390024 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:11.390003    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d7329-h5bnz"
Nov 01 03:40:11.444450 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:11.444416    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d7329-h5bnz"]
Nov 01 03:40:11.488642 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:11.488613    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:40:11.488816 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:11.488721    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:40:11.488816 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:11.488741    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:40:11.488816 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:11.488742    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:40:11.505509 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:11.505478    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds2l6\" (UniqueName: \"kubernetes.io/projected/389fb913-8428-49ca-bf68-4a9475625dc5-kube-api-access-ds2l6\") pod \"maas-api-revoke-keys-e2e-inf-a-7d7329-h5bnz\" (UID: \"389fb913-8428-49ca-bf68-4a9475625dc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d7329-h5bnz"
Nov 01 03:40:11.505668 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:11.505541    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/389fb913-8428-49ca-bf68-4a9475625dc5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7d7329-h5bnz\" (UID: \"389fb913-8428-49ca-bf68-4a9475625dc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d7329-h5bnz"
Nov 01 03:40:11.505715 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:11.505663    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/389fb913-8428-49ca-bf68-4a9475625dc5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7d7329-h5bnz\" (UID: \"389fb913-8428-49ca-bf68-4a9475625dc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d7329-h5bnz"
Nov 01 03:40:11.607151 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:11.607113    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/389fb913-8428-49ca-bf68-4a9475625dc5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7d7329-h5bnz\" (UID: \"389fb913-8428-49ca-bf68-4a9475625dc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d7329-h5bnz"
Nov 01 03:40:11.607319 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:11.607178    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/389fb913-8428-49ca-bf68-4a9475625dc5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7d7329-h5bnz\" (UID: \"389fb913-8428-49ca-bf68-4a9475625dc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d7329-h5bnz"
Nov 01 03:40:11.607519 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:11.607497    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds2l6\" (UniqueName: \"kubernetes.io/projected/389fb913-8428-49ca-bf68-4a9475625dc5-kube-api-access-ds2l6\") pod \"maas-api-revoke-keys-e2e-inf-a-7d7329-h5bnz\" (UID: \"389fb913-8428-49ca-bf68-4a9475625dc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d7329-h5bnz"
Nov 01 03:40:11.607976 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:11.607952    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/389fb913-8428-49ca-bf68-4a9475625dc5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7d7329-h5bnz\" (UID: \"389fb913-8428-49ca-bf68-4a9475625dc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d7329-h5bnz"
Nov 01 03:40:11.609797 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:11.609773    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/389fb913-8428-49ca-bf68-4a9475625dc5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7d7329-h5bnz\" (UID: \"389fb913-8428-49ca-bf68-4a9475625dc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d7329-h5bnz"
Nov 01 03:40:11.668420 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:11.668359    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds2l6\" (UniqueName: \"kubernetes.io/projected/389fb913-8428-49ca-bf68-4a9475625dc5-kube-api-access-ds2l6\") pod \"maas-api-revoke-keys-e2e-inf-a-7d7329-h5bnz\" (UID: \"389fb913-8428-49ca-bf68-4a9475625dc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d7329-h5bnz"
Nov 01 03:40:11.707052 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:11.707027    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d7329-h5bnz"
Nov 01 03:40:12.099516 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:12.099480    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d7329-h5bnz"]
Nov 01 03:40:12.099996 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:40:12.099960    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod389fb913_8428_49ca_bf68_4a9475625dc5.slice/crio-d70be43b7582259dfa0f5c6e1e37304bd77b392b2b1af6ded29ae971ec84e36b WatchSource:0}: Error finding container d70be43b7582259dfa0f5c6e1e37304bd77b392b2b1af6ded29ae971ec84e36b: Status 404 returned error can't find the container with id d70be43b7582259dfa0f5c6e1e37304bd77b392b2b1af6ded29ae971ec84e36b
Nov 01 03:40:12.133765 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:12.133734    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d7329-h5bnz" event={"ID":"389fb913-8428-49ca-bf68-4a9475625dc5","Type":"ContainerStarted","Data":"d70be43b7582259dfa0f5c6e1e37304bd77b392b2b1af6ded29ae971ec84e36b"}
Nov 01 03:40:13.206576 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:13.186991    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d7329-h5bnz" event={"ID":"389fb913-8428-49ca-bf68-4a9475625dc5","Type":"ContainerStarted","Data":"de34d01ca23a5c16a185f85a556db8581c5feb369faf7dcd4ff8b30f19db2a37"}
Nov 01 03:40:13.309316 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:13.309260    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d7329-h5bnz" podStartSLOduration=2.3092455530000002 podStartE2EDuration="2.309245553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:40:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:40:13.305313057 +0000 UTC m=+2017.828319646" watchObservedRunningTime="2026-11-01 03:40:13.309245553 +0000 UTC m=+2017.832252140"
Nov 01 03:40:14.193737 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:14.193698    4930 generic.go:356] "Generic (PLEG): container finished" podID="389fb913-8428-49ca-bf68-4a9475625dc5" containerID="de34d01ca23a5c16a185f85a556db8581c5feb369faf7dcd4ff8b30f19db2a37" exitCode=0
Nov 01 03:40:14.193941 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:14.193785    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d7329-h5bnz" event={"ID":"389fb913-8428-49ca-bf68-4a9475625dc5","Type":"ContainerDied","Data":"de34d01ca23a5c16a185f85a556db8581c5feb369faf7dcd4ff8b30f19db2a37"}
Nov 01 03:40:15.380486 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:15.380457    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d7329-h5bnz"
Nov 01 03:40:15.416241 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:15.416203    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:40:15.486432 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:15.485660    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/389fb913-8428-49ca-bf68-4a9475625dc5-kube-api-access-ds2l6\" (UniqueName: \"kubernetes.io/projected/389fb913-8428-49ca-bf68-4a9475625dc5-kube-api-access-ds2l6\") pod \"389fb913-8428-49ca-bf68-4a9475625dc5\" (UID: \"389fb913-8428-49ca-bf68-4a9475625dc5\") "
Nov 01 03:40:15.486432 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:15.485853    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/389fb913-8428-49ca-bf68-4a9475625dc5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/389fb913-8428-49ca-bf68-4a9475625dc5-maas-api-xks-ca\") pod \"389fb913-8428-49ca-bf68-4a9475625dc5\" (UID: \"389fb913-8428-49ca-bf68-4a9475625dc5\") "
Nov 01 03:40:15.486432 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:15.485950    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/389fb913-8428-49ca-bf68-4a9475625dc5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/389fb913-8428-49ca-bf68-4a9475625dc5-maas-api-openshift-ca\") pod \"389fb913-8428-49ca-bf68-4a9475625dc5\" (UID: \"389fb913-8428-49ca-bf68-4a9475625dc5\") "
Nov 01 03:40:15.487645 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:15.487215    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/389fb913-8428-49ca-bf68-4a9475625dc5-maas-api-openshift-ca" pod "389fb913-8428-49ca-bf68-4a9475625dc5" (UID: "389fb913-8428-49ca-bf68-4a9475625dc5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:40:15.490359 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:15.490326    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/389fb913-8428-49ca-bf68-4a9475625dc5-kube-api-access-ds2l6" pod "389fb913-8428-49ca-bf68-4a9475625dc5" (UID: "389fb913-8428-49ca-bf68-4a9475625dc5"). InnerVolumeSpecName "kube-api-access-ds2l6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:40:15.491102 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:15.491079    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/389fb913-8428-49ca-bf68-4a9475625dc5-maas-api-xks-ca" pod "389fb913-8428-49ca-bf68-4a9475625dc5" (UID: "389fb913-8428-49ca-bf68-4a9475625dc5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:40:15.588373 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:15.588339    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds2l6\" (UniqueName: \"kubernetes.io/projected/389fb913-8428-49ca-bf68-4a9475625dc5-kube-api-access-ds2l6\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:40:15.588373 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:15.588369    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/389fb913-8428-49ca-bf68-4a9475625dc5-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:40:15.588373 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:15.588381    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/389fb913-8428-49ca-bf68-4a9475625dc5-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:40:16.213825 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:16.213796    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d7329-h5bnz"
Nov 01 03:40:16.214015 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:16.213795    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d7329-h5bnz" event={"ID":"389fb913-8428-49ca-bf68-4a9475625dc5","Type":"ContainerDied","Data":"d70be43b7582259dfa0f5c6e1e37304bd77b392b2b1af6ded29ae971ec84e36b"}
Nov 01 03:40:16.214015 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:16.213914    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d70be43b7582259dfa0f5c6e1e37304bd77b392b2b1af6ded29ae971ec84e36b"
Nov 01 03:40:17.075195 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:17.075008    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-22e11b-6b684b568c-jvmc9"
Nov 01 03:40:17.075639 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:17.075325    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22e11b-5ff5d6f995-fkzzs"
Nov 01 03:40:19.122419 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:40:19.122384    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:40:20.451967 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:20.451927    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:40:23.556470 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:23.556432    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:40:23.698075 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:23.698041    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:40:25.360366 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:25.360316    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:40:30.088919 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:30.088879    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:40:34.881794 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:34.881753    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:40:35.775539 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:35.775493    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:40:38.304729 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:38.304691    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:40:39.045883 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:39.045842    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:40:39.209774 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:39.209741    4930 scope.go:122] "RemoveContainer" containerID="713cd81d0c64e18c01734345c7b21605bcb695d3f54ff1e8c04683ac9cb17abb"
Nov 01 03:40:40.233149 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:40.233117    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:40:40.607831 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:40.607794    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:40:40.918444 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:40.918342    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:40:41.172641 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:41.172312    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e8c77d-8cvvc"]
Nov 01 03:40:41.278394 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:41.278345    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-e8c77d-8cvvc"]
Nov 01 03:40:41.550801 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:41.550700    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:40:41.623040 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:41.623012    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7ae0e93-00ce-4e61-907d-691e02b4220e" path="/var/lib/kubelet/pods/a7ae0e93-00ce-4e61-907d-691e02b4220e/volumes"
Nov 01 03:40:41.967137 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:41.967090    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:40:42.229884 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:42.229795    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:40:42.380080 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:42.380033    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:40:42.467648 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:42.467620    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0cb10b-sz879"]
Nov 01 03:40:42.515149 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:42.515079    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-0cb10b-sz879"]
Nov 01 03:40:42.854921 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:42.854873    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:40:43.622482 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:43.622450    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbb22944-8ec6-44e6-b09b-d0d9478afb7a" path="/var/lib/kubelet/pods/dbb22944-8ec6-44e6-b09b-d0d9478afb7a/volumes"
Nov 01 03:40:47.591579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:47.589773    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:40:47.743371 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:47.743328    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:40:49.129093 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:40:49.129052    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:40:49.233716 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:49.233679    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:40:49.441158 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:49.441073    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:40:49.685261 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:49.685225    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"]
Nov 01 03:40:49.685509 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:49.685480    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh" podUID="a4cae562-15e5-4d0b-bcef-62ca774e2369" containerName="istio-proxy" containerID="cri-o://f94ccf4edcf71b1b502e4937f17e375337dffceb203a49f96738a5857c35eeb4" gracePeriod=30
Nov 01 03:40:49.769637 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:49.761068    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:40:50.489534 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:50.489479    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-438641-kw7df"]
Nov 01 03:40:50.492019 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:50.491994    4930 cpu_manager.go:414] "containerMap: removing container" podUID="389fb913-8428-49ca-bf68-4a9475625dc5" containerName="revoke-keys"
Nov 01 03:40:50.492213 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:50.492196    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="389fb913-8428-49ca-bf68-4a9475625dc5" containerName="revoke-keys"
Nov 01 03:40:50.492635 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:50.492619    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="389fb913-8428-49ca-bf68-4a9475625dc5" containerName="revoke-keys"
Nov 01 03:40:50.498916 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:50.498887    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-438641-kw7df"
Nov 01 03:40:50.534897 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:50.534868    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-438641-kw7df"]
Nov 01 03:40:50.594582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:50.584389    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:40:50.594582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:50.584452    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:40:50.613249 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:50.612807    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwn22\" (UniqueName: \"kubernetes.io/projected/b6346712-2068-4299-90f9-df1fcf150cc7-kube-api-access-rwn22\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-438641-kw7df\" (UID: \"b6346712-2068-4299-90f9-df1fcf150cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-438641-kw7df"
Nov 01 03:40:50.613249 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:50.612971    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6346712-2068-4299-90f9-df1fcf150cc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-438641-kw7df\" (UID: \"b6346712-2068-4299-90f9-df1fcf150cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-438641-kw7df"
Nov 01 03:40:50.613249 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:50.613106    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6346712-2068-4299-90f9-df1fcf150cc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-438641-kw7df\" (UID: \"b6346712-2068-4299-90f9-df1fcf150cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-438641-kw7df"
Nov 01 03:40:50.621189 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:50.621165    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:40:50.621305 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:50.621232    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:40:50.714670 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:50.714631    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwn22\" (UniqueName: \"kubernetes.io/projected/b6346712-2068-4299-90f9-df1fcf150cc7-kube-api-access-rwn22\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-438641-kw7df\" (UID: \"b6346712-2068-4299-90f9-df1fcf150cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-438641-kw7df"
Nov 01 03:40:50.714878 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:50.714802    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6346712-2068-4299-90f9-df1fcf150cc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-438641-kw7df\" (UID: \"b6346712-2068-4299-90f9-df1fcf150cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-438641-kw7df"
Nov 01 03:40:50.715032 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:50.714972    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6346712-2068-4299-90f9-df1fcf150cc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-438641-kw7df\" (UID: \"b6346712-2068-4299-90f9-df1fcf150cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-438641-kw7df"
Nov 01 03:40:50.715765 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:50.715743    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6346712-2068-4299-90f9-df1fcf150cc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-438641-kw7df\" (UID: \"b6346712-2068-4299-90f9-df1fcf150cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-438641-kw7df"
Nov 01 03:40:50.717242 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:50.717222    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6346712-2068-4299-90f9-df1fcf150cc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-438641-kw7df\" (UID: \"b6346712-2068-4299-90f9-df1fcf150cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-438641-kw7df"
Nov 01 03:40:50.789337 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:50.789251    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwn22\" (UniqueName: \"kubernetes.io/projected/b6346712-2068-4299-90f9-df1fcf150cc7-kube-api-access-rwn22\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-438641-kw7df\" (UID: \"b6346712-2068-4299-90f9-df1fcf150cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-438641-kw7df"
Nov 01 03:40:50.819724 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:50.819686    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-438641-kw7df"
Nov 01 03:40:50.991068 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:50.990826    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:40:51.153778 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:51.153743    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:40:51.298938 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:51.298915    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-438641-kw7df"]
Nov 01 03:40:51.319921 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:51.319891    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"]
Nov 01 03:40:51.320624 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:51.320236    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz" podUID="ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6" containerName="istio-proxy" containerID="cri-o://505ef1473dac97459c5dddc71051cbf6291897e8c261801c22a904306630ad34" gracePeriod=30
Nov 01 03:40:51.501341 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:51.481116    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:40:51.645837 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:51.645797    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-438641-kw7df" event={"ID":"b6346712-2068-4299-90f9-df1fcf150cc7","Type":"ContainerStarted","Data":"50d61ea8e94014464678b7bfae754febcc293243b07f939831032e3385d705bd"}
Nov 01 03:40:52.655823 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:52.655783    4930 generic.go:356] "Generic (PLEG): container finished" podID="b6346712-2068-4299-90f9-df1fcf150cc7" containerID="a8c532ad7602ce7ca62182dcd377955be598b8bcc319f1dcf2184ecea74dd76c" exitCode=0
Nov 01 03:40:52.656218 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:52.655865    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-438641-kw7df" event={"ID":"b6346712-2068-4299-90f9-df1fcf150cc7","Type":"ContainerDied","Data":"a8c532ad7602ce7ca62182dcd377955be598b8bcc319f1dcf2184ecea74dd76c"}
Nov 01 03:40:52.819664 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:52.819529    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:40:52.908596 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:52.903224    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-634332-btfkd"]
Nov 01 03:40:52.909467 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:52.909436    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-634332-btfkd"
Nov 01 03:40:52.954756 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:52.954727    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-634332-btfkd"]
Nov 01 03:40:53.024647 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:53.024611    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4428f375-da7e-41c1-8e59-e73ff06e76f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-634332-btfkd\" (UID: \"4428f375-da7e-41c1-8e59-e73ff06e76f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-634332-btfkd"
Nov 01 03:40:53.024821 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:53.024752    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4428f375-da7e-41c1-8e59-e73ff06e76f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-634332-btfkd\" (UID: \"4428f375-da7e-41c1-8e59-e73ff06e76f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-634332-btfkd"
Nov 01 03:40:53.024821 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:53.024786    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hwhk\" (UniqueName: \"kubernetes.io/projected/4428f375-da7e-41c1-8e59-e73ff06e76f4-kube-api-access-5hwhk\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-634332-btfkd\" (UID: \"4428f375-da7e-41c1-8e59-e73ff06e76f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-634332-btfkd"
Nov 01 03:40:53.126853 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:53.126820    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4428f375-da7e-41c1-8e59-e73ff06e76f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-634332-btfkd\" (UID: \"4428f375-da7e-41c1-8e59-e73ff06e76f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-634332-btfkd"
Nov 01 03:40:53.127014 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:53.126931    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4428f375-da7e-41c1-8e59-e73ff06e76f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-634332-btfkd\" (UID: \"4428f375-da7e-41c1-8e59-e73ff06e76f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-634332-btfkd"
Nov 01 03:40:53.127014 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:53.126961    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hwhk\" (UniqueName: \"kubernetes.io/projected/4428f375-da7e-41c1-8e59-e73ff06e76f4-kube-api-access-5hwhk\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-634332-btfkd\" (UID: \"4428f375-da7e-41c1-8e59-e73ff06e76f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-634332-btfkd"
Nov 01 03:40:53.127550 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:53.127529    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4428f375-da7e-41c1-8e59-e73ff06e76f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-634332-btfkd\" (UID: \"4428f375-da7e-41c1-8e59-e73ff06e76f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-634332-btfkd"
Nov 01 03:40:53.129474 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:53.129442    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4428f375-da7e-41c1-8e59-e73ff06e76f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-634332-btfkd\" (UID: \"4428f375-da7e-41c1-8e59-e73ff06e76f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-634332-btfkd"
Nov 01 03:40:53.188145 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:53.188095    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hwhk\" (UniqueName: \"kubernetes.io/projected/4428f375-da7e-41c1-8e59-e73ff06e76f4-kube-api-access-5hwhk\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-634332-btfkd\" (UID: \"4428f375-da7e-41c1-8e59-e73ff06e76f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-634332-btfkd"
Nov 01 03:40:53.233420 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:53.233395    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-634332-btfkd"
Nov 01 03:40:53.633398 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:53.633369    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-634332-btfkd"]
Nov 01 03:40:53.636710 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:40:53.636679    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4428f375_da7e_41c1_8e59_e73ff06e76f4.slice/crio-f103c4ca0df7ccd82e5addb509afd0592b1d1de67fd4ec6cb7841f7636d51ca8 WatchSource:0}: Error finding container f103c4ca0df7ccd82e5addb509afd0592b1d1de67fd4ec6cb7841f7636d51ca8: Status 404 returned error can't find the container with id f103c4ca0df7ccd82e5addb509afd0592b1d1de67fd4ec6cb7841f7636d51ca8
Nov 01 03:40:53.666618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:53.666582    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-634332-btfkd" event={"ID":"4428f375-da7e-41c1-8e59-e73ff06e76f4","Type":"ContainerStarted","Data":"f103c4ca0df7ccd82e5addb509afd0592b1d1de67fd4ec6cb7841f7636d51ca8"}
Nov 01 03:40:53.826905 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:53.826880    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-438641-kw7df"
Nov 01 03:40:53.944508 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:53.944427    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6346712-2068-4299-90f9-df1fcf150cc7-kube-api-access-rwn22\" (UniqueName: \"kubernetes.io/projected/b6346712-2068-4299-90f9-df1fcf150cc7-kube-api-access-rwn22\") pod \"b6346712-2068-4299-90f9-df1fcf150cc7\" (UID: \"b6346712-2068-4299-90f9-df1fcf150cc7\") "
Nov 01 03:40:53.944764 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:53.944738    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6346712-2068-4299-90f9-df1fcf150cc7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6346712-2068-4299-90f9-df1fcf150cc7-maas-api-xks-ca\") pod \"b6346712-2068-4299-90f9-df1fcf150cc7\" (UID: \"b6346712-2068-4299-90f9-df1fcf150cc7\") "
Nov 01 03:40:53.945540 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:53.944850    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6346712-2068-4299-90f9-df1fcf150cc7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6346712-2068-4299-90f9-df1fcf150cc7-maas-api-openshift-ca\") pod \"b6346712-2068-4299-90f9-df1fcf150cc7\" (UID: \"b6346712-2068-4299-90f9-df1fcf150cc7\") "
Nov 01 03:40:53.945797 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:53.945762    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6346712-2068-4299-90f9-df1fcf150cc7-maas-api-openshift-ca" pod "b6346712-2068-4299-90f9-df1fcf150cc7" (UID: "b6346712-2068-4299-90f9-df1fcf150cc7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:40:53.947959 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:53.947928    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6346712-2068-4299-90f9-df1fcf150cc7-maas-api-xks-ca" pod "b6346712-2068-4299-90f9-df1fcf150cc7" (UID: "b6346712-2068-4299-90f9-df1fcf150cc7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:40:53.948407 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:53.948381    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6346712-2068-4299-90f9-df1fcf150cc7-kube-api-access-rwn22" pod "b6346712-2068-4299-90f9-df1fcf150cc7" (UID: "b6346712-2068-4299-90f9-df1fcf150cc7"). InnerVolumeSpecName "kube-api-access-rwn22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:40:54.047824 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:54.047787    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6346712-2068-4299-90f9-df1fcf150cc7-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:40:54.047824 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:54.047815    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwn22\" (UniqueName: \"kubernetes.io/projected/b6346712-2068-4299-90f9-df1fcf150cc7-kube-api-access-rwn22\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:40:54.047824 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:54.047825    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6346712-2068-4299-90f9-df1fcf150cc7-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:40:54.676328 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:54.676292    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-438641-kw7df"
Nov 01 03:40:54.676849 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:54.676301    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-438641-kw7df" event={"ID":"b6346712-2068-4299-90f9-df1fcf150cc7","Type":"ContainerDied","Data":"50d61ea8e94014464678b7bfae754febcc293243b07f939831032e3385d705bd"}
Nov 01 03:40:54.676849 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:54.676441    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50d61ea8e94014464678b7bfae754febcc293243b07f939831032e3385d705bd"
Nov 01 03:40:54.685301 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:54.685270    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-634332-btfkd" event={"ID":"4428f375-da7e-41c1-8e59-e73ff06e76f4","Type":"ContainerStarted","Data":"de58bc85f6daad73d7be66a4261d110f37e0ad8df5ffa8e1e0ef5d26ec707018"}
Nov 01 03:40:54.807079 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:54.807007    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-634332-btfkd" podStartSLOduration=2.806984463 podStartE2EDuration="2.806984463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:40:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:40:54.800348288 +0000 UTC m=+2059.323354878" watchObservedRunningTime="2026-11-01 03:40:54.806984463 +0000 UTC m=+2059.329991054"
Nov 01 03:40:55.035313 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.035286    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"
Nov 01 03:40:55.065853 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.065815    4930 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:40:55.066028 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.065883    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz" podUID="ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:40:55.180510 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.180468    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4cae562-15e5-4d0b-bcef-62ca774e2369-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4cae562-15e5-4d0b-bcef-62ca774e2369-workload-certs\") pod \"a4cae562-15e5-4d0b-bcef-62ca774e2369\" (UID: \"a4cae562-15e5-4d0b-bcef-62ca774e2369\") "
Nov 01 03:40:55.180725 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.180539    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4cae562-15e5-4d0b-bcef-62ca774e2369-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4cae562-15e5-4d0b-bcef-62ca774e2369-istiod-ca-cert\") pod \"a4cae562-15e5-4d0b-bcef-62ca774e2369\" (UID: \"a4cae562-15e5-4d0b-bcef-62ca774e2369\") "
Nov 01 03:40:55.180804 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.180731    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4cae562-15e5-4d0b-bcef-62ca774e2369-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4cae562-15e5-4d0b-bcef-62ca774e2369-workload-socket\") pod \"a4cae562-15e5-4d0b-bcef-62ca774e2369\" (UID: \"a4cae562-15e5-4d0b-bcef-62ca774e2369\") "
Nov 01 03:40:55.180900 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.180870    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4cae562-15e5-4d0b-bcef-62ca774e2369-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4cae562-15e5-4d0b-bcef-62ca774e2369-istio-data\") pod \"a4cae562-15e5-4d0b-bcef-62ca774e2369\" (UID: \"a4cae562-15e5-4d0b-bcef-62ca774e2369\") "
Nov 01 03:40:55.180967 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.180915    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4cae562-15e5-4d0b-bcef-62ca774e2369-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4cae562-15e5-4d0b-bcef-62ca774e2369-istio-podinfo\") pod \"a4cae562-15e5-4d0b-bcef-62ca774e2369\" (UID: \"a4cae562-15e5-4d0b-bcef-62ca774e2369\") "
Nov 01 03:40:55.181027 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.180984    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4cae562-15e5-4d0b-bcef-62ca774e2369-istio-token\" (UniqueName: \"kubernetes.io/projected/a4cae562-15e5-4d0b-bcef-62ca774e2369-istio-token\") pod \"a4cae562-15e5-4d0b-bcef-62ca774e2369\" (UID: \"a4cae562-15e5-4d0b-bcef-62ca774e2369\") "
Nov 01 03:40:55.181027 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.181020    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4cae562-15e5-4d0b-bcef-62ca774e2369-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4cae562-15e5-4d0b-bcef-62ca774e2369-istio-envoy\") pod \"a4cae562-15e5-4d0b-bcef-62ca774e2369\" (UID: \"a4cae562-15e5-4d0b-bcef-62ca774e2369\") "
Nov 01 03:40:55.181139 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.181051    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4cae562-15e5-4d0b-bcef-62ca774e2369-kube-api-access-nrtjk\" (UniqueName: \"kubernetes.io/projected/a4cae562-15e5-4d0b-bcef-62ca774e2369-kube-api-access-nrtjk\") pod \"a4cae562-15e5-4d0b-bcef-62ca774e2369\" (UID: \"a4cae562-15e5-4d0b-bcef-62ca774e2369\") "
Nov 01 03:40:55.181224 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.181205    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4cae562-15e5-4d0b-bcef-62ca774e2369-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4cae562-15e5-4d0b-bcef-62ca774e2369-credential-socket\") pod \"a4cae562-15e5-4d0b-bcef-62ca774e2369\" (UID: \"a4cae562-15e5-4d0b-bcef-62ca774e2369\") "
Nov 01 03:40:55.182458 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.182429    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4cae562-15e5-4d0b-bcef-62ca774e2369-istio-data" pod "a4cae562-15e5-4d0b-bcef-62ca774e2369" (UID: "a4cae562-15e5-4d0b-bcef-62ca774e2369"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:40:55.183215 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.182690    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4cae562-15e5-4d0b-bcef-62ca774e2369-workload-certs" pod "a4cae562-15e5-4d0b-bcef-62ca774e2369" (UID: "a4cae562-15e5-4d0b-bcef-62ca774e2369"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:40:55.183215 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.182809    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4cae562-15e5-4d0b-bcef-62ca774e2369-credential-socket" pod "a4cae562-15e5-4d0b-bcef-62ca774e2369" (UID: "a4cae562-15e5-4d0b-bcef-62ca774e2369"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:40:55.183746 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.183518    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4cae562-15e5-4d0b-bcef-62ca774e2369-workload-socket" pod "a4cae562-15e5-4d0b-bcef-62ca774e2369" (UID: "a4cae562-15e5-4d0b-bcef-62ca774e2369"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:40:55.183746 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.183708    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4cae562-15e5-4d0b-bcef-62ca774e2369-istiod-ca-cert" pod "a4cae562-15e5-4d0b-bcef-62ca774e2369" (UID: "a4cae562-15e5-4d0b-bcef-62ca774e2369"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:40:55.186446 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.186418    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4cae562-15e5-4d0b-bcef-62ca774e2369-kube-api-access-nrtjk" pod "a4cae562-15e5-4d0b-bcef-62ca774e2369" (UID: "a4cae562-15e5-4d0b-bcef-62ca774e2369"). InnerVolumeSpecName "kube-api-access-nrtjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:40:55.187021 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.186922    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4cae562-15e5-4d0b-bcef-62ca774e2369-istio-token" pod "a4cae562-15e5-4d0b-bcef-62ca774e2369" (UID: "a4cae562-15e5-4d0b-bcef-62ca774e2369"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:40:55.189119 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.189084    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4cae562-15e5-4d0b-bcef-62ca774e2369-istio-podinfo" pod "a4cae562-15e5-4d0b-bcef-62ca774e2369" (UID: "a4cae562-15e5-4d0b-bcef-62ca774e2369"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:40:55.189228 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.189118    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4cae562-15e5-4d0b-bcef-62ca774e2369-istio-envoy" pod "a4cae562-15e5-4d0b-bcef-62ca774e2369" (UID: "a4cae562-15e5-4d0b-bcef-62ca774e2369"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:40:55.283389 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.283302    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4cae562-15e5-4d0b-bcef-62ca774e2369-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:40:55.283389 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.283335    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4cae562-15e5-4d0b-bcef-62ca774e2369-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:40:55.283389 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.283345    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4cae562-15e5-4d0b-bcef-62ca774e2369-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:40:55.283389 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.283354    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4cae562-15e5-4d0b-bcef-62ca774e2369-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:40:55.283389 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.283363    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4cae562-15e5-4d0b-bcef-62ca774e2369-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:40:55.283389 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.283371    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4cae562-15e5-4d0b-bcef-62ca774e2369-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:40:55.283389 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.283379    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4cae562-15e5-4d0b-bcef-62ca774e2369-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:40:55.283389 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.283388    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrtjk\" (UniqueName: \"kubernetes.io/projected/a4cae562-15e5-4d0b-bcef-62ca774e2369-kube-api-access-nrtjk\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:40:55.283389 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.283397    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4cae562-15e5-4d0b-bcef-62ca774e2369-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:40:55.695200 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.695157    4930 generic.go:356] "Generic (PLEG): container finished" podID="a4cae562-15e5-4d0b-bcef-62ca774e2369" containerID="f94ccf4edcf71b1b502e4937f17e375337dffceb203a49f96738a5857c35eeb4" exitCode=0
Nov 01 03:40:55.695689 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.695219    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"
Nov 01 03:40:55.695689 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.695251    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh" event={"ID":"a4cae562-15e5-4d0b-bcef-62ca774e2369","Type":"ContainerDied","Data":"f94ccf4edcf71b1b502e4937f17e375337dffceb203a49f96738a5857c35eeb4"}
Nov 01 03:40:55.695689 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.695316    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh" event={"ID":"a4cae562-15e5-4d0b-bcef-62ca774e2369","Type":"ContainerDied","Data":"3158af2c7942a3a4bd0f801909f024c67da30e6e6a6d51d25d84e405cef82451"}
Nov 01 03:40:55.695689 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.695350    4930 scope.go:122] "RemoveContainer" containerID="f94ccf4edcf71b1b502e4937f17e375337dffceb203a49f96738a5857c35eeb4"
Nov 01 03:40:55.697941 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.697915    4930 generic.go:356] "Generic (PLEG): container finished" podID="4428f375-da7e-41c1-8e59-e73ff06e76f4" containerID="de58bc85f6daad73d7be66a4261d110f37e0ad8df5ffa8e1e0ef5d26ec707018" exitCode=0
Nov 01 03:40:55.698082 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.697914    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-634332-btfkd" event={"ID":"4428f375-da7e-41c1-8e59-e73ff06e76f4","Type":"ContainerDied","Data":"de58bc85f6daad73d7be66a4261d110f37e0ad8df5ffa8e1e0ef5d26ec707018"}
Nov 01 03:40:55.707499 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.707482    4930 scope.go:122] "RemoveContainer" containerID="de5005bcce6c56f7eea7f9d783b30dcf09982aa9a56cc44a65a2306b807a57be"
Nov 01 03:40:55.713681 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.713658    4930 patch_prober.go:29] interesting pod/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 03:40:55.713798 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.713700    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp" podUID="cbc558d1-0eff-46fb-a780-7ebcc1939928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:40:55.718198 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.718172    4930 scope.go:122] "RemoveContainer" containerID="f94ccf4edcf71b1b502e4937f17e375337dffceb203a49f96738a5857c35eeb4"
Nov 01 03:40:55.718468 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:40:55.718447    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f94ccf4edcf71b1b502e4937f17e375337dffceb203a49f96738a5857c35eeb4\": container with ID starting with f94ccf4edcf71b1b502e4937f17e375337dffceb203a49f96738a5857c35eeb4 not found: ID does not exist" containerID="f94ccf4edcf71b1b502e4937f17e375337dffceb203a49f96738a5857c35eeb4"
Nov 01 03:40:55.718540 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.718475    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f94ccf4edcf71b1b502e4937f17e375337dffceb203a49f96738a5857c35eeb4"} err="failed to get container status \"f94ccf4edcf71b1b502e4937f17e375337dffceb203a49f96738a5857c35eeb4\": rpc error: code = NotFound desc = could not find container \"f94ccf4edcf71b1b502e4937f17e375337dffceb203a49f96738a5857c35eeb4\": container with ID starting with f94ccf4edcf71b1b502e4937f17e375337dffceb203a49f96738a5857c35eeb4 not found: ID does not exist"
Nov 01 03:40:55.718540 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.718491    4930 scope.go:122] "RemoveContainer" containerID="de5005bcce6c56f7eea7f9d783b30dcf09982aa9a56cc44a65a2306b807a57be"
Nov 01 03:40:55.718803 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:40:55.718782    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de5005bcce6c56f7eea7f9d783b30dcf09982aa9a56cc44a65a2306b807a57be\": container with ID starting with de5005bcce6c56f7eea7f9d783b30dcf09982aa9a56cc44a65a2306b807a57be not found: ID does not exist" containerID="de5005bcce6c56f7eea7f9d783b30dcf09982aa9a56cc44a65a2306b807a57be"
Nov 01 03:40:55.718850 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.718810    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de5005bcce6c56f7eea7f9d783b30dcf09982aa9a56cc44a65a2306b807a57be"} err="failed to get container status \"de5005bcce6c56f7eea7f9d783b30dcf09982aa9a56cc44a65a2306b807a57be\": rpc error: code = NotFound desc = could not find container \"de5005bcce6c56f7eea7f9d783b30dcf09982aa9a56cc44a65a2306b807a57be\": container with ID starting with de5005bcce6c56f7eea7f9d783b30dcf09982aa9a56cc44a65a2306b807a57be not found: ID does not exist"
Nov 01 03:40:55.866827 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.866787    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"]
Nov 01 03:40:55.914281 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:55.914200    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-e8c77d-openshift-default-6b788968f9-78mjh"]
Nov 01 03:40:56.707018 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:56.706983    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"
Nov 01 03:40:56.707470 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:56.707445    4930 generic.go:356] "Generic (PLEG): container finished" podID="ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6" containerID="505ef1473dac97459c5dddc71051cbf6291897e8c261801c22a904306630ad34" exitCode=0
Nov 01 03:40:56.707567 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:56.707528    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz" event={"ID":"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6","Type":"ContainerDied","Data":"505ef1473dac97459c5dddc71051cbf6291897e8c261801c22a904306630ad34"}
Nov 01 03:40:56.707643 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:56.707606    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz" event={"ID":"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6","Type":"ContainerDied","Data":"82e5619b8968817fabfffb49ccf3c7a50bbf046b62506a2237bde8677d9acb92"}
Nov 01 03:40:56.707768 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:56.707642    4930 scope.go:122] "RemoveContainer" containerID="505ef1473dac97459c5dddc71051cbf6291897e8c261801c22a904306630ad34"
Nov 01 03:40:56.719457 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:56.719396    4930 scope.go:122] "RemoveContainer" containerID="34dc61e7d67b7c666d5691fecdf5db6b2f65960eaa7d26ce8c08582ae6e53bb6"
Nov 01 03:40:56.744106 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:56.744079    4930 scope.go:122] "RemoveContainer" containerID="505ef1473dac97459c5dddc71051cbf6291897e8c261801c22a904306630ad34"
Nov 01 03:40:56.744614 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:40:56.744586    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"505ef1473dac97459c5dddc71051cbf6291897e8c261801c22a904306630ad34\": container with ID starting with 505ef1473dac97459c5dddc71051cbf6291897e8c261801c22a904306630ad34 not found: ID does not exist" containerID="505ef1473dac97459c5dddc71051cbf6291897e8c261801c22a904306630ad34"
Nov 01 03:40:56.744729 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:56.744626    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"505ef1473dac97459c5dddc71051cbf6291897e8c261801c22a904306630ad34"} err="failed to get container status \"505ef1473dac97459c5dddc71051cbf6291897e8c261801c22a904306630ad34\": rpc error: code = NotFound desc = could not find container \"505ef1473dac97459c5dddc71051cbf6291897e8c261801c22a904306630ad34\": container with ID starting with 505ef1473dac97459c5dddc71051cbf6291897e8c261801c22a904306630ad34 not found: ID does not exist"
Nov 01 03:40:56.744729 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:56.744649    4930 scope.go:122] "RemoveContainer" containerID="34dc61e7d67b7c666d5691fecdf5db6b2f65960eaa7d26ce8c08582ae6e53bb6"
Nov 01 03:40:56.744981 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:40:56.744942    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34dc61e7d67b7c666d5691fecdf5db6b2f65960eaa7d26ce8c08582ae6e53bb6\": container with ID starting with 34dc61e7d67b7c666d5691fecdf5db6b2f65960eaa7d26ce8c08582ae6e53bb6 not found: ID does not exist" containerID="34dc61e7d67b7c666d5691fecdf5db6b2f65960eaa7d26ce8c08582ae6e53bb6"
Nov 01 03:40:56.745051 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:56.744987    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34dc61e7d67b7c666d5691fecdf5db6b2f65960eaa7d26ce8c08582ae6e53bb6"} err="failed to get container status \"34dc61e7d67b7c666d5691fecdf5db6b2f65960eaa7d26ce8c08582ae6e53bb6\": rpc error: code = NotFound desc = could not find container \"34dc61e7d67b7c666d5691fecdf5db6b2f65960eaa7d26ce8c08582ae6e53bb6\": container with ID starting with 34dc61e7d67b7c666d5691fecdf5db6b2f65960eaa7d26ce8c08582ae6e53bb6 not found: ID does not exist"
Nov 01 03:40:56.805738 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:56.805687    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-kube-api-access-j42l5\" (UniqueName: \"kubernetes.io/projected/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-kube-api-access-j42l5\") pod \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\" (UID: \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\") "
Nov 01 03:40:56.805949 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:56.805747    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-workload-certs\") pod \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\" (UID: \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\") "
Nov 01 03:40:56.805949 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:56.805813    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-istio-podinfo\") pod \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\" (UID: \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\") "
Nov 01 03:40:56.805949 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:56.805893    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-istio-envoy\") pod \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\" (UID: \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\") "
Nov 01 03:40:56.805949 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:56.805931    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-istio-data\") pod \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\" (UID: \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\") "
Nov 01 03:40:56.806414 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:56.806050    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-workload-certs" pod "ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6" (UID: "ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:40:56.806414 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:56.806068    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-credential-socket\") pod \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\" (UID: \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\") "
Nov 01 03:40:56.806414 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:56.806240    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-istiod-ca-cert\") pod \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\" (UID: \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\") "
Nov 01 03:40:56.806414 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:56.806310    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-istio-token\" (UniqueName: \"kubernetes.io/projected/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-istio-token\") pod \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\" (UID: \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\") "
Nov 01 03:40:56.807596 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:56.806421    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-workload-socket\") pod \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\" (UID: \"ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6\") "
Nov 01 03:40:56.807596 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:56.807199    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:40:56.807596 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:56.807253    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-credential-socket" pod "ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6" (UID: "ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:40:56.807596 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:56.807290    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-workload-socket" pod "ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6" (UID: "ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:40:56.807596 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:56.807306    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-istiod-ca-cert" pod "ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6" (UID: "ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:40:56.807596 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:56.807362    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-istio-data" pod "ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6" (UID: "ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:40:56.810106 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:56.810082    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-istio-envoy" pod "ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6" (UID: "ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:40:56.817861 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:56.817825    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-istio-podinfo" pod "ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6" (UID: "ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:40:56.818054 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:56.818019    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-kube-api-access-j42l5" pod "ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6" (UID: "ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6"). InnerVolumeSpecName "kube-api-access-j42l5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:40:56.820386 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:56.820353    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-istio-token" pod "ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6" (UID: "ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:40:56.908829 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:56.908755    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:40:56.908829 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:56.908783    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:40:56.908829 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:56.908793    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:40:56.908829 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:56.908802    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:40:56.908829 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:56.908811    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:40:56.908829 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:56.908820    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:40:56.908829 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:56.908828    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j42l5\" (UniqueName: \"kubernetes.io/projected/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-kube-api-access-j42l5\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:40:56.909220 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:56.908838    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:40:56.918496 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:56.918465    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-634332-btfkd"
Nov 01 03:40:57.009856 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:57.009823    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4428f375-da7e-41c1-8e59-e73ff06e76f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4428f375-da7e-41c1-8e59-e73ff06e76f4-maas-api-openshift-ca\") pod \"4428f375-da7e-41c1-8e59-e73ff06e76f4\" (UID: \"4428f375-da7e-41c1-8e59-e73ff06e76f4\") "
Nov 01 03:40:57.010058 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:57.010001    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4428f375-da7e-41c1-8e59-e73ff06e76f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4428f375-da7e-41c1-8e59-e73ff06e76f4-maas-api-xks-ca\") pod \"4428f375-da7e-41c1-8e59-e73ff06e76f4\" (UID: \"4428f375-da7e-41c1-8e59-e73ff06e76f4\") "
Nov 01 03:40:57.010127 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:57.010080    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4428f375-da7e-41c1-8e59-e73ff06e76f4-kube-api-access-5hwhk\" (UniqueName: \"kubernetes.io/projected/4428f375-da7e-41c1-8e59-e73ff06e76f4-kube-api-access-5hwhk\") pod \"4428f375-da7e-41c1-8e59-e73ff06e76f4\" (UID: \"4428f375-da7e-41c1-8e59-e73ff06e76f4\") "
Nov 01 03:40:57.010257 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:57.010228    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4428f375-da7e-41c1-8e59-e73ff06e76f4-maas-api-openshift-ca" pod "4428f375-da7e-41c1-8e59-e73ff06e76f4" (UID: "4428f375-da7e-41c1-8e59-e73ff06e76f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:40:57.010736 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:57.010712    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4428f375-da7e-41c1-8e59-e73ff06e76f4-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:40:57.012155 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:57.012127    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4428f375-da7e-41c1-8e59-e73ff06e76f4-maas-api-xks-ca" pod "4428f375-da7e-41c1-8e59-e73ff06e76f4" (UID: "4428f375-da7e-41c1-8e59-e73ff06e76f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:40:57.012251 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:57.012173    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4428f375-da7e-41c1-8e59-e73ff06e76f4-kube-api-access-5hwhk" pod "4428f375-da7e-41c1-8e59-e73ff06e76f4" (UID: "4428f375-da7e-41c1-8e59-e73ff06e76f4"). InnerVolumeSpecName "kube-api-access-5hwhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:40:57.112079 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:57.112048    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4428f375-da7e-41c1-8e59-e73ff06e76f4-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:40:57.112940 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:57.112082    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hwhk\" (UniqueName: \"kubernetes.io/projected/4428f375-da7e-41c1-8e59-e73ff06e76f4-kube-api-access-5hwhk\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:40:57.623001 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:57.622968    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4cae562-15e5-4d0b-bcef-62ca774e2369" path="/var/lib/kubelet/pods/a4cae562-15e5-4d0b-bcef-62ca774e2369/volumes"
Nov 01 03:40:57.718367 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:57.718338    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"
Nov 01 03:40:57.720251 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:57.720226    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-634332-btfkd"
Nov 01 03:40:57.720392 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:57.720246    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-634332-btfkd" event={"ID":"4428f375-da7e-41c1-8e59-e73ff06e76f4","Type":"ContainerDied","Data":"f103c4ca0df7ccd82e5addb509afd0592b1d1de67fd4ec6cb7841f7636d51ca8"}
Nov 01 03:40:57.720392 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:57.720299    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f103c4ca0df7ccd82e5addb509afd0592b1d1de67fd4ec6cb7841f7636d51ca8"
Nov 01 03:40:57.722380 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:57.722352    4930 generic.go:356] "Generic (PLEG): container finished" podID="cbc558d1-0eff-46fb-a780-7ebcc1939928" containerID="9081f00c6971d46de11917b10aa3ec83e72de04f3eea18d8d3d871a37934e8af" exitCode=0
Nov 01 03:40:57.722530 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:57.722423    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp" event={"ID":"cbc558d1-0eff-46fb-a780-7ebcc1939928","Type":"ContainerDied","Data":"9081f00c6971d46de11917b10aa3ec83e72de04f3eea18d8d3d871a37934e8af"}
Nov 01 03:40:57.722916 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:57.722896    4930 scope.go:122] "RemoveContainer" containerID="9081f00c6971d46de11917b10aa3ec83e72de04f3eea18d8d3d871a37934e8af"
Nov 01 03:40:57.724748 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:57.724715    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:40:57.724850 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:57.724805    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:40:57.724900 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:57.724849    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:40:57.885944 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:57.885858    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"]
Nov 01 03:40:57.894407 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:57.894375    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0cb10b-openshift-default-6bcfff965-x4zhz"]
Nov 01 03:40:58.035262 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:58.035227    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:40:58.733980 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:58.733941    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp" event={"ID":"cbc558d1-0eff-46fb-a780-7ebcc1939928","Type":"ContainerStarted","Data":"c5099fb040f8f6428c70efc92e151d1df3a217145c12039d9916fcd19e52f068"}
Nov 01 03:40:59.183830 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:59.183793    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"
Nov 01 03:40:59.184240 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:59.184207    4930 patch_prober.go:29] interesting pod/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Nov 01 03:40:59.184347 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:59.184265    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp" podUID="cbc558d1-0eff-46fb-a780-7ebcc1939928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Nov 01 03:40:59.623058 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:40:59.623022    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6" path="/var/lib/kubelet/pods/ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6/volumes"
Nov 01 03:41:00.174581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:00.170917    4930 patch_prober.go:29] interesting pod/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Nov 01 03:41:00.174581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:00.170976    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp" podUID="cbc558d1-0eff-46fb-a780-7ebcc1939928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Nov 01 03:41:01.174587 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:01.170644    4930 patch_prober.go:29] interesting pod/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Nov 01 03:41:01.174587 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:01.170701    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp" podUID="cbc558d1-0eff-46fb-a780-7ebcc1939928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Nov 01 03:41:02.089583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:02.089369    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-438641-57f787c447-mk9mn"]
Nov 01 03:41:02.175575 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:02.174064    4930 patch_prober.go:29] interesting pod/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Nov 01 03:41:02.175575 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:02.174140    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp" podUID="cbc558d1-0eff-46fb-a780-7ebcc1939928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Nov 01 03:41:02.210701 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:02.210655    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-438641-668fd65b59-wmqpw"]
Nov 01 03:41:02.428778 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:02.428698    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:02.654488 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:02.641611    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:03.097167 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:03.097107    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:03.169579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:03.168949    4930 patch_prober.go:29] interesting pod/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Nov 01 03:41:03.169579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:03.169014    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp" podUID="cbc558d1-0eff-46fb-a780-7ebcc1939928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Nov 01 03:41:03.239402 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:03.239316    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:03.431632 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:03.431511    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:04.272232 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:04.272194    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"
Nov 01 03:41:04.273519 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:04.273492    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"
Nov 01 03:41:04.274893 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:04.274869    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"
Nov 01 03:41:06.049635 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:06.049603    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0d70ec1-2kwvl"]
Nov 01 03:41:06.099167 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:06.099138    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0d70ec1-2kwvl"]
Nov 01 03:41:07.090179 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:07.090136    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-438641-57f787c447-mk9mn" podUID="463ab8cc-0751-4fcf-a387-4b85b9f09180" containerName="payload-processing" containerID="cri-o://08b9d41ffd65261958fce7198ed8a709790d38e4e5abc0e54252c5e96bfa531d" gracePeriod=25
Nov 01 03:41:07.211029 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:07.210998    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-438641-668fd65b59-wmqpw" podUID="5974826d-4ad8-4b1e-b2ad-322b8d01359e" containerName="payload-pre-processing" containerID="cri-o://41b6a0e5ca29458b9fcfd80c9de4d75951ea4990e6a5911e98ac6756753af89f" gracePeriod=25
Nov 01 03:41:07.317808 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:07.317754    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:07.447359 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:07.447331    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-438641-57f787c447-mk9mn"
Nov 01 03:41:07.517009 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:07.516952    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:07.536796 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:07.536771    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/463ab8cc-0751-4fcf-a387-4b85b9f09180-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/463ab8cc-0751-4fcf-a387-4b85b9f09180-plugins-config-volume\") pod \"463ab8cc-0751-4fcf-a387-4b85b9f09180\" (UID: \"463ab8cc-0751-4fcf-a387-4b85b9f09180\") "
Nov 01 03:41:07.536971 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:07.536812    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/463ab8cc-0751-4fcf-a387-4b85b9f09180-kube-api-access-qrff4\" (UniqueName: \"kubernetes.io/projected/463ab8cc-0751-4fcf-a387-4b85b9f09180-kube-api-access-qrff4\") pod \"463ab8cc-0751-4fcf-a387-4b85b9f09180\" (UID: \"463ab8cc-0751-4fcf-a387-4b85b9f09180\") "
Nov 01 03:41:07.537151 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:07.537126    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/463ab8cc-0751-4fcf-a387-4b85b9f09180-plugins-config-volume" pod "463ab8cc-0751-4fcf-a387-4b85b9f09180" (UID: "463ab8cc-0751-4fcf-a387-4b85b9f09180"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:41:07.537347 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:07.537332    4930 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/463ab8cc-0751-4fcf-a387-4b85b9f09180-plugins-config-volume\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:41:07.539052 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:07.539021    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/463ab8cc-0751-4fcf-a387-4b85b9f09180-kube-api-access-qrff4" pod "463ab8cc-0751-4fcf-a387-4b85b9f09180" (UID: "463ab8cc-0751-4fcf-a387-4b85b9f09180"). InnerVolumeSpecName "kube-api-access-qrff4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:41:07.640170 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:07.640132    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrff4\" (UniqueName: \"kubernetes.io/projected/463ab8cc-0751-4fcf-a387-4b85b9f09180-kube-api-access-qrff4\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:41:07.643310 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:07.643283    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-438641-668fd65b59-wmqpw"
Nov 01 03:41:07.652132 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:07.652097    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d15cd3ef-c4ab-4b6a-bb53-61d3557b92de" path="/var/lib/kubelet/pods/d15cd3ef-c4ab-4b6a-bb53-61d3557b92de/volumes"
Nov 01 03:41:07.750584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:07.744031    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5974826d-4ad8-4b1e-b2ad-322b8d01359e-kube-api-access-h92m2\" (UniqueName: \"kubernetes.io/projected/5974826d-4ad8-4b1e-b2ad-322b8d01359e-kube-api-access-h92m2\") pod \"5974826d-4ad8-4b1e-b2ad-322b8d01359e\" (UID: \"5974826d-4ad8-4b1e-b2ad-322b8d01359e\") "
Nov 01 03:41:07.754576 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:07.752583    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5974826d-4ad8-4b1e-b2ad-322b8d01359e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5974826d-4ad8-4b1e-b2ad-322b8d01359e-plugins-config-volume\") pod \"5974826d-4ad8-4b1e-b2ad-322b8d01359e\" (UID: \"5974826d-4ad8-4b1e-b2ad-322b8d01359e\") "
Nov 01 03:41:07.767654 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:07.767615    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5974826d-4ad8-4b1e-b2ad-322b8d01359e-plugins-config-volume" pod "5974826d-4ad8-4b1e-b2ad-322b8d01359e" (UID: "5974826d-4ad8-4b1e-b2ad-322b8d01359e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:41:07.814805 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:07.814745    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5974826d-4ad8-4b1e-b2ad-322b8d01359e-kube-api-access-h92m2" pod "5974826d-4ad8-4b1e-b2ad-322b8d01359e" (UID: "5974826d-4ad8-4b1e-b2ad-322b8d01359e"). InnerVolumeSpecName "kube-api-access-h92m2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:41:07.876347 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:07.876239    4930 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5974826d-4ad8-4b1e-b2ad-322b8d01359e-plugins-config-volume\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:41:07.876347 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:07.876279    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h92m2\" (UniqueName: \"kubernetes.io/projected/5974826d-4ad8-4b1e-b2ad-322b8d01359e-kube-api-access-h92m2\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:41:08.012314 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:08.012217    4930 generic.go:356] "Generic (PLEG): container finished" podID="5974826d-4ad8-4b1e-b2ad-322b8d01359e" containerID="41b6a0e5ca29458b9fcfd80c9de4d75951ea4990e6a5911e98ac6756753af89f" exitCode=0
Nov 01 03:41:08.012314 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:08.012269    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-438641-668fd65b59-wmqpw"
Nov 01 03:41:08.012314 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:08.012291    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-438641-668fd65b59-wmqpw" event={"ID":"5974826d-4ad8-4b1e-b2ad-322b8d01359e","Type":"ContainerDied","Data":"41b6a0e5ca29458b9fcfd80c9de4d75951ea4990e6a5911e98ac6756753af89f"}
Nov 01 03:41:08.012619 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:08.012361    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-438641-668fd65b59-wmqpw" event={"ID":"5974826d-4ad8-4b1e-b2ad-322b8d01359e","Type":"ContainerDied","Data":"d859cd74100fed0af0e6ebd21107a4e76820f406eec26735c36680946f608e9d"}
Nov 01 03:41:08.012619 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:08.012386    4930 scope.go:122] "RemoveContainer" containerID="41b6a0e5ca29458b9fcfd80c9de4d75951ea4990e6a5911e98ac6756753af89f"
Nov 01 03:41:08.014167 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:08.014146    4930 generic.go:356] "Generic (PLEG): container finished" podID="463ab8cc-0751-4fcf-a387-4b85b9f09180" containerID="08b9d41ffd65261958fce7198ed8a709790d38e4e5abc0e54252c5e96bfa531d" exitCode=0
Nov 01 03:41:08.014286 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:08.014199    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-438641-57f787c447-mk9mn"
Nov 01 03:41:08.014355 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:08.014198    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-438641-57f787c447-mk9mn" event={"ID":"463ab8cc-0751-4fcf-a387-4b85b9f09180","Type":"ContainerDied","Data":"08b9d41ffd65261958fce7198ed8a709790d38e4e5abc0e54252c5e96bfa531d"}
Nov 01 03:41:08.014415 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:08.014348    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-438641-57f787c447-mk9mn" event={"ID":"463ab8cc-0751-4fcf-a387-4b85b9f09180","Type":"ContainerDied","Data":"78deddc28612dc709b47d5367deb7cdb007be44fa829697472339f86502a2bac"}
Nov 01 03:41:08.025252 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:08.025237    4930 scope.go:122] "RemoveContainer" containerID="41b6a0e5ca29458b9fcfd80c9de4d75951ea4990e6a5911e98ac6756753af89f"
Nov 01 03:41:08.025525 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:41:08.025505    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41b6a0e5ca29458b9fcfd80c9de4d75951ea4990e6a5911e98ac6756753af89f\": container with ID starting with 41b6a0e5ca29458b9fcfd80c9de4d75951ea4990e6a5911e98ac6756753af89f not found: ID does not exist" containerID="41b6a0e5ca29458b9fcfd80c9de4d75951ea4990e6a5911e98ac6756753af89f"
Nov 01 03:41:08.025589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:08.025531    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41b6a0e5ca29458b9fcfd80c9de4d75951ea4990e6a5911e98ac6756753af89f"} err="failed to get container status \"41b6a0e5ca29458b9fcfd80c9de4d75951ea4990e6a5911e98ac6756753af89f\": rpc error: code = NotFound desc = could not find container \"41b6a0e5ca29458b9fcfd80c9de4d75951ea4990e6a5911e98ac6756753af89f\": container with ID starting with 41b6a0e5ca29458b9fcfd80c9de4d75951ea4990e6a5911e98ac6756753af89f not found: ID does not exist"
Nov 01 03:41:08.025589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:08.025545    4930 scope.go:122] "RemoveContainer" containerID="08b9d41ffd65261958fce7198ed8a709790d38e4e5abc0e54252c5e96bfa531d"
Nov 01 03:41:08.038501 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:08.038478    4930 scope.go:122] "RemoveContainer" containerID="08b9d41ffd65261958fce7198ed8a709790d38e4e5abc0e54252c5e96bfa531d"
Nov 01 03:41:08.038892 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:41:08.038872    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08b9d41ffd65261958fce7198ed8a709790d38e4e5abc0e54252c5e96bfa531d\": container with ID starting with 08b9d41ffd65261958fce7198ed8a709790d38e4e5abc0e54252c5e96bfa531d not found: ID does not exist" containerID="08b9d41ffd65261958fce7198ed8a709790d38e4e5abc0e54252c5e96bfa531d"
Nov 01 03:41:08.038966 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:08.038898    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08b9d41ffd65261958fce7198ed8a709790d38e4e5abc0e54252c5e96bfa531d"} err="failed to get container status \"08b9d41ffd65261958fce7198ed8a709790d38e4e5abc0e54252c5e96bfa531d\": rpc error: code = NotFound desc = could not find container \"08b9d41ffd65261958fce7198ed8a709790d38e4e5abc0e54252c5e96bfa531d\": container with ID starting with 08b9d41ffd65261958fce7198ed8a709790d38e4e5abc0e54252c5e96bfa531d not found: ID does not exist"
Nov 01 03:41:08.182412 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:08.182381    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-438641-668fd65b59-wmqpw"]
Nov 01 03:41:08.228423 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:08.228395    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-438641-668fd65b59-wmqpw"]
Nov 01 03:41:08.365717 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:08.365685    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-438641-57f787c447-mk9mn"]
Nov 01 03:41:08.441304 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:08.441264    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-438641-57f787c447-mk9mn"]
Nov 01 03:41:09.189046 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:09.189007    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:09.622594 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:09.622539    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="463ab8cc-0751-4fcf-a387-4b85b9f09180" path="/var/lib/kubelet/pods/463ab8cc-0751-4fcf-a387-4b85b9f09180/volumes"
Nov 01 03:41:09.623081 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:09.623066    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5974826d-4ad8-4b1e-b2ad-322b8d01359e" path="/var/lib/kubelet/pods/5974826d-4ad8-4b1e-b2ad-322b8d01359e/volumes"
Nov 01 03:41:13.810747 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:13.810704    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:14.333798 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:14.333763    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-634332-btfkd"]
Nov 01 03:41:14.382906 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:14.382871    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-634332-btfkd"]
Nov 01 03:41:15.058621 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:15.055105    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:15.621743 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:15.621708    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4428f375-da7e-41c1-8e59-e73ff06e76f4" path="/var/lib/kubelet/pods/4428f375-da7e-41c1-8e59-e73ff06e76f4/volumes"
Nov 01 03:41:19.125395 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:41:19.125350    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:41:19.294906 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:19.294868    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:21.494886 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:21.494840    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:21.896108 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:21.896068    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:22.062877 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:22.062838    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:22.696270 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:22.696234    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:23.587975 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.587942    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"]
Nov 01 03:41:23.588641 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.588623    4930 cpu_manager.go:414] "containerMap: removing container" podUID="ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6" containerName="istio-proxy"
Nov 01 03:41:23.588696 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.588643    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6" containerName="istio-proxy"
Nov 01 03:41:23.588696 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.588657    4930 cpu_manager.go:414] "containerMap: removing container" podUID="b6346712-2068-4299-90f9-df1fcf150cc7" containerName="revoke-keys"
Nov 01 03:41:23.588696 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.588663    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6346712-2068-4299-90f9-df1fcf150cc7" containerName="revoke-keys"
Nov 01 03:41:23.588696 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.588673    4930 cpu_manager.go:414] "containerMap: removing container" podUID="a4cae562-15e5-4d0b-bcef-62ca774e2369" containerName="istio-proxy"
Nov 01 03:41:23.588696 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.588679    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4cae562-15e5-4d0b-bcef-62ca774e2369" containerName="istio-proxy"
Nov 01 03:41:23.588696 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.588686    4930 cpu_manager.go:414] "containerMap: removing container" podUID="463ab8cc-0751-4fcf-a387-4b85b9f09180" containerName="payload-processing"
Nov 01 03:41:23.588696 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.588691    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="463ab8cc-0751-4fcf-a387-4b85b9f09180" containerName="payload-processing"
Nov 01 03:41:23.588696 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.588697    4930 cpu_manager.go:414] "containerMap: removing container" podUID="ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6" containerName="istio-proxy"
Nov 01 03:41:23.588954 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.588703    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6" containerName="istio-proxy"
Nov 01 03:41:23.588954 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.588713    4930 cpu_manager.go:414] "containerMap: removing container" podUID="4428f375-da7e-41c1-8e59-e73ff06e76f4" containerName="revoke-keys"
Nov 01 03:41:23.588954 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.588717    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4428f375-da7e-41c1-8e59-e73ff06e76f4" containerName="revoke-keys"
Nov 01 03:41:23.588954 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.588723    4930 cpu_manager.go:414] "containerMap: removing container" podUID="5974826d-4ad8-4b1e-b2ad-322b8d01359e" containerName="payload-pre-processing"
Nov 01 03:41:23.588954 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.588728    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5974826d-4ad8-4b1e-b2ad-322b8d01359e" containerName="payload-pre-processing"
Nov 01 03:41:23.588954 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.588749    4930 cpu_manager.go:414] "containerMap: removing container" podUID="a4cae562-15e5-4d0b-bcef-62ca774e2369" containerName="istio-proxy"
Nov 01 03:41:23.588954 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.588753    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4cae562-15e5-4d0b-bcef-62ca774e2369" containerName="istio-proxy"
Nov 01 03:41:23.588954 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.588855    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6346712-2068-4299-90f9-df1fcf150cc7" containerName="revoke-keys"
Nov 01 03:41:23.588954 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.588863    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4cae562-15e5-4d0b-bcef-62ca774e2369" containerName="istio-proxy"
Nov 01 03:41:23.588954 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.588869    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="463ab8cc-0751-4fcf-a387-4b85b9f09180" containerName="payload-processing"
Nov 01 03:41:23.588954 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.588876    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6" containerName="istio-proxy"
Nov 01 03:41:23.588954 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.588882    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="4428f375-da7e-41c1-8e59-e73ff06e76f4" containerName="revoke-keys"
Nov 01 03:41:23.588954 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.588890    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="5974826d-4ad8-4b1e-b2ad-322b8d01359e" containerName="payload-pre-processing"
Nov 01 03:41:23.588954 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.588898    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4cae562-15e5-4d0b-bcef-62ca774e2369" containerName="istio-proxy"
Nov 01 03:41:23.593911 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.593893    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"
Nov 01 03:41:23.666204 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.666165    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-2c8373-openshift-default-dockercfg-ss7xw\""
Nov 01 03:41:23.686101 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.686066    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:23.703525 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.703497    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27e7c281-1391-4406-9dfa-d770c821678e-credential-socket\") pod \"e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw\" (UID: \"27e7c281-1391-4406-9dfa-d770c821678e\") " pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"
Nov 01 03:41:23.703893 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.703543    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27e7c281-1391-4406-9dfa-d770c821678e-istio-data\") pod \"e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw\" (UID: \"27e7c281-1391-4406-9dfa-d770c821678e\") " pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"
Nov 01 03:41:23.703893 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.703677    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27e7c281-1391-4406-9dfa-d770c821678e-istio-token\") pod \"e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw\" (UID: \"27e7c281-1391-4406-9dfa-d770c821678e\") " pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"
Nov 01 03:41:23.703893 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.703708    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27e7c281-1391-4406-9dfa-d770c821678e-istio-envoy\") pod \"e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw\" (UID: \"27e7c281-1391-4406-9dfa-d770c821678e\") " pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"
Nov 01 03:41:23.703893 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.703835    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcbmj\" (UniqueName: \"kubernetes.io/projected/27e7c281-1391-4406-9dfa-d770c821678e-kube-api-access-dcbmj\") pod \"e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw\" (UID: \"27e7c281-1391-4406-9dfa-d770c821678e\") " pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"
Nov 01 03:41:23.703893 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.703894    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27e7c281-1391-4406-9dfa-d770c821678e-istiod-ca-cert\") pod \"e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw\" (UID: \"27e7c281-1391-4406-9dfa-d770c821678e\") " pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"
Nov 01 03:41:23.704098 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.703968    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27e7c281-1391-4406-9dfa-d770c821678e-workload-certs\") pod \"e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw\" (UID: \"27e7c281-1391-4406-9dfa-d770c821678e\") " pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"
Nov 01 03:41:23.704098 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.704000    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27e7c281-1391-4406-9dfa-d770c821678e-workload-socket\") pod \"e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw\" (UID: \"27e7c281-1391-4406-9dfa-d770c821678e\") " pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"
Nov 01 03:41:23.704180 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.704143    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27e7c281-1391-4406-9dfa-d770c821678e-istio-podinfo\") pod \"e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw\" (UID: \"27e7c281-1391-4406-9dfa-d770c821678e\") " pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"
Nov 01 03:41:23.719152 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.719127    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"]
Nov 01 03:41:23.805495 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.805454    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27e7c281-1391-4406-9dfa-d770c821678e-istiod-ca-cert\") pod \"e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw\" (UID: \"27e7c281-1391-4406-9dfa-d770c821678e\") " pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"
Nov 01 03:41:23.805893 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.805856    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27e7c281-1391-4406-9dfa-d770c821678e-workload-certs\") pod \"e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw\" (UID: \"27e7c281-1391-4406-9dfa-d770c821678e\") " pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"
Nov 01 03:41:23.806027 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.805915    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27e7c281-1391-4406-9dfa-d770c821678e-workload-socket\") pod \"e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw\" (UID: \"27e7c281-1391-4406-9dfa-d770c821678e\") " pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"
Nov 01 03:41:23.806110 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.806050    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27e7c281-1391-4406-9dfa-d770c821678e-istio-podinfo\") pod \"e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw\" (UID: \"27e7c281-1391-4406-9dfa-d770c821678e\") " pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"
Nov 01 03:41:23.806171 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.806129    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27e7c281-1391-4406-9dfa-d770c821678e-credential-socket\") pod \"e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw\" (UID: \"27e7c281-1391-4406-9dfa-d770c821678e\") " pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"
Nov 01 03:41:23.806348 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.806176    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27e7c281-1391-4406-9dfa-d770c821678e-istio-data\") pod \"e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw\" (UID: \"27e7c281-1391-4406-9dfa-d770c821678e\") " pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"
Nov 01 03:41:23.806348 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.806180    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27e7c281-1391-4406-9dfa-d770c821678e-istiod-ca-cert\") pod \"e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw\" (UID: \"27e7c281-1391-4406-9dfa-d770c821678e\") " pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"
Nov 01 03:41:23.806348 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.806274    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27e7c281-1391-4406-9dfa-d770c821678e-istio-token\") pod \"e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw\" (UID: \"27e7c281-1391-4406-9dfa-d770c821678e\") " pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"
Nov 01 03:41:23.806348 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.806301    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27e7c281-1391-4406-9dfa-d770c821678e-istio-envoy\") pod \"e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw\" (UID: \"27e7c281-1391-4406-9dfa-d770c821678e\") " pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"
Nov 01 03:41:23.806640 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.806372    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcbmj\" (UniqueName: \"kubernetes.io/projected/27e7c281-1391-4406-9dfa-d770c821678e-kube-api-access-dcbmj\") pod \"e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw\" (UID: \"27e7c281-1391-4406-9dfa-d770c821678e\") " pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"
Nov 01 03:41:23.806640 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.806456    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27e7c281-1391-4406-9dfa-d770c821678e-workload-socket\") pod \"e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw\" (UID: \"27e7c281-1391-4406-9dfa-d770c821678e\") " pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"
Nov 01 03:41:23.806640 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.806509    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27e7c281-1391-4406-9dfa-d770c821678e-istio-data\") pod \"e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw\" (UID: \"27e7c281-1391-4406-9dfa-d770c821678e\") " pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"
Nov 01 03:41:23.806824 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.806759    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27e7c281-1391-4406-9dfa-d770c821678e-workload-certs\") pod \"e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw\" (UID: \"27e7c281-1391-4406-9dfa-d770c821678e\") " pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"
Nov 01 03:41:23.807053 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.807032    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27e7c281-1391-4406-9dfa-d770c821678e-credential-socket\") pod \"e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw\" (UID: \"27e7c281-1391-4406-9dfa-d770c821678e\") " pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"
Nov 01 03:41:23.809356 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.809332    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27e7c281-1391-4406-9dfa-d770c821678e-istio-envoy\") pod \"e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw\" (UID: \"27e7c281-1391-4406-9dfa-d770c821678e\") " pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"
Nov 01 03:41:23.809700 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.809678    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27e7c281-1391-4406-9dfa-d770c821678e-istio-podinfo\") pod \"e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw\" (UID: \"27e7c281-1391-4406-9dfa-d770c821678e\") " pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"
Nov 01 03:41:23.869481 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.869414    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27e7c281-1391-4406-9dfa-d770c821678e-istio-token\") pod \"e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw\" (UID: \"27e7c281-1391-4406-9dfa-d770c821678e\") " pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"
Nov 01 03:41:23.869789 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.869770    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcbmj\" (UniqueName: \"kubernetes.io/projected/27e7c281-1391-4406-9dfa-d770c821678e-kube-api-access-dcbmj\") pod \"e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw\" (UID: \"27e7c281-1391-4406-9dfa-d770c821678e\") " pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"
Nov 01 03:41:23.922889 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:23.922856    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"
Nov 01 03:41:24.035261 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:24.035159    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:24.377027 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:24.376999    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"]
Nov 01 03:41:24.377293 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:41:24.377270    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27e7c281_1391_4406_9dfa_d770c821678e.slice/crio-eec9740092ce9b07e3d448b4ec0de2b89003a62251c19905e444706b174e651f WatchSource:0}: Error finding container eec9740092ce9b07e3d448b4ec0de2b89003a62251c19905e444706b174e651f: Status 404 returned error can't find the container with id eec9740092ce9b07e3d448b4ec0de2b89003a62251c19905e444706b174e651f
Nov 01 03:41:24.379993 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:24.379963    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:41:24.380091 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:24.380026    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:41:24.380091 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:24.380056    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:41:24.589297 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:24.589259    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:25.203158 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:25.203117    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw" event={"ID":"27e7c281-1391-4406-9dfa-d770c821678e","Type":"ContainerStarted","Data":"8ea181d9ef9afc7b4206e5a29ea369170cb7f6f25bf1a8fce106fa4f75966d9b"}
Nov 01 03:41:25.203584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:25.203188    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw" event={"ID":"27e7c281-1391-4406-9dfa-d770c821678e","Type":"ContainerStarted","Data":"eec9740092ce9b07e3d448b4ec0de2b89003a62251c19905e444706b174e651f"}
Nov 01 03:41:25.425172 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:25.425120    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw" podStartSLOduration=2.425104881 podStartE2EDuration="2.425104881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:41:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:41:25.372633515 +0000 UTC m=+2089.895640102" watchObservedRunningTime="2026-11-01 03:41:25.425104881 +0000 UTC m=+2089.948111469"
Nov 01 03:41:25.923181 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:25.923149    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"
Nov 01 03:41:25.929180 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:25.929152    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"
Nov 01 03:41:26.211236 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:26.211157    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"
Nov 01 03:41:26.212040 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:26.212019    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"
Nov 01 03:41:28.136133 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:28.136093    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:28.384859 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:28.384821    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:28.534268 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:28.534147    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:29.630950 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:29.630883    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:31.663592 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:31.660657    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:31.809198 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:31.809163    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:31.980570 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:31.980464    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2c8373-755cc448b7-8hkdb"]
Nov 01 03:41:31.981752 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:31.981728    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecf14a6b-aa67-4d3b-be7a-63b65eaa51e6" containerName="istio-proxy"
Nov 01 03:41:31.985474 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:31.985456    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c8373-755cc448b7-8hkdb"
Nov 01 03:41:32.043831 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:32.040655    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2c8373-755cc448b7-8hkdb"]
Nov 01 03:41:32.059582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:32.056876    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-2c8373\""
Nov 01 03:41:32.059582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:32.056898    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-2c8373-dockercfg-n4xxj\""
Nov 01 03:41:32.059582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:32.058112    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5dcc946-5a58-43bd-88c5-520f7ecb6104-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2c8373-755cc448b7-8hkdb\" (UID: \"f5dcc946-5a58-43bd-88c5-520f7ecb6104\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c8373-755cc448b7-8hkdb"
Nov 01 03:41:32.059582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:32.058167    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlv4f\" (UniqueName: \"kubernetes.io/projected/f5dcc946-5a58-43bd-88c5-520f7ecb6104-kube-api-access-qlv4f\") pod \"payload-pre-processing-e2e-api-a-2c8373-755cc448b7-8hkdb\" (UID: \"f5dcc946-5a58-43bd-88c5-520f7ecb6104\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c8373-755cc448b7-8hkdb"
Nov 01 03:41:32.126579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:32.126471    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2c8373-966cf6b8c-8n27c"]
Nov 01 03:41:32.159582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:32.158928    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2c8373-966cf6b8c-8n27c"
Nov 01 03:41:32.169584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:32.161286    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5dcc946-5a58-43bd-88c5-520f7ecb6104-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2c8373-755cc448b7-8hkdb\" (UID: \"f5dcc946-5a58-43bd-88c5-520f7ecb6104\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c8373-755cc448b7-8hkdb"
Nov 01 03:41:32.169584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:32.161353    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlv4f\" (UniqueName: \"kubernetes.io/projected/f5dcc946-5a58-43bd-88c5-520f7ecb6104-kube-api-access-qlv4f\") pod \"payload-pre-processing-e2e-api-a-2c8373-755cc448b7-8hkdb\" (UID: \"f5dcc946-5a58-43bd-88c5-520f7ecb6104\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c8373-755cc448b7-8hkdb"
Nov 01 03:41:32.256885 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:32.254907    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5dcc946-5a58-43bd-88c5-520f7ecb6104-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2c8373-755cc448b7-8hkdb\" (UID: \"f5dcc946-5a58-43bd-88c5-520f7ecb6104\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c8373-755cc448b7-8hkdb"
Nov 01 03:41:32.291619 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:32.282285    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2c8373-966cf6b8c-8n27c"]
Nov 01 03:41:32.316261 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:32.311375    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt4w2\" (UniqueName: \"kubernetes.io/projected/c031a873-378d-4081-845e-891f20c1354f-kube-api-access-jt4w2\") pod \"payload-processing-e2e-api-a-2c8373-966cf6b8c-8n27c\" (UID: \"c031a873-378d-4081-845e-891f20c1354f\") " pod="openshift-ingress/payload-processing-e2e-api-a-2c8373-966cf6b8c-8n27c"
Nov 01 03:41:32.316967 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:32.316929    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c031a873-378d-4081-845e-891f20c1354f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2c8373-966cf6b8c-8n27c\" (UID: \"c031a873-378d-4081-845e-891f20c1354f\") " pod="openshift-ingress/payload-processing-e2e-api-a-2c8373-966cf6b8c-8n27c"
Nov 01 03:41:32.326595 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:32.321661    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlv4f\" (UniqueName: \"kubernetes.io/projected/f5dcc946-5a58-43bd-88c5-520f7ecb6104-kube-api-access-qlv4f\") pod \"payload-pre-processing-e2e-api-a-2c8373-755cc448b7-8hkdb\" (UID: \"f5dcc946-5a58-43bd-88c5-520f7ecb6104\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c8373-755cc448b7-8hkdb"
Nov 01 03:41:32.418858 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:32.418823    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c031a873-378d-4081-845e-891f20c1354f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2c8373-966cf6b8c-8n27c\" (UID: \"c031a873-378d-4081-845e-891f20c1354f\") " pod="openshift-ingress/payload-processing-e2e-api-a-2c8373-966cf6b8c-8n27c"
Nov 01 03:41:32.419054 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:32.418908    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt4w2\" (UniqueName: \"kubernetes.io/projected/c031a873-378d-4081-845e-891f20c1354f-kube-api-access-jt4w2\") pod \"payload-processing-e2e-api-a-2c8373-966cf6b8c-8n27c\" (UID: \"c031a873-378d-4081-845e-891f20c1354f\") " pod="openshift-ingress/payload-processing-e2e-api-a-2c8373-966cf6b8c-8n27c"
Nov 01 03:41:32.419729 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:32.419704    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c031a873-378d-4081-845e-891f20c1354f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2c8373-966cf6b8c-8n27c\" (UID: \"c031a873-378d-4081-845e-891f20c1354f\") " pod="openshift-ingress/payload-processing-e2e-api-a-2c8373-966cf6b8c-8n27c"
Nov 01 03:41:32.479901 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:32.479874    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt4w2\" (UniqueName: \"kubernetes.io/projected/c031a873-378d-4081-845e-891f20c1354f-kube-api-access-jt4w2\") pod \"payload-processing-e2e-api-a-2c8373-966cf6b8c-8n27c\" (UID: \"c031a873-378d-4081-845e-891f20c1354f\") " pod="openshift-ingress/payload-processing-e2e-api-a-2c8373-966cf6b8c-8n27c"
Nov 01 03:41:32.602676 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:32.602643    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c8373-755cc448b7-8hkdb"
Nov 01 03:41:32.632829 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:32.632797    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2c8373-966cf6b8c-8n27c"
Nov 01 03:41:32.991907 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:41:32.991880    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5dcc946_5a58_43bd_88c5_520f7ecb6104.slice/crio-ea69cc77e0037d2568bfeba1dc3807420c1e11ad0e428d450749e2894d797009 WatchSource:0}: Error finding container ea69cc77e0037d2568bfeba1dc3807420c1e11ad0e428d450749e2894d797009: Status 404 returned error can't find the container with id ea69cc77e0037d2568bfeba1dc3807420c1e11ad0e428d450749e2894d797009
Nov 01 03:41:32.994694 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:32.994667    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2c8373-755cc448b7-8hkdb"]
Nov 01 03:41:33.041823 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:33.041798    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2c8373-966cf6b8c-8n27c"]
Nov 01 03:41:33.052047 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:41:33.052019    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc031a873_378d_4081_845e_891f20c1354f.slice/crio-1b227f221c952cb0830ab5181f6c2a0e8e6cda81495bd0c11d6d2d6783a2ff40 WatchSource:0}: Error finding container 1b227f221c952cb0830ab5181f6c2a0e8e6cda81495bd0c11d6d2d6783a2ff40: Status 404 returned error can't find the container with id 1b227f221c952cb0830ab5181f6c2a0e8e6cda81495bd0c11d6d2d6783a2ff40
Nov 01 03:41:33.328192 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:33.328103    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c8373-755cc448b7-8hkdb" event={"ID":"f5dcc946-5a58-43bd-88c5-520f7ecb6104","Type":"ContainerStarted","Data":"fef784a71930a0146fb4b7c723a394e75d36b6b8ad3b0c0098bfb7b7852460d5"}
Nov 01 03:41:33.328192 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:33.328179    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c8373-755cc448b7-8hkdb" event={"ID":"f5dcc946-5a58-43bd-88c5-520f7ecb6104","Type":"ContainerStarted","Data":"ea69cc77e0037d2568bfeba1dc3807420c1e11ad0e428d450749e2894d797009"}
Nov 01 03:41:33.328430 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:33.328394    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c8373-755cc448b7-8hkdb"
Nov 01 03:41:33.330090 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:33.330062    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2c8373-966cf6b8c-8n27c" event={"ID":"c031a873-378d-4081-845e-891f20c1354f","Type":"ContainerStarted","Data":"d599992fc0cc58dfa484cc279f60a4bdf20647d98b723e5e9a7312f81ee2271d"}
Nov 01 03:41:33.330223 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:33.330112    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-2c8373-966cf6b8c-8n27c"
Nov 01 03:41:33.330223 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:33.330129    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2c8373-966cf6b8c-8n27c" event={"ID":"c031a873-378d-4081-845e-891f20c1354f","Type":"ContainerStarted","Data":"1b227f221c952cb0830ab5181f6c2a0e8e6cda81495bd0c11d6d2d6783a2ff40"}
Nov 01 03:41:33.401461 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:33.401421    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:33.443946 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:33.443901    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c8373-755cc448b7-8hkdb" podStartSLOduration=2.443887728 podStartE2EDuration="2.443887728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:41:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:41:33.443331728 +0000 UTC m=+2097.966338317" watchObservedRunningTime="2026-11-01 03:41:33.443887728 +0000 UTC m=+2097.966894317"
Nov 01 03:41:33.554426 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:33.554373    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:33.568138 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:33.565380    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-2c8373-966cf6b8c-8n27c" podStartSLOduration=1.565359891 podStartE2EDuration="1.565359891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:41:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:41:33.561934691 +0000 UTC m=+2098.084941281" watchObservedRunningTime="2026-11-01 03:41:33.565359891 +0000 UTC m=+2098.088366479"
Nov 01 03:41:34.195715 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:34.195679    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:35.336123 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:35.336082    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:35.726534 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:35.725381    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:36.003520 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:36.003421    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:36.296689 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:36.277232    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fdbdfefd-ttbq5"]
Nov 01 03:41:36.298063 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:36.297946    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fdbdfefd-ttbq5"]
Nov 01 03:41:37.016070 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:37.016031    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:37.301936 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:37.301842    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:37.452898 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:37.452860    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:37.623264 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:37.623227    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92ed3fc3-dd13-42e8-a20e-84e0ec2de11a" path="/var/lib/kubelet/pods/92ed3fc3-dd13-42e8-a20e-84e0ec2de11a/volumes"
Nov 01 03:41:37.623621 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:37.623606    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d7329-h5bnz"]
Nov 01 03:41:37.669967 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:37.669937    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d7329-h5bnz"]
Nov 01 03:41:38.062846 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:38.062761    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:38.203951 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:38.203920    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jh955_f29ea61e-2a78-4c11-9566-f52497e135ff/console-operator/2.log"
Nov 01 03:41:38.228316 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:38.228291    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jh955_f29ea61e-2a78-4c11-9566-f52497e135ff/console-operator/2.log"
Nov 01 03:41:38.660592 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:38.660521    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:39.287352 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:39.287321    4930 scope.go:122] "RemoveContainer" containerID="770208bb7fb2c282f97ca9f490ee6b863372e8b9ad72db52b3e46f50eb7bf7f2"
Nov 01 03:41:39.622342 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:39.622309    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="389fb913-8428-49ca-bf68-4a9475625dc5" path="/var/lib/kubelet/pods/389fb913-8428-49ca-bf68-4a9475625dc5/volumes"
Nov 01 03:41:40.300960 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:40.300910    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:40.704024 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:40.703986    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f2383c-jlfkt"]
Nov 01 03:41:40.710191 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:40.710169    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f2383c-jlfkt"
Nov 01 03:41:40.731750 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:40.731722    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/714d732d-0c4f-410f-986b-5ea4b6807902-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f2383c-jlfkt\" (UID: \"714d732d-0c4f-410f-986b-5ea4b6807902\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f2383c-jlfkt"
Nov 01 03:41:40.731903 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:40.731883    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vphpj\" (UniqueName: \"kubernetes.io/projected/714d732d-0c4f-410f-986b-5ea4b6807902-kube-api-access-vphpj\") pod \"maas-api-revoke-keys-e2e-api-a-f2383c-jlfkt\" (UID: \"714d732d-0c4f-410f-986b-5ea4b6807902\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f2383c-jlfkt"
Nov 01 03:41:40.732041 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:40.732017    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/714d732d-0c4f-410f-986b-5ea4b6807902-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f2383c-jlfkt\" (UID: \"714d732d-0c4f-410f-986b-5ea4b6807902\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f2383c-jlfkt"
Nov 01 03:41:40.763853 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:40.763283    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:41:40.763853 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:40.763353    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:41:40.763853 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:40.763631    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:41:40.767316 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:40.767294    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f2383c-jlfkt"]
Nov 01 03:41:40.811249 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:40.811214    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:41:40.833617 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:40.833590    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/714d732d-0c4f-410f-986b-5ea4b6807902-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f2383c-jlfkt\" (UID: \"714d732d-0c4f-410f-986b-5ea4b6807902\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f2383c-jlfkt"
Nov 01 03:41:40.833751 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:40.833710    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vphpj\" (UniqueName: \"kubernetes.io/projected/714d732d-0c4f-410f-986b-5ea4b6807902-kube-api-access-vphpj\") pod \"maas-api-revoke-keys-e2e-api-a-f2383c-jlfkt\" (UID: \"714d732d-0c4f-410f-986b-5ea4b6807902\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f2383c-jlfkt"
Nov 01 03:41:40.833817 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:40.833801    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/714d732d-0c4f-410f-986b-5ea4b6807902-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f2383c-jlfkt\" (UID: \"714d732d-0c4f-410f-986b-5ea4b6807902\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f2383c-jlfkt"
Nov 01 03:41:40.834303 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:40.834282    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/714d732d-0c4f-410f-986b-5ea4b6807902-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f2383c-jlfkt\" (UID: \"714d732d-0c4f-410f-986b-5ea4b6807902\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f2383c-jlfkt"
Nov 01 03:41:40.836284 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:40.836261    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/714d732d-0c4f-410f-986b-5ea4b6807902-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f2383c-jlfkt\" (UID: \"714d732d-0c4f-410f-986b-5ea4b6807902\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f2383c-jlfkt"
Nov 01 03:41:40.894394 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:40.894364    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vphpj\" (UniqueName: \"kubernetes.io/projected/714d732d-0c4f-410f-986b-5ea4b6807902-kube-api-access-vphpj\") pod \"maas-api-revoke-keys-e2e-api-a-f2383c-jlfkt\" (UID: \"714d732d-0c4f-410f-986b-5ea4b6807902\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f2383c-jlfkt"
Nov 01 03:41:41.025839 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:41.025771    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f2383c-jlfkt"
Nov 01 03:41:41.412901 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:41:41.412872    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod714d732d_0c4f_410f_986b_5ea4b6807902.slice/crio-3430d55c5510d6462ece2ce474ec270553a2c299225855deede961b6642300f9 WatchSource:0}: Error finding container 3430d55c5510d6462ece2ce474ec270553a2c299225855deede961b6642300f9: Status 404 returned error can't find the container with id 3430d55c5510d6462ece2ce474ec270553a2c299225855deede961b6642300f9
Nov 01 03:41:41.413210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:41.413052    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f2383c-jlfkt"]
Nov 01 03:41:41.447132 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:41.447106    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f2383c-jlfkt" event={"ID":"714d732d-0c4f-410f-986b-5ea4b6807902","Type":"ContainerStarted","Data":"3430d55c5510d6462ece2ce474ec270553a2c299225855deede961b6642300f9"}
Nov 01 03:41:42.340734 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:42.340704    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:42.457986 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:42.457954    4930 generic.go:356] "Generic (PLEG): container finished" podID="714d732d-0c4f-410f-986b-5ea4b6807902" containerID="8e376d0c566316a35d7dc4cb79578e5a4c0ce5065de2784974117ebbc4f3084f" exitCode=0
Nov 01 03:41:42.458460 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:42.458036    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f2383c-jlfkt" event={"ID":"714d732d-0c4f-410f-986b-5ea4b6807902","Type":"ContainerDied","Data":"8e376d0c566316a35d7dc4cb79578e5a4c0ce5065de2784974117ebbc4f3084f"}
Nov 01 03:41:42.545514 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:42.545478    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:42.960164 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:42.960124    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:43.781315 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:43.781240    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f2383c-jlfkt"
Nov 01 03:41:43.905686 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:43.905651    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/714d732d-0c4f-410f-986b-5ea4b6807902-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/714d732d-0c4f-410f-986b-5ea4b6807902-maas-api-openshift-ca\") pod \"714d732d-0c4f-410f-986b-5ea4b6807902\" (UID: \"714d732d-0c4f-410f-986b-5ea4b6807902\") "
Nov 01 03:41:43.905902 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:43.905743    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/714d732d-0c4f-410f-986b-5ea4b6807902-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/714d732d-0c4f-410f-986b-5ea4b6807902-maas-api-xks-ca\") pod \"714d732d-0c4f-410f-986b-5ea4b6807902\" (UID: \"714d732d-0c4f-410f-986b-5ea4b6807902\") "
Nov 01 03:41:43.905980 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:43.905919    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/714d732d-0c4f-410f-986b-5ea4b6807902-kube-api-access-vphpj\" (UniqueName: \"kubernetes.io/projected/714d732d-0c4f-410f-986b-5ea4b6807902-kube-api-access-vphpj\") pod \"714d732d-0c4f-410f-986b-5ea4b6807902\" (UID: \"714d732d-0c4f-410f-986b-5ea4b6807902\") "
Nov 01 03:41:43.906146 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:43.906115    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/714d732d-0c4f-410f-986b-5ea4b6807902-maas-api-openshift-ca" pod "714d732d-0c4f-410f-986b-5ea4b6807902" (UID: "714d732d-0c4f-410f-986b-5ea4b6807902"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:41:43.906794 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:43.906765    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/714d732d-0c4f-410f-986b-5ea4b6807902-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:41:43.927342 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:43.927307    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/714d732d-0c4f-410f-986b-5ea4b6807902-kube-api-access-vphpj" pod "714d732d-0c4f-410f-986b-5ea4b6807902" (UID: "714d732d-0c4f-410f-986b-5ea4b6807902"). InnerVolumeSpecName "kube-api-access-vphpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:41:43.927477 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:43.927312    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/714d732d-0c4f-410f-986b-5ea4b6807902-maas-api-xks-ca" pod "714d732d-0c4f-410f-986b-5ea4b6807902" (UID: "714d732d-0c4f-410f-986b-5ea4b6807902"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:41:44.008156 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:44.008124    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vphpj\" (UniqueName: \"kubernetes.io/projected/714d732d-0c4f-410f-986b-5ea4b6807902-kube-api-access-vphpj\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:41:44.008156 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:44.008155    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/714d732d-0c4f-410f-986b-5ea4b6807902-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:41:44.350684 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:44.350652    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c8373-755cc448b7-8hkdb"
Nov 01 03:41:44.351479 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:44.351458    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-2c8373-966cf6b8c-8n27c"
Nov 01 03:41:44.460636 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:44.460599    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:44.541423 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:44.535085    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f2383c-jlfkt" event={"ID":"714d732d-0c4f-410f-986b-5ea4b6807902","Type":"ContainerDied","Data":"3430d55c5510d6462ece2ce474ec270553a2c299225855deede961b6642300f9"}
Nov 01 03:41:44.541423 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:44.535137    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3430d55c5510d6462ece2ce474ec270553a2c299225855deede961b6642300f9"
Nov 01 03:41:44.541423 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:44.535189    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f2383c-jlfkt"
Nov 01 03:41:44.609274 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:44.609191    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:45.043706 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:45.043601    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:45.201912 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:45.201876    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:45.474755 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:45.474640    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"]
Nov 01 03:41:45.475287 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:45.475246    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66" podUID="68863127-fc50-48b2-bf3e-c6c99fb4a29e" containerName="istio-proxy" containerID="cri-o://ca9e4eebeb18dd2da1bb6784c3f5bb73a66ac98460eb42744c83599dbaa27a1d" gracePeriod=30
Nov 01 03:41:45.769252 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:45.769166    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:48.055865 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:48.055771    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:48.264014 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:48.263975    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:48.381750 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:48.381714    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2058fb-zh7wl"]
Nov 01 03:41:48.390812 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:48.390778    4930 cpu_manager.go:414] "containerMap: removing container" podUID="714d732d-0c4f-410f-986b-5ea4b6807902" containerName="revoke-keys"
Nov 01 03:41:48.391014 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:48.390999    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="714d732d-0c4f-410f-986b-5ea4b6807902" containerName="revoke-keys"
Nov 01 03:41:48.391344 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:48.391323    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="714d732d-0c4f-410f-986b-5ea4b6807902" containerName="revoke-keys"
Nov 01 03:41:48.408132 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:48.408105    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2058fb-zh7wl"
Nov 01 03:41:48.443182 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:48.443145    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2058fb-zh7wl"]
Nov 01 03:41:48.492580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:48.484508    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4434213-7e88-4fff-bcd5-c7eca39696ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-2058fb-zh7wl\" (UID: \"e4434213-7e88-4fff-bcd5-c7eca39696ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2058fb-zh7wl"
Nov 01 03:41:48.523170 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:48.523124    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dvz6\" (UniqueName: \"kubernetes.io/projected/e4434213-7e88-4fff-bcd5-c7eca39696ff-kube-api-access-7dvz6\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-2058fb-zh7wl\" (UID: \"e4434213-7e88-4fff-bcd5-c7eca39696ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2058fb-zh7wl"
Nov 01 03:41:48.523631 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:48.523589    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4434213-7e88-4fff-bcd5-c7eca39696ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-2058fb-zh7wl\" (UID: \"e4434213-7e88-4fff-bcd5-c7eca39696ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2058fb-zh7wl"
Nov 01 03:41:48.524402 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:48.524375    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:41:48.526631 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:48.525765    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:41:48.526631 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:48.525830    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:41:48.526631 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:48.526529    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:41:48.640103 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:48.640004    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:48.644198 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:48.644149    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dvz6\" (UniqueName: \"kubernetes.io/projected/e4434213-7e88-4fff-bcd5-c7eca39696ff-kube-api-access-7dvz6\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-2058fb-zh7wl\" (UID: \"e4434213-7e88-4fff-bcd5-c7eca39696ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2058fb-zh7wl"
Nov 01 03:41:48.644801 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:48.644774    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4434213-7e88-4fff-bcd5-c7eca39696ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-2058fb-zh7wl\" (UID: \"e4434213-7e88-4fff-bcd5-c7eca39696ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2058fb-zh7wl"
Nov 01 03:41:48.645134 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:48.645099    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4434213-7e88-4fff-bcd5-c7eca39696ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-2058fb-zh7wl\" (UID: \"e4434213-7e88-4fff-bcd5-c7eca39696ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2058fb-zh7wl"
Nov 01 03:41:48.646218 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:48.645698    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4434213-7e88-4fff-bcd5-c7eca39696ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-2058fb-zh7wl\" (UID: \"e4434213-7e88-4fff-bcd5-c7eca39696ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2058fb-zh7wl"
Nov 01 03:41:48.649207 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:48.649186    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4434213-7e88-4fff-bcd5-c7eca39696ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-2058fb-zh7wl\" (UID: \"e4434213-7e88-4fff-bcd5-c7eca39696ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2058fb-zh7wl"
Nov 01 03:41:48.718519 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:48.718480    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dvz6\" (UniqueName: \"kubernetes.io/projected/e4434213-7e88-4fff-bcd5-c7eca39696ff-kube-api-access-7dvz6\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-2058fb-zh7wl\" (UID: \"e4434213-7e88-4fff-bcd5-c7eca39696ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2058fb-zh7wl"
Nov 01 03:41:48.763583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:48.752870    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2058fb-zh7wl"
Nov 01 03:41:49.077651 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:49.032984    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:49.249869 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:49.249719    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-79txj"]
Nov 01 03:41:49.643218 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:41:49.637594    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:41:49.646309 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:49.646274    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-79txj"
Nov 01 03:41:49.676586 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:49.669240    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-79txj"]
Nov 01 03:41:49.676586 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:49.669315    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fdbdfefd-6bcb9c8569-tp459"]
Nov 01 03:41:49.738048 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:49.735991    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-fdbdfefd\""
Nov 01 03:41:49.738048 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:49.737723    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fdbdfefd-6bcb9c8569-tp459"]
Nov 01 03:41:49.738048 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:49.737835    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fdbdfefd-6bcb9c8569-tp459"
Nov 01 03:41:49.750751 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:49.750209    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/088b8dfa-d4c7-4667-a118-f3f5cabde5e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-79txj\" (UID: \"088b8dfa-d4c7-4667-a118-f3f5cabde5e3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-79txj"
Nov 01 03:41:49.750751 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:49.750297    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nphp9\" (UniqueName: \"kubernetes.io/projected/088b8dfa-d4c7-4667-a118-f3f5cabde5e3-kube-api-access-nphp9\") pod \"payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-79txj\" (UID: \"088b8dfa-d4c7-4667-a118-f3f5cabde5e3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-79txj"
Nov 01 03:41:49.793577 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:49.792859    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-fdbdfefd-dockercfg-n2p6s\""
Nov 01 03:41:49.851937 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:49.851905    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/088b8dfa-d4c7-4667-a118-f3f5cabde5e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-79txj\" (UID: \"088b8dfa-d4c7-4667-a118-f3f5cabde5e3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-79txj"
Nov 01 03:41:49.851937 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:49.851940    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpz4c\" (UniqueName: \"kubernetes.io/projected/d6670d91-12e9-4621-8f9c-627f6d722be1-kube-api-access-tpz4c\") pod \"payload-processing-e2e-mt-fdbdfefd-6bcb9c8569-tp459\" (UID: \"d6670d91-12e9-4621-8f9c-627f6d722be1\") " pod="openshift-ingress/payload-processing-e2e-mt-fdbdfefd-6bcb9c8569-tp459"
Nov 01 03:41:49.852180 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:49.852006    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nphp9\" (UniqueName: \"kubernetes.io/projected/088b8dfa-d4c7-4667-a118-f3f5cabde5e3-kube-api-access-nphp9\") pod \"payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-79txj\" (UID: \"088b8dfa-d4c7-4667-a118-f3f5cabde5e3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-79txj"
Nov 01 03:41:49.852243 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:49.852187    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6670d91-12e9-4621-8f9c-627f6d722be1-plugins-config-volume\") pod \"payload-processing-e2e-mt-fdbdfefd-6bcb9c8569-tp459\" (UID: \"d6670d91-12e9-4621-8f9c-627f6d722be1\") " pod="openshift-ingress/payload-processing-e2e-mt-fdbdfefd-6bcb9c8569-tp459"
Nov 01 03:41:49.852817 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:49.852796    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/088b8dfa-d4c7-4667-a118-f3f5cabde5e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-79txj\" (UID: \"088b8dfa-d4c7-4667-a118-f3f5cabde5e3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-79txj"
Nov 01 03:41:49.924110 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:49.921374    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nphp9\" (UniqueName: \"kubernetes.io/projected/088b8dfa-d4c7-4667-a118-f3f5cabde5e3-kube-api-access-nphp9\") pod \"payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-79txj\" (UID: \"088b8dfa-d4c7-4667-a118-f3f5cabde5e3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-79txj"
Nov 01 03:41:49.954015 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:49.953988    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6670d91-12e9-4621-8f9c-627f6d722be1-plugins-config-volume\") pod \"payload-processing-e2e-mt-fdbdfefd-6bcb9c8569-tp459\" (UID: \"d6670d91-12e9-4621-8f9c-627f6d722be1\") " pod="openshift-ingress/payload-processing-e2e-mt-fdbdfefd-6bcb9c8569-tp459"
Nov 01 03:41:49.954181 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:49.954117    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpz4c\" (UniqueName: \"kubernetes.io/projected/d6670d91-12e9-4621-8f9c-627f6d722be1-kube-api-access-tpz4c\") pod \"payload-processing-e2e-mt-fdbdfefd-6bcb9c8569-tp459\" (UID: \"d6670d91-12e9-4621-8f9c-627f6d722be1\") " pod="openshift-ingress/payload-processing-e2e-mt-fdbdfefd-6bcb9c8569-tp459"
Nov 01 03:41:49.954956 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:49.954899    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6670d91-12e9-4621-8f9c-627f6d722be1-plugins-config-volume\") pod \"payload-processing-e2e-mt-fdbdfefd-6bcb9c8569-tp459\" (UID: \"d6670d91-12e9-4621-8f9c-627f6d722be1\") " pod="openshift-ingress/payload-processing-e2e-mt-fdbdfefd-6bcb9c8569-tp459"
Nov 01 03:41:50.028460 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:50.028413    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2058fb-zh7wl"]
Nov 01 03:41:50.045985 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:50.045961    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-79txj"
Nov 01 03:41:50.046762 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:50.046549    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpz4c\" (UniqueName: \"kubernetes.io/projected/d6670d91-12e9-4621-8f9c-627f6d722be1-kube-api-access-tpz4c\") pod \"payload-processing-e2e-mt-fdbdfefd-6bcb9c8569-tp459\" (UID: \"d6670d91-12e9-4621-8f9c-627f6d722be1\") " pod="openshift-ingress/payload-processing-e2e-mt-fdbdfefd-6bcb9c8569-tp459"
Nov 01 03:41:50.056002 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:50.055978    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fdbdfefd-6bcb9c8569-tp459"
Nov 01 03:41:50.221800 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:50.221661    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2058fb-zh7wl" event={"ID":"e4434213-7e88-4fff-bcd5-c7eca39696ff","Type":"ContainerStarted","Data":"5cff08fd89cfeee21ed3a9e8534bf70958ca7237eaba215938593a2db5b5d7c1"}
Nov 01 03:41:50.807464 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:50.806755    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-79txj"]
Nov 01 03:41:50.827000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:50.826971    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fdbdfefd-6bcb9c8569-tp459"]
Nov 01 03:41:50.858523 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:41:50.858492    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod088b8dfa_d4c7_4667_a118_f3f5cabde5e3.slice/crio-19bd7da35c14b72314953aa01abd9dfdc6cb0cb821181c51cf3c57fb64be6e0a WatchSource:0}: Error finding container 19bd7da35c14b72314953aa01abd9dfdc6cb0cb821181c51cf3c57fb64be6e0a: Status 404 returned error can't find the container with id 19bd7da35c14b72314953aa01abd9dfdc6cb0cb821181c51cf3c57fb64be6e0a
Nov 01 03:41:51.199031 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.198944    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f2383c-7959b95765-bjksj"]
Nov 01 03:41:51.231805 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.231724    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-79txj" event={"ID":"088b8dfa-d4c7-4667-a118-f3f5cabde5e3","Type":"ContainerStarted","Data":"4275033917f7d0eb7a2870cdb89104c159d4a6f82382970f4dda5e10d396e8fa"}
Nov 01 03:41:51.231805 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.231773    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-79txj" event={"ID":"088b8dfa-d4c7-4667-a118-f3f5cabde5e3","Type":"ContainerStarted","Data":"19bd7da35c14b72314953aa01abd9dfdc6cb0cb821181c51cf3c57fb64be6e0a"}
Nov 01 03:41:51.232295 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.231877    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-79txj"
Nov 01 03:41:51.233993 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.233935    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fdbdfefd-6bcb9c8569-tp459" event={"ID":"d6670d91-12e9-4621-8f9c-627f6d722be1","Type":"ContainerStarted","Data":"b5cc9d605899ff707c98dc66e9ffc287c1b0eabe8a4d5f55e5a6f3050f64e723"}
Nov 01 03:41:51.233993 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.233972    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fdbdfefd-6bcb9c8569-tp459" event={"ID":"d6670d91-12e9-4621-8f9c-627f6d722be1","Type":"ContainerStarted","Data":"7eb32af0f2ea95600728e24da73de58e3a7034a196268aba8eb3bf0926b488d6"}
Nov 01 03:41:51.234714 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.234476    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-fdbdfefd-6bcb9c8569-tp459"
Nov 01 03:41:51.236213 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.236195    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2058fb-zh7wl" event={"ID":"e4434213-7e88-4fff-bcd5-c7eca39696ff","Type":"ContainerStarted","Data":"079e52026a3008e76c0eb15672a6ff2ab50eef50da4def67a41b611ef249b18a"}
Nov 01 03:41:51.238366 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.238346    4930 generic.go:356] "Generic (PLEG): container finished" podID="68863127-fc50-48b2-bf3e-c6c99fb4a29e" containerID="ca9e4eebeb18dd2da1bb6784c3f5bb73a66ac98460eb42744c83599dbaa27a1d" exitCode=0
Nov 01 03:41:51.238452 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.238412    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66" event={"ID":"68863127-fc50-48b2-bf3e-c6c99fb4a29e","Type":"ContainerDied","Data":"ca9e4eebeb18dd2da1bb6784c3f5bb73a66ac98460eb42744c83599dbaa27a1d"}
Nov 01 03:41:51.238499 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.238454    4930 scope.go:122] "RemoveContainer" containerID="128cc4e340b3f0a09ae965072f66c97a64b1bca0f1657670db7c53f1454d9b75"
Nov 01 03:41:51.340141 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.340112    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f2383c-64bdcdb46c-2z655"]
Nov 01 03:41:51.553176 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.553141    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"
Nov 01 03:41:51.593295 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.593250    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:51.632623 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.632233    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68863127-fc50-48b2-bf3e-c6c99fb4a29e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68863127-fc50-48b2-bf3e-c6c99fb4a29e-istiod-ca-cert\") pod \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\" (UID: \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\") "
Nov 01 03:41:51.632623 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.632285    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68863127-fc50-48b2-bf3e-c6c99fb4a29e-kube-api-access-jrbd9\" (UniqueName: \"kubernetes.io/projected/68863127-fc50-48b2-bf3e-c6c99fb4a29e-kube-api-access-jrbd9\") pod \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\" (UID: \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\") "
Nov 01 03:41:51.632623 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.632471    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68863127-fc50-48b2-bf3e-c6c99fb4a29e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68863127-fc50-48b2-bf3e-c6c99fb4a29e-istio-envoy\") pod \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\" (UID: \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\") "
Nov 01 03:41:51.632623 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.632542    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68863127-fc50-48b2-bf3e-c6c99fb4a29e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68863127-fc50-48b2-bf3e-c6c99fb4a29e-workload-certs\") pod \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\" (UID: \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\") "
Nov 01 03:41:51.632623 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.632623    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68863127-fc50-48b2-bf3e-c6c99fb4a29e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68863127-fc50-48b2-bf3e-c6c99fb4a29e-istio-data\") pod \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\" (UID: \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\") "
Nov 01 03:41:51.633053 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.632746    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68863127-fc50-48b2-bf3e-c6c99fb4a29e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68863127-fc50-48b2-bf3e-c6c99fb4a29e-istio-podinfo\") pod \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\" (UID: \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\") "
Nov 01 03:41:51.633053 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.632836    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68863127-fc50-48b2-bf3e-c6c99fb4a29e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68863127-fc50-48b2-bf3e-c6c99fb4a29e-workload-socket\") pod \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\" (UID: \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\") "
Nov 01 03:41:51.633053 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.632937    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68863127-fc50-48b2-bf3e-c6c99fb4a29e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68863127-fc50-48b2-bf3e-c6c99fb4a29e-credential-socket\") pod \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\" (UID: \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\") "
Nov 01 03:41:51.633053 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.632991    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68863127-fc50-48b2-bf3e-c6c99fb4a29e-istio-token\" (UniqueName: \"kubernetes.io/projected/68863127-fc50-48b2-bf3e-c6c99fb4a29e-istio-token\") pod \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\" (UID: \"68863127-fc50-48b2-bf3e-c6c99fb4a29e\") "
Nov 01 03:41:51.640296 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.640257    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68863127-fc50-48b2-bf3e-c6c99fb4a29e-istio-token" pod "68863127-fc50-48b2-bf3e-c6c99fb4a29e" (UID: "68863127-fc50-48b2-bf3e-c6c99fb4a29e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:41:51.640462 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.640425    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68863127-fc50-48b2-bf3e-c6c99fb4a29e-istio-data" pod "68863127-fc50-48b2-bf3e-c6c99fb4a29e" (UID: "68863127-fc50-48b2-bf3e-c6c99fb4a29e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:41:51.640688 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.640664    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68863127-fc50-48b2-bf3e-c6c99fb4a29e-workload-certs" pod "68863127-fc50-48b2-bf3e-c6c99fb4a29e" (UID: "68863127-fc50-48b2-bf3e-c6c99fb4a29e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:41:51.640897 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.640877    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68863127-fc50-48b2-bf3e-c6c99fb4a29e-workload-socket" pod "68863127-fc50-48b2-bf3e-c6c99fb4a29e" (UID: "68863127-fc50-48b2-bf3e-c6c99fb4a29e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:41:51.641686 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.641640    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68863127-fc50-48b2-bf3e-c6c99fb4a29e-credential-socket" pod "68863127-fc50-48b2-bf3e-c6c99fb4a29e" (UID: "68863127-fc50-48b2-bf3e-c6c99fb4a29e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:41:51.643817 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.643788    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68863127-fc50-48b2-bf3e-c6c99fb4a29e-istiod-ca-cert" pod "68863127-fc50-48b2-bf3e-c6c99fb4a29e" (UID: "68863127-fc50-48b2-bf3e-c6c99fb4a29e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:41:51.649263 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.649209    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2058fb-zh7wl" podStartSLOduration=3.649188068 podStartE2EDuration="3.649188068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:41:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:41:51.638928864 +0000 UTC m=+2116.161935453" watchObservedRunningTime="2026-11-01 03:41:51.649188068 +0000 UTC m=+2116.172194648"
Nov 01 03:41:51.649894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.649858    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68863127-fc50-48b2-bf3e-c6c99fb4a29e-kube-api-access-jrbd9" pod "68863127-fc50-48b2-bf3e-c6c99fb4a29e" (UID: "68863127-fc50-48b2-bf3e-c6c99fb4a29e"). InnerVolumeSpecName "kube-api-access-jrbd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:41:51.650187 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.650150    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68863127-fc50-48b2-bf3e-c6c99fb4a29e-istio-envoy" pod "68863127-fc50-48b2-bf3e-c6c99fb4a29e" (UID: "68863127-fc50-48b2-bf3e-c6c99fb4a29e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:41:51.650898 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.650864    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68863127-fc50-48b2-bf3e-c6c99fb4a29e-istio-podinfo" pod "68863127-fc50-48b2-bf3e-c6c99fb4a29e" (UID: "68863127-fc50-48b2-bf3e-c6c99fb4a29e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:41:51.737173 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.737117    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68863127-fc50-48b2-bf3e-c6c99fb4a29e-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:41:51.737437 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.737403    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68863127-fc50-48b2-bf3e-c6c99fb4a29e-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:41:51.737572 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.737544    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68863127-fc50-48b2-bf3e-c6c99fb4a29e-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:41:51.737699 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.737671    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68863127-fc50-48b2-bf3e-c6c99fb4a29e-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:41:51.737817 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.737802    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68863127-fc50-48b2-bf3e-c6c99fb4a29e-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:41:51.743299 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.737904    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68863127-fc50-48b2-bf3e-c6c99fb4a29e-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:41:51.743299 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.740147    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68863127-fc50-48b2-bf3e-c6c99fb4a29e-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:41:51.743299 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.740191    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrbd9\" (UniqueName: \"kubernetes.io/projected/68863127-fc50-48b2-bf3e-c6c99fb4a29e-kube-api-access-jrbd9\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:41:51.743299 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.740206    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68863127-fc50-48b2-bf3e-c6c99fb4a29e-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:41:51.747352 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.747208    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:51.762372 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.762314    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-fdbdfefd-6bcb9c8569-tp459" podStartSLOduration=2.762296987 podStartE2EDuration="2.762296987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:41:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:41:51.762194163 +0000 UTC m=+2116.285200748" watchObservedRunningTime="2026-11-01 03:41:51.762296987 +0000 UTC m=+2116.285303576"
Nov 01 03:41:51.897589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:51.894336    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-79txj" podStartSLOduration=2.894315834 podStartE2EDuration="2.894315834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:41:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:41:51.891282127 +0000 UTC m=+2116.414288726" watchObservedRunningTime="2026-11-01 03:41:51.894315834 +0000 UTC m=+2116.417322422"
Nov 01 03:41:52.251294 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:52.251215    4930 generic.go:356] "Generic (PLEG): container finished" podID="e4434213-7e88-4fff-bcd5-c7eca39696ff" containerID="079e52026a3008e76c0eb15672a6ff2ab50eef50da4def67a41b611ef249b18a" exitCode=0
Nov 01 03:41:52.251790 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:52.251279    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2058fb-zh7wl" event={"ID":"e4434213-7e88-4fff-bcd5-c7eca39696ff","Type":"ContainerDied","Data":"079e52026a3008e76c0eb15672a6ff2ab50eef50da4def67a41b611ef249b18a"}
Nov 01 03:41:52.253015 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:52.252988    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"
Nov 01 03:41:52.253137 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:52.253015    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66" event={"ID":"68863127-fc50-48b2-bf3e-c6c99fb4a29e","Type":"ContainerDied","Data":"9a7c3815409fbc9121f471b9f7bc46179c0c25dc92d8d6fbfc9bd0ed62e5815f"}
Nov 01 03:41:52.253137 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:52.253061    4930 scope.go:122] "RemoveContainer" containerID="ca9e4eebeb18dd2da1bb6784c3f5bb73a66ac98460eb42744c83599dbaa27a1d"
Nov 01 03:41:52.490668 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:52.490624    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"]
Nov 01 03:41:52.494773 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:52.494746    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7d7329-openshift-default-64b6c5fdc9-p8c66"]
Nov 01 03:41:53.483712 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:53.483683    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2058fb-zh7wl"
Nov 01 03:41:53.585462 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:53.585411    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4434213-7e88-4fff-bcd5-c7eca39696ff-kube-api-access-7dvz6\" (UniqueName: \"kubernetes.io/projected/e4434213-7e88-4fff-bcd5-c7eca39696ff-kube-api-access-7dvz6\") pod \"e4434213-7e88-4fff-bcd5-c7eca39696ff\" (UID: \"e4434213-7e88-4fff-bcd5-c7eca39696ff\") "
Nov 01 03:41:53.585656 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:53.585505    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4434213-7e88-4fff-bcd5-c7eca39696ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4434213-7e88-4fff-bcd5-c7eca39696ff-maas-api-openshift-ca\") pod \"e4434213-7e88-4fff-bcd5-c7eca39696ff\" (UID: \"e4434213-7e88-4fff-bcd5-c7eca39696ff\") "
Nov 01 03:41:53.586174 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:53.585850    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4434213-7e88-4fff-bcd5-c7eca39696ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4434213-7e88-4fff-bcd5-c7eca39696ff-maas-api-xks-ca\") pod \"e4434213-7e88-4fff-bcd5-c7eca39696ff\" (UID: \"e4434213-7e88-4fff-bcd5-c7eca39696ff\") "
Nov 01 03:41:53.586296 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:53.586210    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4434213-7e88-4fff-bcd5-c7eca39696ff-maas-api-openshift-ca" pod "e4434213-7e88-4fff-bcd5-c7eca39696ff" (UID: "e4434213-7e88-4fff-bcd5-c7eca39696ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:41:53.586756 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:53.586732    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4434213-7e88-4fff-bcd5-c7eca39696ff-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:41:53.612579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:53.588408    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4434213-7e88-4fff-bcd5-c7eca39696ff-maas-api-xks-ca" pod "e4434213-7e88-4fff-bcd5-c7eca39696ff" (UID: "e4434213-7e88-4fff-bcd5-c7eca39696ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:41:53.612579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:53.588765    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4434213-7e88-4fff-bcd5-c7eca39696ff-kube-api-access-7dvz6" pod "e4434213-7e88-4fff-bcd5-c7eca39696ff" (UID: "e4434213-7e88-4fff-bcd5-c7eca39696ff"). InnerVolumeSpecName "kube-api-access-7dvz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:41:53.640739 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:53.638701    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68863127-fc50-48b2-bf3e-c6c99fb4a29e" path="/var/lib/kubelet/pods/68863127-fc50-48b2-bf3e-c6c99fb4a29e/volumes"
Nov 01 03:41:53.698627 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:53.691197    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:53.698627 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:53.692340    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4434213-7e88-4fff-bcd5-c7eca39696ff-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:41:53.698627 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:53.692373    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dvz6\" (UniqueName: \"kubernetes.io/projected/e4434213-7e88-4fff-bcd5-c7eca39696ff-kube-api-access-7dvz6\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:41:54.282041 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:54.282001    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2058fb-zh7wl" event={"ID":"e4434213-7e88-4fff-bcd5-c7eca39696ff","Type":"ContainerDied","Data":"5cff08fd89cfeee21ed3a9e8534bf70958ca7237eaba215938593a2db5b5d7c1"}
Nov 01 03:41:54.282258 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:54.282067    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cff08fd89cfeee21ed3a9e8534bf70958ca7237eaba215938593a2db5b5d7c1"
Nov 01 03:41:54.282258 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:54.282143    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2058fb-zh7wl"
Nov 01 03:41:56.200768 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:56.200720    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-f2383c-7959b95765-bjksj" podUID="2c62951f-01c4-4c25-9267-d741fb752efd" containerName="payload-processing" containerID="cri-o://f6de633e509a57caaf2d74d3d2ce96203f0cfba5fdf0169d0f636db4796f742b" gracePeriod=25
Nov 01 03:41:56.742949 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:56.340760    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f2383c-64bdcdb46c-2z655" podUID="58a93bc7-186b-4bfc-970c-045241a5cfd8" containerName="payload-pre-processing" containerID="cri-o://d1015b6878c59d80bf7598e648a3f9dce92c8bc1d52937216c4e3f6eea8ed69d" gracePeriod=25
Nov 01 03:41:56.742949 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:56.600861    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f2383c-7959b95765-bjksj"
Nov 01 03:41:56.742949 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:56.648881    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c62951f-01c4-4c25-9267-d741fb752efd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c62951f-01c4-4c25-9267-d741fb752efd-plugins-config-volume\") pod \"2c62951f-01c4-4c25-9267-d741fb752efd\" (UID: \"2c62951f-01c4-4c25-9267-d741fb752efd\") "
Nov 01 03:41:56.742949 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:56.649046    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c62951f-01c4-4c25-9267-d741fb752efd-kube-api-access-zp87g\" (UniqueName: \"kubernetes.io/projected/2c62951f-01c4-4c25-9267-d741fb752efd-kube-api-access-zp87g\") pod \"2c62951f-01c4-4c25-9267-d741fb752efd\" (UID: \"2c62951f-01c4-4c25-9267-d741fb752efd\") "
Nov 01 03:41:56.742949 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:56.650511    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c62951f-01c4-4c25-9267-d741fb752efd-plugins-config-volume" pod "2c62951f-01c4-4c25-9267-d741fb752efd" (UID: "2c62951f-01c4-4c25-9267-d741fb752efd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:41:56.742949 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:56.659966    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c62951f-01c4-4c25-9267-d741fb752efd-kube-api-access-zp87g" pod "2c62951f-01c4-4c25-9267-d741fb752efd" (UID: "2c62951f-01c4-4c25-9267-d741fb752efd"). InnerVolumeSpecName "kube-api-access-zp87g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:41:56.742949 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:56.735638    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f2383c-64bdcdb46c-2z655"
Nov 01 03:41:56.750204 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:56.750175    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58a93bc7-186b-4bfc-970c-045241a5cfd8-kube-api-access-pbfgh\" (UniqueName: \"kubernetes.io/projected/58a93bc7-186b-4bfc-970c-045241a5cfd8-kube-api-access-pbfgh\") pod \"58a93bc7-186b-4bfc-970c-045241a5cfd8\" (UID: \"58a93bc7-186b-4bfc-970c-045241a5cfd8\") "
Nov 01 03:41:56.750413 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:56.750394    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58a93bc7-186b-4bfc-970c-045241a5cfd8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58a93bc7-186b-4bfc-970c-045241a5cfd8-plugins-config-volume\") pod \"58a93bc7-186b-4bfc-970c-045241a5cfd8\" (UID: \"58a93bc7-186b-4bfc-970c-045241a5cfd8\") "
Nov 01 03:41:56.750909 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:56.750887    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58a93bc7-186b-4bfc-970c-045241a5cfd8-plugins-config-volume" pod "58a93bc7-186b-4bfc-970c-045241a5cfd8" (UID: "58a93bc7-186b-4bfc-970c-045241a5cfd8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:41:56.751155 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:56.751136    4930 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c62951f-01c4-4c25-9267-d741fb752efd-plugins-config-volume\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:41:56.751253 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:56.751161    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp87g\" (UniqueName: \"kubernetes.io/projected/2c62951f-01c4-4c25-9267-d741fb752efd-kube-api-access-zp87g\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:41:56.751253 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:56.751176    4930 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58a93bc7-186b-4bfc-970c-045241a5cfd8-plugins-config-volume\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:41:56.753099 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:56.753072    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58a93bc7-186b-4bfc-970c-045241a5cfd8-kube-api-access-pbfgh" pod "58a93bc7-186b-4bfc-970c-045241a5cfd8" (UID: "58a93bc7-186b-4bfc-970c-045241a5cfd8"). InnerVolumeSpecName "kube-api-access-pbfgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:41:56.852604 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:56.852430    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbfgh\" (UniqueName: \"kubernetes.io/projected/58a93bc7-186b-4bfc-970c-045241a5cfd8-kube-api-access-pbfgh\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:41:57.314950 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:57.314865    4930 generic.go:356] "Generic (PLEG): container finished" podID="7d9844bf-a8fe-4904-87fa-fb427c29dbd6" containerID="cbb49232e7db24c7465743c1aecb38f7063b1da1d6f87afac56dbc6c2f4dce11" exitCode=0
Nov 01 03:41:57.315365 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:57.314942    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x" event={"ID":"7d9844bf-a8fe-4904-87fa-fb427c29dbd6","Type":"ContainerDied","Data":"cbb49232e7db24c7465743c1aecb38f7063b1da1d6f87afac56dbc6c2f4dce11"}
Nov 01 03:41:57.315514 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:57.315491    4930 scope.go:122] "RemoveContainer" containerID="cbb49232e7db24c7465743c1aecb38f7063b1da1d6f87afac56dbc6c2f4dce11"
Nov 01 03:41:57.316751 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:57.316721    4930 generic.go:356] "Generic (PLEG): container finished" podID="58a93bc7-186b-4bfc-970c-045241a5cfd8" containerID="d1015b6878c59d80bf7598e648a3f9dce92c8bc1d52937216c4e3f6eea8ed69d" exitCode=0
Nov 01 03:41:57.317215 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:57.316988    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f2383c-64bdcdb46c-2z655"
Nov 01 03:41:57.317215 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:57.317133    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f2383c-64bdcdb46c-2z655" event={"ID":"58a93bc7-186b-4bfc-970c-045241a5cfd8","Type":"ContainerDied","Data":"d1015b6878c59d80bf7598e648a3f9dce92c8bc1d52937216c4e3f6eea8ed69d"}
Nov 01 03:41:57.317215 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:57.317174    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f2383c-64bdcdb46c-2z655" event={"ID":"58a93bc7-186b-4bfc-970c-045241a5cfd8","Type":"ContainerDied","Data":"d4ae2380a26fec95adf02ea42c362abe81ea49f8a7b4371030fc033545260bca"}
Nov 01 03:41:57.317215 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:57.317202    4930 scope.go:122] "RemoveContainer" containerID="d1015b6878c59d80bf7598e648a3f9dce92c8bc1d52937216c4e3f6eea8ed69d"
Nov 01 03:41:57.317793 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:57.317760    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:41:57.317909 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:57.317848    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:41:57.317909 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:57.317899    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:41:57.319913 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:57.319886    4930 generic.go:356] "Generic (PLEG): container finished" podID="2c62951f-01c4-4c25-9267-d741fb752efd" containerID="f6de633e509a57caaf2d74d3d2ce96203f0cfba5fdf0169d0f636db4796f742b" exitCode=0
Nov 01 03:41:57.320197 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:57.319968    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f2383c-7959b95765-bjksj" event={"ID":"2c62951f-01c4-4c25-9267-d741fb752efd","Type":"ContainerDied","Data":"f6de633e509a57caaf2d74d3d2ce96203f0cfba5fdf0169d0f636db4796f742b"}
Nov 01 03:41:57.320197 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:57.320013    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f2383c-7959b95765-bjksj" event={"ID":"2c62951f-01c4-4c25-9267-d741fb752efd","Type":"ContainerDied","Data":"b01ba786fa8bf8789683a7b137c4c6098d010ab5e8396d7e6258e63cf3311ce6"}
Nov 01 03:41:57.320197 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:57.320092    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f2383c-7959b95765-bjksj"
Nov 01 03:41:57.330094 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:57.330047    4930 scope.go:122] "RemoveContainer" containerID="d1015b6878c59d80bf7598e648a3f9dce92c8bc1d52937216c4e3f6eea8ed69d"
Nov 01 03:41:57.330385 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:41:57.330363    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1015b6878c59d80bf7598e648a3f9dce92c8bc1d52937216c4e3f6eea8ed69d\": container with ID starting with d1015b6878c59d80bf7598e648a3f9dce92c8bc1d52937216c4e3f6eea8ed69d not found: ID does not exist" containerID="d1015b6878c59d80bf7598e648a3f9dce92c8bc1d52937216c4e3f6eea8ed69d"
Nov 01 03:41:57.330609 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:57.330394    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1015b6878c59d80bf7598e648a3f9dce92c8bc1d52937216c4e3f6eea8ed69d"} err="failed to get container status \"d1015b6878c59d80bf7598e648a3f9dce92c8bc1d52937216c4e3f6eea8ed69d\": rpc error: code = NotFound desc = could not find container \"d1015b6878c59d80bf7598e648a3f9dce92c8bc1d52937216c4e3f6eea8ed69d\": container with ID starting with d1015b6878c59d80bf7598e648a3f9dce92c8bc1d52937216c4e3f6eea8ed69d not found: ID does not exist"
Nov 01 03:41:57.330609 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:57.330412    4930 scope.go:122] "RemoveContainer" containerID="f6de633e509a57caaf2d74d3d2ce96203f0cfba5fdf0169d0f636db4796f742b"
Nov 01 03:41:57.346105 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:57.346082    4930 scope.go:122] "RemoveContainer" containerID="f6de633e509a57caaf2d74d3d2ce96203f0cfba5fdf0169d0f636db4796f742b"
Nov 01 03:41:57.346410 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:41:57.346393    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6de633e509a57caaf2d74d3d2ce96203f0cfba5fdf0169d0f636db4796f742b\": container with ID starting with f6de633e509a57caaf2d74d3d2ce96203f0cfba5fdf0169d0f636db4796f742b not found: ID does not exist" containerID="f6de633e509a57caaf2d74d3d2ce96203f0cfba5fdf0169d0f636db4796f742b"
Nov 01 03:41:57.346520 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:57.346419    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6de633e509a57caaf2d74d3d2ce96203f0cfba5fdf0169d0f636db4796f742b"} err="failed to get container status \"f6de633e509a57caaf2d74d3d2ce96203f0cfba5fdf0169d0f636db4796f742b\": rpc error: code = NotFound desc = could not find container \"f6de633e509a57caaf2d74d3d2ce96203f0cfba5fdf0169d0f636db4796f742b\": container with ID starting with f6de633e509a57caaf2d74d3d2ce96203f0cfba5fdf0169d0f636db4796f742b not found: ID does not exist"
Nov 01 03:41:57.577094 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:57.577010    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f2383c-7959b95765-bjksj"]
Nov 01 03:41:57.696011 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:57.584111    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f2383c-7959b95765-bjksj"]
Nov 01 03:41:57.696011 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:57.623047    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c62951f-01c4-4c25-9267-d741fb752efd" path="/var/lib/kubelet/pods/2c62951f-01c4-4c25-9267-d741fb752efd/volumes"
Nov 01 03:41:57.696011 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:57.677673    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f2383c-64bdcdb46c-2z655"]
Nov 01 03:41:57.696011 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:57.681884    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f2383c-64bdcdb46c-2z655"]
Nov 01 03:41:58.330126 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:58.330040    4930 generic.go:356] "Generic (PLEG): container finished" podID="b72d8427-c8a5-4d67-be83-57a2f54dddb4" containerID="6b1f11a10a7deb1b747125dcba38e6a0e7b6b526073041c52e6abadd79caeeac" exitCode=0
Nov 01 03:41:58.330126 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:58.330106    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb" event={"ID":"b72d8427-c8a5-4d67-be83-57a2f54dddb4","Type":"ContainerDied","Data":"6b1f11a10a7deb1b747125dcba38e6a0e7b6b526073041c52e6abadd79caeeac"}
Nov 01 03:41:58.330649 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:58.330627    4930 scope.go:122] "RemoveContainer" containerID="6b1f11a10a7deb1b747125dcba38e6a0e7b6b526073041c52e6abadd79caeeac"
Nov 01 03:41:58.332266 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:58.332242    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x" event={"ID":"7d9844bf-a8fe-4904-87fa-fb427c29dbd6","Type":"ContainerStarted","Data":"393f985c36b58e02c5a6e6aaebd4acbac702a308cda4c445dff49ff48a13ae19"}
Nov 01 03:41:58.332506 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:58.332479    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:41:58.332643 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:58.332597    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:41:58.332723 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:58.332650    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:41:58.825605 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:58.825540    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:41:59.090960 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:59.090859    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"
Nov 01 03:41:59.091223 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:59.091198    4930 patch_prober.go:29] interesting pod/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 01 03:41:59.091305 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:59.091251    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x" podUID="7d9844bf-a8fe-4904-87fa-fb427c29dbd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 01 03:41:59.343617 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:59.343532    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb" event={"ID":"b72d8427-c8a5-4d67-be83-57a2f54dddb4","Type":"ContainerStarted","Data":"869a7fc0c617e8e088d899f5a9ad9ec5551dbeacd8a4f27d810b9993c32857a4"}
Nov 01 03:41:59.410738 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:59.410697    4930 patch_prober.go:29] interesting pod/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 01 03:41:59.411032 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:59.411001    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb" podUID="b72d8427-c8a5-4d67-be83-57a2f54dddb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 01 03:41:59.411486 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:59.410841    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"
Nov 01 03:41:59.648216 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:41:59.648136    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58a93bc7-186b-4bfc-970c-045241a5cfd8" path="/var/lib/kubelet/pods/58a93bc7-186b-4bfc-970c-045241a5cfd8/volumes"
Nov 01 03:42:00.121499 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:00.121463    4930 patch_prober.go:29] interesting pod/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 01 03:42:00.122377 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:00.121942    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x" podUID="7d9844bf-a8fe-4904-87fa-fb427c29dbd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 01 03:42:00.242450 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:00.242395    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:00.405637 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:00.400433    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:00.432056 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:00.409947    4930 patch_prober.go:29] interesting pod/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 01 03:42:00.432056 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:00.410004    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb" podUID="b72d8427-c8a5-4d67-be83-57a2f54dddb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 01 03:42:01.102583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:01.098632    4930 patch_prober.go:29] interesting pod/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 01 03:42:01.102583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:01.098692    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x" podUID="7d9844bf-a8fe-4904-87fa-fb427c29dbd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 01 03:42:01.426650 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:01.424779    4930 patch_prober.go:29] interesting pod/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 01 03:42:01.426650 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:01.424853    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb" podUID="b72d8427-c8a5-4d67-be83-57a2f54dddb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 01 03:42:02.098578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:02.097977    4930 patch_prober.go:29] interesting pod/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 01 03:42:02.098578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:02.098042    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x" podUID="7d9844bf-a8fe-4904-87fa-fb427c29dbd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 01 03:42:02.256943 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:02.256916    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-79txj"
Nov 01 03:42:02.412018 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:02.411928    4930 patch_prober.go:29] interesting pod/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 01 03:42:02.412270 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:02.412240    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb" podUID="b72d8427-c8a5-4d67-be83-57a2f54dddb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 01 03:42:03.094974 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:03.094934    4930 patch_prober.go:29] interesting pod/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 01 03:42:03.095580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:03.095529    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x" podUID="7d9844bf-a8fe-4904-87fa-fb427c29dbd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 01 03:42:03.309832 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:03.309790    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-fdbdfefd-6bcb9c8569-tp459"
Nov 01 03:42:03.424290 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:03.424205    4930 patch_prober.go:29] interesting pod/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 01 03:42:03.424543 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:03.424512    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb" podUID="b72d8427-c8a5-4d67-be83-57a2f54dddb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 01 03:42:03.731635 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:03.703153    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:03.947670 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:03.947635    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss"]
Nov 01 03:42:03.948270 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:03.948219    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss" podUID="6522cd84-3c6c-4280-9cb7-02eb8916b24d" containerName="istio-proxy" containerID="cri-o://b09efc044d9f444cbc90815da3059fbc3c8c18a835c2f06233f51cc908b7a3ed" gracePeriod=30
Nov 01 03:42:03.957986 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:03.957958    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl"]
Nov 01 03:42:03.958250 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:03.958195    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl" podUID="8c02ff41-a095-4c49-a78c-c78f920a42a4" containerName="istio-proxy" containerID="cri-o://7aea0a0f098ea4c0bfcbbfa96e06ef16db65fcfcff65ad262369f16211547e62" gracePeriod=30
Nov 01 03:42:04.034386 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:04.034294    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4"]
Nov 01 03:42:04.041115 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:04.041060    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4" podUID="53f53373-f6f8-43db-a686-ab971adec842" containerName="istio-proxy" containerID="cri-o://7fe2a79cdbde1663d62b3eaced58e2205eecf109b361f7a1313060c2f04a70dc" gracePeriod=30
Nov 01 03:42:04.110032 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:04.109986    4930 patch_prober.go:29] interesting pod/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Nov 01 03:42:04.110533 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:04.110062    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x" podUID="7d9844bf-a8fe-4904-87fa-fb427c29dbd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Nov 01 03:42:04.118134 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:04.118096    4930 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:42:04.118286 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:04.118169    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss" podUID="6522cd84-3c6c-4280-9cb7-02eb8916b24d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:42:04.409684 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:04.409642    4930 patch_prober.go:29] interesting pod/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 01 03:42:04.409880 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:04.409732    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb" podUID="b72d8427-c8a5-4d67-be83-57a2f54dddb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 01 03:42:04.717375 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:04.717282    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"
Nov 01 03:42:05.096266 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:05.096231    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"
Nov 01 03:42:05.418167 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:05.418080    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"
Nov 01 03:42:05.536402 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:05.536007    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"
Nov 01 03:42:05.668299 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:05.668220    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"
Nov 01 03:42:05.668299 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:05.668281    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"
Nov 01 03:42:06.364080 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:06.364044    4930 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:42:06.364256 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:06.364110    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl" podUID="8c02ff41-a095-4c49-a78c-c78f920a42a4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:42:07.782770 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:07.782734    4930 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:42:07.783173 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:07.782786    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4" podUID="53f53373-f6f8-43db-a686-ab971adec842" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:42:08.433279 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:08.433238    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:09.352057 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.352025    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss"
Nov 01 03:42:09.372455 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.372430    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4"
Nov 01 03:42:09.378494 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.378473    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl"
Nov 01 03:42:09.444863 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.444830    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6522cd84-3c6c-4280-9cb7-02eb8916b24d-istio-token\" (UniqueName: \"kubernetes.io/projected/6522cd84-3c6c-4280-9cb7-02eb8916b24d-istio-token\") pod \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\" (UID: \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\") "
Nov 01 03:42:09.445008 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.444891    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c02ff41-a095-4c49-a78c-c78f920a42a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c02ff41-a095-4c49-a78c-c78f920a42a4-credential-socket\") pod \"8c02ff41-a095-4c49-a78c-c78f920a42a4\" (UID: \"8c02ff41-a095-4c49-a78c-c78f920a42a4\") "
Nov 01 03:42:09.445008 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.444949    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6522cd84-3c6c-4280-9cb7-02eb8916b24d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6522cd84-3c6c-4280-9cb7-02eb8916b24d-istio-data\") pod \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\" (UID: \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\") "
Nov 01 03:42:09.445134 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.445034    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c02ff41-a095-4c49-a78c-c78f920a42a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c02ff41-a095-4c49-a78c-c78f920a42a4-workload-socket\") pod \"8c02ff41-a095-4c49-a78c-c78f920a42a4\" (UID: \"8c02ff41-a095-4c49-a78c-c78f920a42a4\") "
Nov 01 03:42:09.445134 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.445095    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53f53373-f6f8-43db-a686-ab971adec842-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53f53373-f6f8-43db-a686-ab971adec842-istio-envoy\") pod \"53f53373-f6f8-43db-a686-ab971adec842\" (UID: \"53f53373-f6f8-43db-a686-ab971adec842\") "
Nov 01 03:42:09.445244 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.445153    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c02ff41-a095-4c49-a78c-c78f920a42a4-credential-socket" pod "8c02ff41-a095-4c49-a78c-c78f920a42a4" (UID: "8c02ff41-a095-4c49-a78c-c78f920a42a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:09.445244 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.445159    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c02ff41-a095-4c49-a78c-c78f920a42a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c02ff41-a095-4c49-a78c-c78f920a42a4-istio-envoy\") pod \"8c02ff41-a095-4c49-a78c-c78f920a42a4\" (UID: \"8c02ff41-a095-4c49-a78c-c78f920a42a4\") "
Nov 01 03:42:09.445374 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.445244    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6522cd84-3c6c-4280-9cb7-02eb8916b24d-istio-data" pod "6522cd84-3c6c-4280-9cb7-02eb8916b24d" (UID: "6522cd84-3c6c-4280-9cb7-02eb8916b24d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:09.445374 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.445278    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6522cd84-3c6c-4280-9cb7-02eb8916b24d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6522cd84-3c6c-4280-9cb7-02eb8916b24d-istiod-ca-cert\") pod \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\" (UID: \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\") "
Nov 01 03:42:09.445374 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.445330    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53f53373-f6f8-43db-a686-ab971adec842-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f53373-f6f8-43db-a686-ab971adec842-credential-socket\") pod \"53f53373-f6f8-43db-a686-ab971adec842\" (UID: \"53f53373-f6f8-43db-a686-ab971adec842\") "
Nov 01 03:42:09.445374 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.445348    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c02ff41-a095-4c49-a78c-c78f920a42a4-workload-socket" pod "8c02ff41-a095-4c49-a78c-c78f920a42a4" (UID: "8c02ff41-a095-4c49-a78c-c78f920a42a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:09.445374 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.445369    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6522cd84-3c6c-4280-9cb7-02eb8916b24d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6522cd84-3c6c-4280-9cb7-02eb8916b24d-istio-podinfo\") pod \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\" (UID: \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\") "
Nov 01 03:42:09.445775 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.445411    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53f53373-f6f8-43db-a686-ab971adec842-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53f53373-f6f8-43db-a686-ab971adec842-istio-podinfo\") pod \"53f53373-f6f8-43db-a686-ab971adec842\" (UID: \"53f53373-f6f8-43db-a686-ab971adec842\") "
Nov 01 03:42:09.445775 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.445446    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6522cd84-3c6c-4280-9cb7-02eb8916b24d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6522cd84-3c6c-4280-9cb7-02eb8916b24d-istio-envoy\") pod \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\" (UID: \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\") "
Nov 01 03:42:09.445775 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.445471    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53f53373-f6f8-43db-a686-ab971adec842-istio-token\" (UniqueName: \"kubernetes.io/projected/53f53373-f6f8-43db-a686-ab971adec842-istio-token\") pod \"53f53373-f6f8-43db-a686-ab971adec842\" (UID: \"53f53373-f6f8-43db-a686-ab971adec842\") "
Nov 01 03:42:09.445775 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.445509    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c02ff41-a095-4c49-a78c-c78f920a42a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c02ff41-a095-4c49-a78c-c78f920a42a4-istio-podinfo\") pod \"8c02ff41-a095-4c49-a78c-c78f920a42a4\" (UID: \"8c02ff41-a095-4c49-a78c-c78f920a42a4\") "
Nov 01 03:42:09.445775 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.445539    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6522cd84-3c6c-4280-9cb7-02eb8916b24d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6522cd84-3c6c-4280-9cb7-02eb8916b24d-credential-socket\") pod \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\" (UID: \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\") "
Nov 01 03:42:09.445775 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.445609    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53f53373-f6f8-43db-a686-ab971adec842-credential-socket" pod "53f53373-f6f8-43db-a686-ab971adec842" (UID: "53f53373-f6f8-43db-a686-ab971adec842"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:09.445775 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.445682    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53f53373-f6f8-43db-a686-ab971adec842-kube-api-access-m52bc\" (UniqueName: \"kubernetes.io/projected/53f53373-f6f8-43db-a686-ab971adec842-kube-api-access-m52bc\") pod \"53f53373-f6f8-43db-a686-ab971adec842\" (UID: \"53f53373-f6f8-43db-a686-ab971adec842\") "
Nov 01 03:42:09.445775 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.445727    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6522cd84-3c6c-4280-9cb7-02eb8916b24d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6522cd84-3c6c-4280-9cb7-02eb8916b24d-workload-certs\") pod \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\" (UID: \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\") "
Nov 01 03:42:09.446279 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.445782    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c02ff41-a095-4c49-a78c-c78f920a42a4-istio-token\" (UniqueName: \"kubernetes.io/projected/8c02ff41-a095-4c49-a78c-c78f920a42a4-istio-token\") pod \"8c02ff41-a095-4c49-a78c-c78f920a42a4\" (UID: \"8c02ff41-a095-4c49-a78c-c78f920a42a4\") "
Nov 01 03:42:09.446279 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.445813    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53f53373-f6f8-43db-a686-ab971adec842-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f53373-f6f8-43db-a686-ab971adec842-workload-socket\") pod \"53f53373-f6f8-43db-a686-ab971adec842\" (UID: \"53f53373-f6f8-43db-a686-ab971adec842\") "
Nov 01 03:42:09.446279 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.445683    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6522cd84-3c6c-4280-9cb7-02eb8916b24d-istiod-ca-cert" pod "6522cd84-3c6c-4280-9cb7-02eb8916b24d" (UID: "6522cd84-3c6c-4280-9cb7-02eb8916b24d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:42:09.446279 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.445856    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c02ff41-a095-4c49-a78c-c78f920a42a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c02ff41-a095-4c49-a78c-c78f920a42a4-workload-certs\") pod \"8c02ff41-a095-4c49-a78c-c78f920a42a4\" (UID: \"8c02ff41-a095-4c49-a78c-c78f920a42a4\") "
Nov 01 03:42:09.446279 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.446101    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6522cd84-3c6c-4280-9cb7-02eb8916b24d-credential-socket" pod "6522cd84-3c6c-4280-9cb7-02eb8916b24d" (UID: "6522cd84-3c6c-4280-9cb7-02eb8916b24d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:09.446279 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.446201    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6522cd84-3c6c-4280-9cb7-02eb8916b24d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6522cd84-3c6c-4280-9cb7-02eb8916b24d-workload-socket\") pod \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\" (UID: \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\") "
Nov 01 03:42:09.446279 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.446265    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c02ff41-a095-4c49-a78c-c78f920a42a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c02ff41-a095-4c49-a78c-c78f920a42a4-istiod-ca-cert\") pod \"8c02ff41-a095-4c49-a78c-c78f920a42a4\" (UID: \"8c02ff41-a095-4c49-a78c-c78f920a42a4\") "
Nov 01 03:42:09.446749 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.446298    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53f53373-f6f8-43db-a686-ab971adec842-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53f53373-f6f8-43db-a686-ab971adec842-istiod-ca-cert\") pod \"53f53373-f6f8-43db-a686-ab971adec842\" (UID: \"53f53373-f6f8-43db-a686-ab971adec842\") "
Nov 01 03:42:09.446749 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.446325    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53f53373-f6f8-43db-a686-ab971adec842-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53f53373-f6f8-43db-a686-ab971adec842-istio-data\") pod \"53f53373-f6f8-43db-a686-ab971adec842\" (UID: \"53f53373-f6f8-43db-a686-ab971adec842\") "
Nov 01 03:42:09.446749 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.446358    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c02ff41-a095-4c49-a78c-c78f920a42a4-kube-api-access-vcqmj\" (UniqueName: \"kubernetes.io/projected/8c02ff41-a095-4c49-a78c-c78f920a42a4-kube-api-access-vcqmj\") pod \"8c02ff41-a095-4c49-a78c-c78f920a42a4\" (UID: \"8c02ff41-a095-4c49-a78c-c78f920a42a4\") "
Nov 01 03:42:09.446749 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.446396    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6522cd84-3c6c-4280-9cb7-02eb8916b24d-workload-certs" pod "6522cd84-3c6c-4280-9cb7-02eb8916b24d" (UID: "6522cd84-3c6c-4280-9cb7-02eb8916b24d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:09.446749 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.446405    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c02ff41-a095-4c49-a78c-c78f920a42a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c02ff41-a095-4c49-a78c-c78f920a42a4-istio-data\") pod \"8c02ff41-a095-4c49-a78c-c78f920a42a4\" (UID: \"8c02ff41-a095-4c49-a78c-c78f920a42a4\") "
Nov 01 03:42:09.446749 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.446485    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6522cd84-3c6c-4280-9cb7-02eb8916b24d-kube-api-access-kg4fl\" (UniqueName: \"kubernetes.io/projected/6522cd84-3c6c-4280-9cb7-02eb8916b24d-kube-api-access-kg4fl\") pod \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\" (UID: \"6522cd84-3c6c-4280-9cb7-02eb8916b24d\") "
Nov 01 03:42:09.446749 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.446597    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c02ff41-a095-4c49-a78c-c78f920a42a4-istio-data" pod "8c02ff41-a095-4c49-a78c-c78f920a42a4" (UID: "8c02ff41-a095-4c49-a78c-c78f920a42a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:09.446749 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.446618    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53f53373-f6f8-43db-a686-ab971adec842-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53f53373-f6f8-43db-a686-ab971adec842-workload-certs\") pod \"53f53373-f6f8-43db-a686-ab971adec842\" (UID: \"53f53373-f6f8-43db-a686-ab971adec842\") "
Nov 01 03:42:09.447244 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.446906    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c02ff41-a095-4c49-a78c-c78f920a42a4-istiod-ca-cert" pod "8c02ff41-a095-4c49-a78c-c78f920a42a4" (UID: "8c02ff41-a095-4c49-a78c-c78f920a42a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:42:09.447244 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.446964    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53f53373-f6f8-43db-a686-ab971adec842-istiod-ca-cert" pod "53f53373-f6f8-43db-a686-ab971adec842" (UID: "53f53373-f6f8-43db-a686-ab971adec842"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:42:09.447997 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.447238    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53f53373-f6f8-43db-a686-ab971adec842-workload-socket" pod "53f53373-f6f8-43db-a686-ab971adec842" (UID: "53f53373-f6f8-43db-a686-ab971adec842"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:09.447997 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.447534    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c02ff41-a095-4c49-a78c-c78f920a42a4-workload-certs" pod "8c02ff41-a095-4c49-a78c-c78f920a42a4" (UID: "8c02ff41-a095-4c49-a78c-c78f920a42a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:09.447997 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.447686    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6522cd84-3c6c-4280-9cb7-02eb8916b24d-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.447997 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.447690    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6522cd84-3c6c-4280-9cb7-02eb8916b24d-workload-socket" pod "6522cd84-3c6c-4280-9cb7-02eb8916b24d" (UID: "6522cd84-3c6c-4280-9cb7-02eb8916b24d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:09.447997 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.447712    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6522cd84-3c6c-4280-9cb7-02eb8916b24d-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.447997 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.447728    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f53373-f6f8-43db-a686-ab971adec842-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.447997 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.447742    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c02ff41-a095-4c49-a78c-c78f920a42a4-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.447997 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.447757    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53f53373-f6f8-43db-a686-ab971adec842-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.447997 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.447777    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c02ff41-a095-4c49-a78c-c78f920a42a4-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.447997 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.447795    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53f53373-f6f8-43db-a686-ab971adec842-istio-data" pod "53f53373-f6f8-43db-a686-ab971adec842" (UID: "53f53373-f6f8-43db-a686-ab971adec842"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:09.447997 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.447799    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c02ff41-a095-4c49-a78c-c78f920a42a4-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.450447 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.448086    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6522cd84-3c6c-4280-9cb7-02eb8916b24d-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.450447 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.448111    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c02ff41-a095-4c49-a78c-c78f920a42a4-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.450447 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.448137    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6522cd84-3c6c-4280-9cb7-02eb8916b24d-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.450447 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.448156    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f53373-f6f8-43db-a686-ab971adec842-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.450447 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.448376    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c02ff41-a095-4c49-a78c-c78f920a42a4-istio-envoy" pod "8c02ff41-a095-4c49-a78c-c78f920a42a4" (UID: "8c02ff41-a095-4c49-a78c-c78f920a42a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:09.450447 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.448448    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53f53373-f6f8-43db-a686-ab971adec842-workload-certs" pod "53f53373-f6f8-43db-a686-ab971adec842" (UID: "53f53373-f6f8-43db-a686-ab971adec842"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:09.450447 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.448477    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6522cd84-3c6c-4280-9cb7-02eb8916b24d-istio-token" pod "6522cd84-3c6c-4280-9cb7-02eb8916b24d" (UID: "6522cd84-3c6c-4280-9cb7-02eb8916b24d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:42:09.450447 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.450441    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53f53373-f6f8-43db-a686-ab971adec842-istio-podinfo" pod "53f53373-f6f8-43db-a686-ab971adec842" (UID: "53f53373-f6f8-43db-a686-ab971adec842"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:42:09.451548 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.451512    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c02ff41-a095-4c49-a78c-c78f920a42a4-istio-podinfo" pod "8c02ff41-a095-4c49-a78c-c78f920a42a4" (UID: "8c02ff41-a095-4c49-a78c-c78f920a42a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:42:09.451548 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.451529    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53f53373-f6f8-43db-a686-ab971adec842-istio-envoy" pod "53f53373-f6f8-43db-a686-ab971adec842" (UID: "53f53373-f6f8-43db-a686-ab971adec842"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:09.451763 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.451598    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c02ff41-a095-4c49-a78c-c78f920a42a4-kube-api-access-vcqmj" pod "8c02ff41-a095-4c49-a78c-c78f920a42a4" (UID: "8c02ff41-a095-4c49-a78c-c78f920a42a4"). InnerVolumeSpecName "kube-api-access-vcqmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:42:09.451763 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.451694    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53f53373-f6f8-43db-a686-ab971adec842-kube-api-access-m52bc" pod "53f53373-f6f8-43db-a686-ab971adec842" (UID: "53f53373-f6f8-43db-a686-ab971adec842"). InnerVolumeSpecName "kube-api-access-m52bc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:42:09.451874 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.451820    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c02ff41-a095-4c49-a78c-c78f920a42a4-istio-token" pod "8c02ff41-a095-4c49-a78c-c78f920a42a4" (UID: "8c02ff41-a095-4c49-a78c-c78f920a42a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:42:09.451991 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.451969    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6522cd84-3c6c-4280-9cb7-02eb8916b24d-istio-envoy" pod "6522cd84-3c6c-4280-9cb7-02eb8916b24d" (UID: "6522cd84-3c6c-4280-9cb7-02eb8916b24d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:09.452062 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.451974    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6522cd84-3c6c-4280-9cb7-02eb8916b24d-istio-podinfo" pod "6522cd84-3c6c-4280-9cb7-02eb8916b24d" (UID: "6522cd84-3c6c-4280-9cb7-02eb8916b24d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:42:09.452139 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.452116    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53f53373-f6f8-43db-a686-ab971adec842-istio-token" pod "53f53373-f6f8-43db-a686-ab971adec842" (UID: "53f53373-f6f8-43db-a686-ab971adec842"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:42:09.452549 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.452529    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6522cd84-3c6c-4280-9cb7-02eb8916b24d-kube-api-access-kg4fl" pod "6522cd84-3c6c-4280-9cb7-02eb8916b24d" (UID: "6522cd84-3c6c-4280-9cb7-02eb8916b24d"). InnerVolumeSpecName "kube-api-access-kg4fl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:42:09.549836 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.549720    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c02ff41-a095-4c49-a78c-c78f920a42a4-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.549836 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.549751    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6522cd84-3c6c-4280-9cb7-02eb8916b24d-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.549836 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.549761    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53f53373-f6f8-43db-a686-ab971adec842-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.549836 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.549770    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6522cd84-3c6c-4280-9cb7-02eb8916b24d-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.549836 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.549782    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53f53373-f6f8-43db-a686-ab971adec842-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.549836 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.549793    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c02ff41-a095-4c49-a78c-c78f920a42a4-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.549836 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.549802    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m52bc\" (UniqueName: \"kubernetes.io/projected/53f53373-f6f8-43db-a686-ab971adec842-kube-api-access-m52bc\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.549836 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.549813    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c02ff41-a095-4c49-a78c-c78f920a42a4-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.549836 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.549821    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c02ff41-a095-4c49-a78c-c78f920a42a4-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.549836 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.549829    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6522cd84-3c6c-4280-9cb7-02eb8916b24d-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.549836 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.549838    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53f53373-f6f8-43db-a686-ab971adec842-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.549836 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.549847    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcqmj\" (UniqueName: \"kubernetes.io/projected/8c02ff41-a095-4c49-a78c-c78f920a42a4-kube-api-access-vcqmj\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.550651 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.549856    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg4fl\" (UniqueName: \"kubernetes.io/projected/6522cd84-3c6c-4280-9cb7-02eb8916b24d-kube-api-access-kg4fl\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.550651 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.549864    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53f53373-f6f8-43db-a686-ab971adec842-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.550651 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.549874    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6522cd84-3c6c-4280-9cb7-02eb8916b24d-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.550651 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.549882    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53f53373-f6f8-43db-a686-ab971adec842-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:09.707392 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.707212    4930 generic.go:356] "Generic (PLEG): container finished" podID="53f53373-f6f8-43db-a686-ab971adec842" containerID="7fe2a79cdbde1663d62b3eaced58e2205eecf109b361f7a1313060c2f04a70dc" exitCode=0
Nov 01 03:42:09.707392 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.707293    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4"
Nov 01 03:42:09.707392 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.707303    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4" event={"ID":"53f53373-f6f8-43db-a686-ab971adec842","Type":"ContainerDied","Data":"7fe2a79cdbde1663d62b3eaced58e2205eecf109b361f7a1313060c2f04a70dc"}
Nov 01 03:42:09.707392 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.707398    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4" event={"ID":"53f53373-f6f8-43db-a686-ab971adec842","Type":"ContainerDied","Data":"e58fd47f953d2bce3a5099092c73a2e880c03b43e379fadcd640ae25fcdaf5ac"}
Nov 01 03:42:09.707822 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.707429    4930 scope.go:122] "RemoveContainer" containerID="7fe2a79cdbde1663d62b3eaced58e2205eecf109b361f7a1313060c2f04a70dc"
Nov 01 03:42:09.709380 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.709355    4930 generic.go:356] "Generic (PLEG): container finished" podID="8c02ff41-a095-4c49-a78c-c78f920a42a4" containerID="7aea0a0f098ea4c0bfcbbfa96e06ef16db65fcfcff65ad262369f16211547e62" exitCode=0
Nov 01 03:42:09.709504 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.709465    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl" event={"ID":"8c02ff41-a095-4c49-a78c-c78f920a42a4","Type":"ContainerDied","Data":"7aea0a0f098ea4c0bfcbbfa96e06ef16db65fcfcff65ad262369f16211547e62"}
Nov 01 03:42:09.709504 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.709499    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl" event={"ID":"8c02ff41-a095-4c49-a78c-c78f920a42a4","Type":"ContainerDied","Data":"e4a9fbbb148d55524a5f04f6135d26c387aba96447dd87b0c6215a39b3a33cd2"}
Nov 01 03:42:09.709659 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.709642    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl"
Nov 01 03:42:09.711969 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.711947    4930 generic.go:356] "Generic (PLEG): container finished" podID="6522cd84-3c6c-4280-9cb7-02eb8916b24d" containerID="b09efc044d9f444cbc90815da3059fbc3c8c18a835c2f06233f51cc908b7a3ed" exitCode=0
Nov 01 03:42:09.712093 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.711980    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss" event={"ID":"6522cd84-3c6c-4280-9cb7-02eb8916b24d","Type":"ContainerDied","Data":"b09efc044d9f444cbc90815da3059fbc3c8c18a835c2f06233f51cc908b7a3ed"}
Nov 01 03:42:09.712093 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.712009    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss" event={"ID":"6522cd84-3c6c-4280-9cb7-02eb8916b24d","Type":"ContainerDied","Data":"f73be621b68afb71cee1ddedd185a5670ca5137e650a6c8cff258d72d9838668"}
Nov 01 03:42:09.712093 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.712064    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss"
Nov 01 03:42:09.721426 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.721407    4930 scope.go:122] "RemoveContainer" containerID="7fe2a79cdbde1663d62b3eaced58e2205eecf109b361f7a1313060c2f04a70dc"
Nov 01 03:42:09.721762 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:42:09.721743    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fe2a79cdbde1663d62b3eaced58e2205eecf109b361f7a1313060c2f04a70dc\": container with ID starting with 7fe2a79cdbde1663d62b3eaced58e2205eecf109b361f7a1313060c2f04a70dc not found: ID does not exist" containerID="7fe2a79cdbde1663d62b3eaced58e2205eecf109b361f7a1313060c2f04a70dc"
Nov 01 03:42:09.721819 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.721771    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fe2a79cdbde1663d62b3eaced58e2205eecf109b361f7a1313060c2f04a70dc"} err="failed to get container status \"7fe2a79cdbde1663d62b3eaced58e2205eecf109b361f7a1313060c2f04a70dc\": rpc error: code = NotFound desc = could not find container \"7fe2a79cdbde1663d62b3eaced58e2205eecf109b361f7a1313060c2f04a70dc\": container with ID starting with 7fe2a79cdbde1663d62b3eaced58e2205eecf109b361f7a1313060c2f04a70dc not found: ID does not exist"
Nov 01 03:42:09.721819 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.721786    4930 scope.go:122] "RemoveContainer" containerID="7aea0a0f098ea4c0bfcbbfa96e06ef16db65fcfcff65ad262369f16211547e62"
Nov 01 03:42:09.731446 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.731426    4930 scope.go:122] "RemoveContainer" containerID="7aea0a0f098ea4c0bfcbbfa96e06ef16db65fcfcff65ad262369f16211547e62"
Nov 01 03:42:09.731859 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:42:09.731840    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aea0a0f098ea4c0bfcbbfa96e06ef16db65fcfcff65ad262369f16211547e62\": container with ID starting with 7aea0a0f098ea4c0bfcbbfa96e06ef16db65fcfcff65ad262369f16211547e62 not found: ID does not exist" containerID="7aea0a0f098ea4c0bfcbbfa96e06ef16db65fcfcff65ad262369f16211547e62"
Nov 01 03:42:09.731917 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.731864    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aea0a0f098ea4c0bfcbbfa96e06ef16db65fcfcff65ad262369f16211547e62"} err="failed to get container status \"7aea0a0f098ea4c0bfcbbfa96e06ef16db65fcfcff65ad262369f16211547e62\": rpc error: code = NotFound desc = could not find container \"7aea0a0f098ea4c0bfcbbfa96e06ef16db65fcfcff65ad262369f16211547e62\": container with ID starting with 7aea0a0f098ea4c0bfcbbfa96e06ef16db65fcfcff65ad262369f16211547e62 not found: ID does not exist"
Nov 01 03:42:09.731917 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.731880    4930 scope.go:122] "RemoveContainer" containerID="b09efc044d9f444cbc90815da3059fbc3c8c18a835c2f06233f51cc908b7a3ed"
Nov 01 03:42:09.742006 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.741986    4930 scope.go:122] "RemoveContainer" containerID="b09efc044d9f444cbc90815da3059fbc3c8c18a835c2f06233f51cc908b7a3ed"
Nov 01 03:42:09.742298 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:42:09.742273    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b09efc044d9f444cbc90815da3059fbc3c8c18a835c2f06233f51cc908b7a3ed\": container with ID starting with b09efc044d9f444cbc90815da3059fbc3c8c18a835c2f06233f51cc908b7a3ed not found: ID does not exist" containerID="b09efc044d9f444cbc90815da3059fbc3c8c18a835c2f06233f51cc908b7a3ed"
Nov 01 03:42:09.742375 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.742302    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b09efc044d9f444cbc90815da3059fbc3c8c18a835c2f06233f51cc908b7a3ed"} err="failed to get container status \"b09efc044d9f444cbc90815da3059fbc3c8c18a835c2f06233f51cc908b7a3ed\": rpc error: code = NotFound desc = could not find container \"b09efc044d9f444cbc90815da3059fbc3c8c18a835c2f06233f51cc908b7a3ed\": container with ID starting with b09efc044d9f444cbc90815da3059fbc3c8c18a835c2f06233f51cc908b7a3ed not found: ID does not exist"
Nov 01 03:42:09.890675 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.890614    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl"]
Nov 01 03:42:09.894580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:09.894529    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-rbxgl"]
Nov 01 03:42:10.000304 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:10.000267    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4"]
Nov 01 03:42:10.004189 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:10.004165    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-tqgp4"]
Nov 01 03:42:10.112550 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:10.112522    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss"]
Nov 01 03:42:10.120354 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:10.120324    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-nvwss"]
Nov 01 03:42:11.623143 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:11.623110    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53f53373-f6f8-43db-a686-ab971adec842" path="/var/lib/kubelet/pods/53f53373-f6f8-43db-a686-ab971adec842/volumes"
Nov 01 03:42:11.623639 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:11.623624    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6522cd84-3c6c-4280-9cb7-02eb8916b24d" path="/var/lib/kubelet/pods/6522cd84-3c6c-4280-9cb7-02eb8916b24d/volumes"
Nov 01 03:42:11.624091 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:11.624077    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c02ff41-a095-4c49-a78c-c78f920a42a4" path="/var/lib/kubelet/pods/8c02ff41-a095-4c49-a78c-c78f920a42a4/volumes"
Nov 01 03:42:12.470758 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:12.470722    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:13.227805 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:13.227768    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:17.404860 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:17.404818    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:18.641440 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:18.641403    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:19.007529 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:19.007446    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:19.155545 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:42:19.155494    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:42:19.309908 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:19.309814    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:19.547056 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:19.546986    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2058fb-zh7wl"]
Nov 01 03:42:19.635651 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:19.635624    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2058fb-zh7wl"]
Nov 01 03:42:19.930130 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:19.930055    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:20.472180 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:20.472146    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:20.882865 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:20.882817    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:21.163177 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:21.163090    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:21.395595 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:21.395544    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-438641-kw7df"]
Nov 01 03:42:21.442817 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:21.442747    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-438641-kw7df"]
Nov 01 03:42:21.621485 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:21.621452    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6346712-2068-4299-90f9-df1fcf150cc7" path="/var/lib/kubelet/pods/b6346712-2068-4299-90f9-df1fcf150cc7/volumes"
Nov 01 03:42:21.622048 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:21.622031    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4434213-7e88-4fff-bcd5-c7eca39696ff" path="/var/lib/kubelet/pods/e4434213-7e88-4fff-bcd5-c7eca39696ff/volumes"
Nov 01 03:42:21.788609 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:21.788511    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:25.754100 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:25.754062    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:26.281787 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:26.281741    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:26.461783 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:26.461745    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:26.675627 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:26.675551    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:27.477335 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:27.477300    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-686153-t2d8c"]
Nov 01 03:42:27.477991 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:27.477973    4930 cpu_manager.go:414] "containerMap: removing container" podUID="2c62951f-01c4-4c25-9267-d741fb752efd" containerName="payload-processing"
Nov 01 03:42:27.478050 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:27.477992    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c62951f-01c4-4c25-9267-d741fb752efd" containerName="payload-processing"
Nov 01 03:42:27.478050 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:27.478009    4930 cpu_manager.go:414] "containerMap: removing container" podUID="6522cd84-3c6c-4280-9cb7-02eb8916b24d" containerName="istio-proxy"
Nov 01 03:42:27.478050 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:27.478015    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6522cd84-3c6c-4280-9cb7-02eb8916b24d" containerName="istio-proxy"
Nov 01 03:42:27.478050 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:27.478026    4930 cpu_manager.go:414] "containerMap: removing container" podUID="8c02ff41-a095-4c49-a78c-c78f920a42a4" containerName="istio-proxy"
Nov 01 03:42:27.478050 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:27.478031    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c02ff41-a095-4c49-a78c-c78f920a42a4" containerName="istio-proxy"
Nov 01 03:42:27.478050 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:27.478044    4930 cpu_manager.go:414] "containerMap: removing container" podUID="58a93bc7-186b-4bfc-970c-045241a5cfd8" containerName="payload-pre-processing"
Nov 01 03:42:27.478050 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:27.478049    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58a93bc7-186b-4bfc-970c-045241a5cfd8" containerName="payload-pre-processing"
Nov 01 03:42:27.478263 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:27.478066    4930 cpu_manager.go:414] "containerMap: removing container" podUID="53f53373-f6f8-43db-a686-ab971adec842" containerName="istio-proxy"
Nov 01 03:42:27.478263 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:27.478071    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53f53373-f6f8-43db-a686-ab971adec842" containerName="istio-proxy"
Nov 01 03:42:27.478263 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:27.478089    4930 cpu_manager.go:414] "containerMap: removing container" podUID="68863127-fc50-48b2-bf3e-c6c99fb4a29e" containerName="istio-proxy"
Nov 01 03:42:27.478263 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:27.478095    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68863127-fc50-48b2-bf3e-c6c99fb4a29e" containerName="istio-proxy"
Nov 01 03:42:27.478263 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:27.478102    4930 cpu_manager.go:414] "containerMap: removing container" podUID="68863127-fc50-48b2-bf3e-c6c99fb4a29e" containerName="istio-proxy"
Nov 01 03:42:27.478263 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:27.478107    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68863127-fc50-48b2-bf3e-c6c99fb4a29e" containerName="istio-proxy"
Nov 01 03:42:27.478263 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:27.478120    4930 cpu_manager.go:414] "containerMap: removing container" podUID="e4434213-7e88-4fff-bcd5-c7eca39696ff" containerName="revoke-keys"
Nov 01 03:42:27.478263 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:27.478124    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4434213-7e88-4fff-bcd5-c7eca39696ff" containerName="revoke-keys"
Nov 01 03:42:27.478263 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:27.478224    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="68863127-fc50-48b2-bf3e-c6c99fb4a29e" containerName="istio-proxy"
Nov 01 03:42:27.478263 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:27.478232    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="68863127-fc50-48b2-bf3e-c6c99fb4a29e" containerName="istio-proxy"
Nov 01 03:42:27.478263 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:27.478241    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4434213-7e88-4fff-bcd5-c7eca39696ff" containerName="revoke-keys"
Nov 01 03:42:27.478263 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:27.478248    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c62951f-01c4-4c25-9267-d741fb752efd" containerName="payload-processing"
Nov 01 03:42:27.478263 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:27.478256    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="6522cd84-3c6c-4280-9cb7-02eb8916b24d" containerName="istio-proxy"
Nov 01 03:42:27.478263 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:27.478262    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c02ff41-a095-4c49-a78c-c78f920a42a4" containerName="istio-proxy"
Nov 01 03:42:27.478263 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:27.478268    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="58a93bc7-186b-4bfc-970c-045241a5cfd8" containerName="payload-pre-processing"
Nov 01 03:42:27.478787 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:27.478275    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="53f53373-f6f8-43db-a686-ab971adec842" containerName="istio-proxy"
Nov 01 03:42:27.482010 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:27.481992    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-686153-t2d8c"
Nov 01 03:42:27.535076 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:27.535042    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:42:27.584816 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:27.584748    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:42:27.584816 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:27.584772    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:42:27.585054 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:27.584862    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:42:27.589210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:27.589184    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-686153-t2d8c"]
Nov 01 03:42:27.658042 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:27.658012    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/814f4d6b-537e-4fe3-8c66-c1606e87d162-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-686153-t2d8c\" (UID: \"814f4d6b-537e-4fe3-8c66-c1606e87d162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-686153-t2d8c"
Nov 01 03:42:27.658197 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:27.658046    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tq89\" (UniqueName: \"kubernetes.io/projected/814f4d6b-537e-4fe3-8c66-c1606e87d162-kube-api-access-6tq89\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-686153-t2d8c\" (UID: \"814f4d6b-537e-4fe3-8c66-c1606e87d162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-686153-t2d8c"
Nov 01 03:42:27.658197 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:27.658075    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/814f4d6b-537e-4fe3-8c66-c1606e87d162-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-686153-t2d8c\" (UID: \"814f4d6b-537e-4fe3-8c66-c1606e87d162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-686153-t2d8c"
Nov 01 03:42:27.760279 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:27.760187    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/814f4d6b-537e-4fe3-8c66-c1606e87d162-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-686153-t2d8c\" (UID: \"814f4d6b-537e-4fe3-8c66-c1606e87d162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-686153-t2d8c"
Nov 01 03:42:27.760279 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:27.760240    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tq89\" (UniqueName: \"kubernetes.io/projected/814f4d6b-537e-4fe3-8c66-c1606e87d162-kube-api-access-6tq89\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-686153-t2d8c\" (UID: \"814f4d6b-537e-4fe3-8c66-c1606e87d162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-686153-t2d8c"
Nov 01 03:42:27.760515 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:27.760285    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/814f4d6b-537e-4fe3-8c66-c1606e87d162-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-686153-t2d8c\" (UID: \"814f4d6b-537e-4fe3-8c66-c1606e87d162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-686153-t2d8c"
Nov 01 03:42:27.761074 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:27.761048    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/814f4d6b-537e-4fe3-8c66-c1606e87d162-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-686153-t2d8c\" (UID: \"814f4d6b-537e-4fe3-8c66-c1606e87d162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-686153-t2d8c"
Nov 01 03:42:27.762952 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:27.762923    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/814f4d6b-537e-4fe3-8c66-c1606e87d162-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-686153-t2d8c\" (UID: \"814f4d6b-537e-4fe3-8c66-c1606e87d162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-686153-t2d8c"
Nov 01 03:42:27.822704 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:27.822681    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tq89\" (UniqueName: \"kubernetes.io/projected/814f4d6b-537e-4fe3-8c66-c1606e87d162-kube-api-access-6tq89\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-686153-t2d8c\" (UID: \"814f4d6b-537e-4fe3-8c66-c1606e87d162\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-686153-t2d8c"
Nov 01 03:42:28.095722 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:28.095687    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-686153-t2d8c"
Nov 01 03:42:28.480712 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:28.480681    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-686153-t2d8c"]
Nov 01 03:42:28.642021 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:28.641928    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:28.870036 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:28.870008    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:28.915052 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:28.914987    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-686153-t2d8c" event={"ID":"814f4d6b-537e-4fe3-8c66-c1606e87d162","Type":"ContainerStarted","Data":"0c497e2b6a9baa51c7324bd3b76f45d6aaa806cd0fe0f52ad420b05723337047"}
Nov 01 03:42:28.915052 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:28.915032    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-686153-t2d8c" event={"ID":"814f4d6b-537e-4fe3-8c66-c1606e87d162","Type":"ContainerStarted","Data":"be030d2d44ef150e7c7c745c8f729f02cdf237a7770e8738fc5eccfe19ad15e4"}
Nov 01 03:42:29.026769 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:29.026726    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:29.030641 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:29.030610    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh"]
Nov 01 03:42:29.030919 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:29.030894    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh" podUID="a0d37273-4961-471d-a3e6-3b48c1cc0acb" containerName="istio-proxy" containerID="cri-o://26c389b331289fee223f580d31fa222247d670b55724104f98aabb7abe0f91ec" gracePeriod=30
Nov 01 03:42:29.034990 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:29.034965    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"]
Nov 01 03:42:29.035204 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:29.035179    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8" podUID="a30e9e3f-be06-449e-84a9-e1e1b8be4d1d" containerName="istio-proxy" containerID="cri-o://6c690034d6f66d2aa5b9954df542badb601e62dc8f3ff86c35bdff923f8753af" gracePeriod=30
Nov 01 03:42:29.050738 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:29.050699    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-686153-t2d8c" podStartSLOduration=2.050687138 podStartE2EDuration="2.050687138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:42:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:42:29.050291507 +0000 UTC m=+2153.573298095" watchObservedRunningTime="2026-11-01 03:42:29.050687138 +0000 UTC m=+2153.573693725"
Nov 01 03:42:29.848291 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:29.848261    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:29.925045 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:29.925011    4930 generic.go:356] "Generic (PLEG): container finished" podID="814f4d6b-537e-4fe3-8c66-c1606e87d162" containerID="0c497e2b6a9baa51c7324bd3b76f45d6aaa806cd0fe0f52ad420b05723337047" exitCode=0
Nov 01 03:42:29.925212 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:29.925101    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-686153-t2d8c" event={"ID":"814f4d6b-537e-4fe3-8c66-c1606e87d162","Type":"ContainerDied","Data":"0c497e2b6a9baa51c7324bd3b76f45d6aaa806cd0fe0f52ad420b05723337047"}
Nov 01 03:42:30.304529 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:30.304429    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:30.632671 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:30.632640    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:30.670577 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:30.659525    4930 patch_prober.go:29] interesting pod/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:42:30.670577 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:30.659654    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8" podUID="a30e9e3f-be06-449e-84a9-e1e1b8be4d1d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:42:30.831593 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:30.831531    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:30.988748 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:30.988705    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:31.104984 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:31.104951    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-686153-t2d8c"
Nov 01 03:42:31.171921 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:31.171892    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/814f4d6b-537e-4fe3-8c66-c1606e87d162-kube-api-access-6tq89\" (UniqueName: \"kubernetes.io/projected/814f4d6b-537e-4fe3-8c66-c1606e87d162-kube-api-access-6tq89\") pod \"814f4d6b-537e-4fe3-8c66-c1606e87d162\" (UID: \"814f4d6b-537e-4fe3-8c66-c1606e87d162\") "
Nov 01 03:42:31.172083 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:31.171994    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/814f4d6b-537e-4fe3-8c66-c1606e87d162-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/814f4d6b-537e-4fe3-8c66-c1606e87d162-maas-api-xks-ca\") pod \"814f4d6b-537e-4fe3-8c66-c1606e87d162\" (UID: \"814f4d6b-537e-4fe3-8c66-c1606e87d162\") "
Nov 01 03:42:31.172134 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:31.172118    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/814f4d6b-537e-4fe3-8c66-c1606e87d162-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/814f4d6b-537e-4fe3-8c66-c1606e87d162-maas-api-openshift-ca\") pod \"814f4d6b-537e-4fe3-8c66-c1606e87d162\" (UID: \"814f4d6b-537e-4fe3-8c66-c1606e87d162\") "
Nov 01 03:42:31.172567 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:31.172528    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/814f4d6b-537e-4fe3-8c66-c1606e87d162-maas-api-openshift-ca" pod "814f4d6b-537e-4fe3-8c66-c1606e87d162" (UID: "814f4d6b-537e-4fe3-8c66-c1606e87d162"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:42:31.172989 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:31.172968    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/814f4d6b-537e-4fe3-8c66-c1606e87d162-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:31.174448 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:31.174420    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/814f4d6b-537e-4fe3-8c66-c1606e87d162-maas-api-xks-ca" pod "814f4d6b-537e-4fe3-8c66-c1606e87d162" (UID: "814f4d6b-537e-4fe3-8c66-c1606e87d162"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:42:31.174641 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:31.174614    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/814f4d6b-537e-4fe3-8c66-c1606e87d162-kube-api-access-6tq89" pod "814f4d6b-537e-4fe3-8c66-c1606e87d162" (UID: "814f4d6b-537e-4fe3-8c66-c1606e87d162"). InnerVolumeSpecName "kube-api-access-6tq89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:42:31.230714 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:31.230683    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:31.276910 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:31.274531    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tq89\" (UniqueName: \"kubernetes.io/projected/814f4d6b-537e-4fe3-8c66-c1606e87d162-kube-api-access-6tq89\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:31.276910 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:31.274588    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/814f4d6b-537e-4fe3-8c66-c1606e87d162-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:31.942395 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:31.942304    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-686153-t2d8c"
Nov 01 03:42:31.942395 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:31.942336    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-686153-t2d8c" event={"ID":"814f4d6b-537e-4fe3-8c66-c1606e87d162","Type":"ContainerDied","Data":"be030d2d44ef150e7c7c745c8f729f02cdf237a7770e8738fc5eccfe19ad15e4"}
Nov 01 03:42:31.942395 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:31.942384    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be030d2d44ef150e7c7c745c8f729f02cdf237a7770e8738fc5eccfe19ad15e4"
Nov 01 03:42:34.343080 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.343055    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh"
Nov 01 03:42:34.381681 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.381642    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"
Nov 01 03:42:34.421155 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.421129    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0d37273-4961-471d-a3e6-3b48c1cc0acb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0d37273-4961-471d-a3e6-3b48c1cc0acb-istiod-ca-cert\") pod \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\" (UID: \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\") "
Nov 01 03:42:34.421329 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.421293    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0d37273-4961-471d-a3e6-3b48c1cc0acb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d37273-4961-471d-a3e6-3b48c1cc0acb-credential-socket\") pod \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\" (UID: \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\") "
Nov 01 03:42:34.421401 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.421335    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0d37273-4961-471d-a3e6-3b48c1cc0acb-kube-api-access-9f2dd\" (UniqueName: \"kubernetes.io/projected/a0d37273-4961-471d-a3e6-3b48c1cc0acb-kube-api-access-9f2dd\") pod \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\" (UID: \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\") "
Nov 01 03:42:34.421401 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.421379    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0d37273-4961-471d-a3e6-3b48c1cc0acb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d37273-4961-471d-a3e6-3b48c1cc0acb-workload-socket\") pod \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\" (UID: \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\") "
Nov 01 03:42:34.421526 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.421414    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0d37273-4961-471d-a3e6-3b48c1cc0acb-istio-token\" (UniqueName: \"kubernetes.io/projected/a0d37273-4961-471d-a3e6-3b48c1cc0acb-istio-token\") pod \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\" (UID: \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\") "
Nov 01 03:42:34.421526 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.421464    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0d37273-4961-471d-a3e6-3b48c1cc0acb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0d37273-4961-471d-a3e6-3b48c1cc0acb-istio-data\") pod \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\" (UID: \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\") "
Nov 01 03:42:34.421660 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.421541    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0d37273-4961-471d-a3e6-3b48c1cc0acb-istiod-ca-cert" pod "a0d37273-4961-471d-a3e6-3b48c1cc0acb" (UID: "a0d37273-4961-471d-a3e6-3b48c1cc0acb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:42:34.421660 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.421587    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0d37273-4961-471d-a3e6-3b48c1cc0acb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0d37273-4961-471d-a3e6-3b48c1cc0acb-workload-certs\") pod \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\" (UID: \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\") "
Nov 01 03:42:34.421660 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.421636    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0d37273-4961-471d-a3e6-3b48c1cc0acb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0d37273-4961-471d-a3e6-3b48c1cc0acb-istio-podinfo\") pod \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\" (UID: \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\") "
Nov 01 03:42:34.421817 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.421717    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0d37273-4961-471d-a3e6-3b48c1cc0acb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0d37273-4961-471d-a3e6-3b48c1cc0acb-istio-envoy\") pod \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\" (UID: \"a0d37273-4961-471d-a3e6-3b48c1cc0acb\") "
Nov 01 03:42:34.422168 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.422054    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d37273-4961-471d-a3e6-3b48c1cc0acb-credential-socket" pod "a0d37273-4961-471d-a3e6-3b48c1cc0acb" (UID: "a0d37273-4961-471d-a3e6-3b48c1cc0acb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:34.422168 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.422073    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d37273-4961-471d-a3e6-3b48c1cc0acb-workload-certs" pod "a0d37273-4961-471d-a3e6-3b48c1cc0acb" (UID: "a0d37273-4961-471d-a3e6-3b48c1cc0acb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:34.422168 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.422134    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d37273-4961-471d-a3e6-3b48c1cc0acb-workload-socket" pod "a0d37273-4961-471d-a3e6-3b48c1cc0acb" (UID: "a0d37273-4961-471d-a3e6-3b48c1cc0acb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:34.422405 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.422331    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d37273-4961-471d-a3e6-3b48c1cc0acb-istio-data" pod "a0d37273-4961-471d-a3e6-3b48c1cc0acb" (UID: "a0d37273-4961-471d-a3e6-3b48c1cc0acb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:34.422631 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.422513    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0d37273-4961-471d-a3e6-3b48c1cc0acb-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:34.422631 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.422542    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d37273-4961-471d-a3e6-3b48c1cc0acb-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:34.422631 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.422574    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d37273-4961-471d-a3e6-3b48c1cc0acb-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:34.422631 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.422589    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0d37273-4961-471d-a3e6-3b48c1cc0acb-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:34.422631 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.422608    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0d37273-4961-471d-a3e6-3b48c1cc0acb-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:34.424217 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.424195    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0d37273-4961-471d-a3e6-3b48c1cc0acb-istio-token" pod "a0d37273-4961-471d-a3e6-3b48c1cc0acb" (UID: "a0d37273-4961-471d-a3e6-3b48c1cc0acb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:42:34.424399 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.424346    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0d37273-4961-471d-a3e6-3b48c1cc0acb-istio-podinfo" pod "a0d37273-4961-471d-a3e6-3b48c1cc0acb" (UID: "a0d37273-4961-471d-a3e6-3b48c1cc0acb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:42:34.424488 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.424470    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0d37273-4961-471d-a3e6-3b48c1cc0acb-kube-api-access-9f2dd" pod "a0d37273-4961-471d-a3e6-3b48c1cc0acb" (UID: "a0d37273-4961-471d-a3e6-3b48c1cc0acb"). InnerVolumeSpecName "kube-api-access-9f2dd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:42:34.424568 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.424489    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d37273-4961-471d-a3e6-3b48c1cc0acb-istio-envoy" pod "a0d37273-4961-471d-a3e6-3b48c1cc0acb" (UID: "a0d37273-4961-471d-a3e6-3b48c1cc0acb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:34.523426 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.523393    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-credential-socket\") pod \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\" (UID: \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\") "
Nov 01 03:42:34.523676 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.523510    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-workload-certs\") pod \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\" (UID: \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\") "
Nov 01 03:42:34.523676 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.523626    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-istio-envoy\") pod \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\" (UID: \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\") "
Nov 01 03:42:34.523832 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.523719    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-istio-token\" (UniqueName: \"kubernetes.io/projected/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-istio-token\") pod \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\" (UID: \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\") "
Nov 01 03:42:34.523832 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.523766    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-credential-socket" pod "a30e9e3f-be06-449e-84a9-e1e1b8be4d1d" (UID: "a30e9e3f-be06-449e-84a9-e1e1b8be4d1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:34.523832 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.523791    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-kube-api-access-8cc4j\" (UniqueName: \"kubernetes.io/projected/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-kube-api-access-8cc4j\") pod \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\" (UID: \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\") "
Nov 01 03:42:34.523832 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.523828    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-istiod-ca-cert\") pod \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\" (UID: \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\") "
Nov 01 03:42:34.524026 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.523879    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-istio-data\") pod \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\" (UID: \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\") "
Nov 01 03:42:34.524026 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.523929    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-workload-certs" pod "a30e9e3f-be06-449e-84a9-e1e1b8be4d1d" (UID: "a30e9e3f-be06-449e-84a9-e1e1b8be4d1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:34.524026 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.523944    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-istio-podinfo\") pod \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\" (UID: \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\") "
Nov 01 03:42:34.524168 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.524024    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-workload-socket\") pod \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\" (UID: \"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d\") "
Nov 01 03:42:34.524257 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.524234    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-istio-data" pod "a30e9e3f-be06-449e-84a9-e1e1b8be4d1d" (UID: "a30e9e3f-be06-449e-84a9-e1e1b8be4d1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:34.524909 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.524679    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f2dd\" (UniqueName: \"kubernetes.io/projected/a0d37273-4961-471d-a3e6-3b48c1cc0acb-kube-api-access-9f2dd\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:34.524909 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.524706    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:34.524909 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.524723    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0d37273-4961-471d-a3e6-3b48c1cc0acb-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:34.524909 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.524738    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:34.524909 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.524753    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0d37273-4961-471d-a3e6-3b48c1cc0acb-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:34.524909 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.524767    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0d37273-4961-471d-a3e6-3b48c1cc0acb-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:34.524909 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.524781    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:34.524909 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.524849    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-workload-socket" pod "a30e9e3f-be06-449e-84a9-e1e1b8be4d1d" (UID: "a30e9e3f-be06-449e-84a9-e1e1b8be4d1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:34.524909 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.524850    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-istiod-ca-cert" pod "a30e9e3f-be06-449e-84a9-e1e1b8be4d1d" (UID: "a30e9e3f-be06-449e-84a9-e1e1b8be4d1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:42:34.526181 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.526144    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-istio-envoy" pod "a30e9e3f-be06-449e-84a9-e1e1b8be4d1d" (UID: "a30e9e3f-be06-449e-84a9-e1e1b8be4d1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:42:34.526411 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.526392    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-istio-token" pod "a30e9e3f-be06-449e-84a9-e1e1b8be4d1d" (UID: "a30e9e3f-be06-449e-84a9-e1e1b8be4d1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:42:34.526978 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.526954    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-istio-podinfo" pod "a30e9e3f-be06-449e-84a9-e1e1b8be4d1d" (UID: "a30e9e3f-be06-449e-84a9-e1e1b8be4d1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:42:34.527100 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.527081    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-kube-api-access-8cc4j" pod "a30e9e3f-be06-449e-84a9-e1e1b8be4d1d" (UID: "a30e9e3f-be06-449e-84a9-e1e1b8be4d1d"). InnerVolumeSpecName "kube-api-access-8cc4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:42:34.703918 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.701518    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:34.703918 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.701578    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:34.703918 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.701595    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:34.703918 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.701611    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cc4j\" (UniqueName: \"kubernetes.io/projected/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-kube-api-access-8cc4j\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:34.703918 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.701628    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:34.703918 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.701641    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:34.968862 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.968761    4930 generic.go:356] "Generic (PLEG): container finished" podID="a0d37273-4961-471d-a3e6-3b48c1cc0acb" containerID="26c389b331289fee223f580d31fa222247d670b55724104f98aabb7abe0f91ec" exitCode=0
Nov 01 03:42:34.968862 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.968814    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh"
Nov 01 03:42:34.968862 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.968846    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh" event={"ID":"a0d37273-4961-471d-a3e6-3b48c1cc0acb","Type":"ContainerDied","Data":"26c389b331289fee223f580d31fa222247d670b55724104f98aabb7abe0f91ec"}
Nov 01 03:42:34.969149 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.968907    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh" event={"ID":"a0d37273-4961-471d-a3e6-3b48c1cc0acb","Type":"ContainerDied","Data":"3094de6a14a30413d34d67451711d3cd511eaf434aff879dd33cb88ea3b239f2"}
Nov 01 03:42:34.969149 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.968943    4930 scope.go:122] "RemoveContainer" containerID="26c389b331289fee223f580d31fa222247d670b55724104f98aabb7abe0f91ec"
Nov 01 03:42:34.971372 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.971345    4930 generic.go:356] "Generic (PLEG): container finished" podID="a30e9e3f-be06-449e-84a9-e1e1b8be4d1d" containerID="6c690034d6f66d2aa5b9954df542badb601e62dc8f3ff86c35bdff923f8753af" exitCode=0
Nov 01 03:42:34.971525 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.971408    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8" event={"ID":"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d","Type":"ContainerDied","Data":"6c690034d6f66d2aa5b9954df542badb601e62dc8f3ff86c35bdff923f8753af"}
Nov 01 03:42:34.971525 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.971419    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"
Nov 01 03:42:34.971525 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.971451    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8" event={"ID":"a30e9e3f-be06-449e-84a9-e1e1b8be4d1d","Type":"ContainerDied","Data":"92e72d774936e9ce7b022fa14cdab8af4b94cd83a2f2c562795b051a50dee4f7"}
Nov 01 03:42:34.982115 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.982091    4930 scope.go:122] "RemoveContainer" containerID="26c389b331289fee223f580d31fa222247d670b55724104f98aabb7abe0f91ec"
Nov 01 03:42:34.982435 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:42:34.982412    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26c389b331289fee223f580d31fa222247d670b55724104f98aabb7abe0f91ec\": container with ID starting with 26c389b331289fee223f580d31fa222247d670b55724104f98aabb7abe0f91ec not found: ID does not exist" containerID="26c389b331289fee223f580d31fa222247d670b55724104f98aabb7abe0f91ec"
Nov 01 03:42:34.982515 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.982445    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26c389b331289fee223f580d31fa222247d670b55724104f98aabb7abe0f91ec"} err="failed to get container status \"26c389b331289fee223f580d31fa222247d670b55724104f98aabb7abe0f91ec\": rpc error: code = NotFound desc = could not find container \"26c389b331289fee223f580d31fa222247d670b55724104f98aabb7abe0f91ec\": container with ID starting with 26c389b331289fee223f580d31fa222247d670b55724104f98aabb7abe0f91ec not found: ID does not exist"
Nov 01 03:42:34.982515 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.982468    4930 scope.go:122] "RemoveContainer" containerID="6c690034d6f66d2aa5b9954df542badb601e62dc8f3ff86c35bdff923f8753af"
Nov 01 03:42:34.995305 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:34.995287    4930 scope.go:122] "RemoveContainer" containerID="190679fadf2457c52f69239758b5f78a2f5b923dfef156f24191b6a81729fe1b"
Nov 01 03:42:35.007693 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:35.007672    4930 scope.go:122] "RemoveContainer" containerID="6c690034d6f66d2aa5b9954df542badb601e62dc8f3ff86c35bdff923f8753af"
Nov 01 03:42:35.007931 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:42:35.007913    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c690034d6f66d2aa5b9954df542badb601e62dc8f3ff86c35bdff923f8753af\": container with ID starting with 6c690034d6f66d2aa5b9954df542badb601e62dc8f3ff86c35bdff923f8753af not found: ID does not exist" containerID="6c690034d6f66d2aa5b9954df542badb601e62dc8f3ff86c35bdff923f8753af"
Nov 01 03:42:35.008013 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:35.007943    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c690034d6f66d2aa5b9954df542badb601e62dc8f3ff86c35bdff923f8753af"} err="failed to get container status \"6c690034d6f66d2aa5b9954df542badb601e62dc8f3ff86c35bdff923f8753af\": rpc error: code = NotFound desc = could not find container \"6c690034d6f66d2aa5b9954df542badb601e62dc8f3ff86c35bdff923f8753af\": container with ID starting with 6c690034d6f66d2aa5b9954df542badb601e62dc8f3ff86c35bdff923f8753af not found: ID does not exist"
Nov 01 03:42:35.008013 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:35.007969    4930 scope.go:122] "RemoveContainer" containerID="190679fadf2457c52f69239758b5f78a2f5b923dfef156f24191b6a81729fe1b"
Nov 01 03:42:35.008223 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:42:35.008202    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"190679fadf2457c52f69239758b5f78a2f5b923dfef156f24191b6a81729fe1b\": container with ID starting with 190679fadf2457c52f69239758b5f78a2f5b923dfef156f24191b6a81729fe1b not found: ID does not exist" containerID="190679fadf2457c52f69239758b5f78a2f5b923dfef156f24191b6a81729fe1b"
Nov 01 03:42:35.008282 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:35.008231    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"190679fadf2457c52f69239758b5f78a2f5b923dfef156f24191b6a81729fe1b"} err="failed to get container status \"190679fadf2457c52f69239758b5f78a2f5b923dfef156f24191b6a81729fe1b\": rpc error: code = NotFound desc = could not find container \"190679fadf2457c52f69239758b5f78a2f5b923dfef156f24191b6a81729fe1b\": container with ID starting with 190679fadf2457c52f69239758b5f78a2f5b923dfef156f24191b6a81729fe1b not found: ID does not exist"
Nov 01 03:42:35.056127 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:35.056093    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:35.138985 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:35.138957    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh"]
Nov 01 03:42:35.188186 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:35.188155    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-gxjlh"]
Nov 01 03:42:35.227826 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:35.227733    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:35.261471 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:35.261444    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"]
Nov 01 03:42:35.279274 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:35.279246    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-438641-openshift-default-974f6f5d5-bkhr8"]
Nov 01 03:42:35.373678 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:35.373642    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:35.631008 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:35.630975    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0d37273-4961-471d-a3e6-3b48c1cc0acb" path="/var/lib/kubelet/pods/a0d37273-4961-471d-a3e6-3b48c1cc0acb/volumes"
Nov 01 03:42:35.631487 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:35.631471    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a30e9e3f-be06-449e-84a9-e1e1b8be4d1d" path="/var/lib/kubelet/pods/a30e9e3f-be06-449e-84a9-e1e1b8be4d1d/volumes"
Nov 01 03:42:36.473768 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:36.473730    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:38.980386 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:38.980358    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:39.347804 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:39.347773    4930 scope.go:122] "RemoveContainer" containerID="244a5506ec20e7a6907fa7e303a54d16f6d4076b95af5826c92cb350092fe42d"
Nov 01 03:42:39.429105 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:39.429074    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:39.828034 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:39.827941    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:40.465479 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:40.465447    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:40.625629 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:40.622322    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-686153-76d64ff869-c69fk"]
Nov 01 03:42:40.657404 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:40.657365    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-686153-78c947b5ff-hcd4s"]
Nov 01 03:42:40.733373 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:40.733286    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:41.020349 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:41.020227    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:41.166964 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:41.166926    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:42.892412 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:42.892366    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:43.047688 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:43.047651    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:43.244736 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:43.244647    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:44.995973 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:44.995926    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:45.174232 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:45.174176    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:45.324721 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:45.324638    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:45.626614 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:45.626578    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-686153-76d64ff869-c69fk" podUID="97b6f511-4996-4752-887c-75bee107d98b" containerName="payload-processing" containerID="cri-o://02add003872f691767f15c5f508aebd4144de22a899943be6e2727c2f64efbe8" gracePeriod=25
Nov 01 03:42:45.658626 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:45.658533    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-686153-78c947b5ff-hcd4s" podUID="11d7b556-9fe7-4910-9e18-8f8db5995f80" containerName="payload-pre-processing" containerID="cri-o://38db2cd143d25e22ffbde90415d0d49a53447b755ca60e3d7e041a3c5ca61eb9" gracePeriod=25
Nov 01 03:42:46.035599 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:46.034448    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-686153-76d64ff869-c69fk"
Nov 01 03:42:46.066385 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:46.066362    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-686153-78c947b5ff-hcd4s"
Nov 01 03:42:46.117216 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:46.117188    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97b6f511-4996-4752-887c-75bee107d98b-kube-api-access-z6bhv\" (UniqueName: \"kubernetes.io/projected/97b6f511-4996-4752-887c-75bee107d98b-kube-api-access-z6bhv\") pod \"97b6f511-4996-4752-887c-75bee107d98b\" (UID: \"97b6f511-4996-4752-887c-75bee107d98b\") "
Nov 01 03:42:46.117437 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:46.117230    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97b6f511-4996-4752-887c-75bee107d98b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b6f511-4996-4752-887c-75bee107d98b-plugins-config-volume\") pod \"97b6f511-4996-4752-887c-75bee107d98b\" (UID: \"97b6f511-4996-4752-887c-75bee107d98b\") "
Nov 01 03:42:46.117437 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:46.117257    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11d7b556-9fe7-4910-9e18-8f8db5995f80-kube-api-access-5qbn2\" (UniqueName: \"kubernetes.io/projected/11d7b556-9fe7-4910-9e18-8f8db5995f80-kube-api-access-5qbn2\") pod \"11d7b556-9fe7-4910-9e18-8f8db5995f80\" (UID: \"11d7b556-9fe7-4910-9e18-8f8db5995f80\") "
Nov 01 03:42:46.117683 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:46.117660    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97b6f511-4996-4752-887c-75bee107d98b-plugins-config-volume" pod "97b6f511-4996-4752-887c-75bee107d98b" (UID: "97b6f511-4996-4752-887c-75bee107d98b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:42:46.119484 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:46.119460    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97b6f511-4996-4752-887c-75bee107d98b-kube-api-access-z6bhv" pod "97b6f511-4996-4752-887c-75bee107d98b" (UID: "97b6f511-4996-4752-887c-75bee107d98b"). InnerVolumeSpecName "kube-api-access-z6bhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:42:46.119613 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:46.119483    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11d7b556-9fe7-4910-9e18-8f8db5995f80-kube-api-access-5qbn2" pod "11d7b556-9fe7-4910-9e18-8f8db5995f80" (UID: "11d7b556-9fe7-4910-9e18-8f8db5995f80"). InnerVolumeSpecName "kube-api-access-5qbn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:42:46.152764 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:46.152668    4930 generic.go:356] "Generic (PLEG): container finished" podID="97b6f511-4996-4752-887c-75bee107d98b" containerID="02add003872f691767f15c5f508aebd4144de22a899943be6e2727c2f64efbe8" exitCode=0
Nov 01 03:42:46.152764 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:46.152717    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-686153-76d64ff869-c69fk"
Nov 01 03:42:46.152764 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:46.152752    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-686153-76d64ff869-c69fk" event={"ID":"97b6f511-4996-4752-887c-75bee107d98b","Type":"ContainerDied","Data":"02add003872f691767f15c5f508aebd4144de22a899943be6e2727c2f64efbe8"}
Nov 01 03:42:46.153019 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:46.152838    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-686153-76d64ff869-c69fk" event={"ID":"97b6f511-4996-4752-887c-75bee107d98b","Type":"ContainerDied","Data":"7e2eed8fec5671ea401ee02a17d60bd00e53fa740d6c7574cf8c61bf930c5674"}
Nov 01 03:42:46.153019 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:46.152865    4930 scope.go:122] "RemoveContainer" containerID="02add003872f691767f15c5f508aebd4144de22a899943be6e2727c2f64efbe8"
Nov 01 03:42:46.155164 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:46.155135    4930 generic.go:356] "Generic (PLEG): container finished" podID="11d7b556-9fe7-4910-9e18-8f8db5995f80" containerID="38db2cd143d25e22ffbde90415d0d49a53447b755ca60e3d7e041a3c5ca61eb9" exitCode=0
Nov 01 03:42:46.155477 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:46.155164    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-686153-78c947b5ff-hcd4s" event={"ID":"11d7b556-9fe7-4910-9e18-8f8db5995f80","Type":"ContainerDied","Data":"38db2cd143d25e22ffbde90415d0d49a53447b755ca60e3d7e041a3c5ca61eb9"}
Nov 01 03:42:46.155477 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:46.155219    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-686153-78c947b5ff-hcd4s"
Nov 01 03:42:46.155477 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:46.155233    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-686153-78c947b5ff-hcd4s" event={"ID":"11d7b556-9fe7-4910-9e18-8f8db5995f80","Type":"ContainerDied","Data":"874926fd8f2dba8c99f64ef51d589c5d197d5d235aabac750c98be47249e2c29"}
Nov 01 03:42:46.166459 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:46.166438    4930 scope.go:122] "RemoveContainer" containerID="02add003872f691767f15c5f508aebd4144de22a899943be6e2727c2f64efbe8"
Nov 01 03:42:46.166950 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:42:46.166921    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02add003872f691767f15c5f508aebd4144de22a899943be6e2727c2f64efbe8\": container with ID starting with 02add003872f691767f15c5f508aebd4144de22a899943be6e2727c2f64efbe8 not found: ID does not exist" containerID="02add003872f691767f15c5f508aebd4144de22a899943be6e2727c2f64efbe8"
Nov 01 03:42:46.167049 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:46.166957    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02add003872f691767f15c5f508aebd4144de22a899943be6e2727c2f64efbe8"} err="failed to get container status \"02add003872f691767f15c5f508aebd4144de22a899943be6e2727c2f64efbe8\": rpc error: code = NotFound desc = could not find container \"02add003872f691767f15c5f508aebd4144de22a899943be6e2727c2f64efbe8\": container with ID starting with 02add003872f691767f15c5f508aebd4144de22a899943be6e2727c2f64efbe8 not found: ID does not exist"
Nov 01 03:42:46.167049 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:46.166981    4930 scope.go:122] "RemoveContainer" containerID="38db2cd143d25e22ffbde90415d0d49a53447b755ca60e3d7e041a3c5ca61eb9"
Nov 01 03:42:46.184235 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:46.184213    4930 scope.go:122] "RemoveContainer" containerID="38db2cd143d25e22ffbde90415d0d49a53447b755ca60e3d7e041a3c5ca61eb9"
Nov 01 03:42:46.184707 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:42:46.184678    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38db2cd143d25e22ffbde90415d0d49a53447b755ca60e3d7e041a3c5ca61eb9\": container with ID starting with 38db2cd143d25e22ffbde90415d0d49a53447b755ca60e3d7e041a3c5ca61eb9 not found: ID does not exist" containerID="38db2cd143d25e22ffbde90415d0d49a53447b755ca60e3d7e041a3c5ca61eb9"
Nov 01 03:42:46.184820 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:46.184714    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38db2cd143d25e22ffbde90415d0d49a53447b755ca60e3d7e041a3c5ca61eb9"} err="failed to get container status \"38db2cd143d25e22ffbde90415d0d49a53447b755ca60e3d7e041a3c5ca61eb9\": rpc error: code = NotFound desc = could not find container \"38db2cd143d25e22ffbde90415d0d49a53447b755ca60e3d7e041a3c5ca61eb9\": container with ID starting with 38db2cd143d25e22ffbde90415d0d49a53447b755ca60e3d7e041a3c5ca61eb9 not found: ID does not exist"
Nov 01 03:42:46.218843 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:46.218820    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11d7b556-9fe7-4910-9e18-8f8db5995f80-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11d7b556-9fe7-4910-9e18-8f8db5995f80-plugins-config-volume\") pod \"11d7b556-9fe7-4910-9e18-8f8db5995f80\" (UID: \"11d7b556-9fe7-4910-9e18-8f8db5995f80\") "
Nov 01 03:42:46.219154 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:46.219135    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11d7b556-9fe7-4910-9e18-8f8db5995f80-plugins-config-volume" pod "11d7b556-9fe7-4910-9e18-8f8db5995f80" (UID: "11d7b556-9fe7-4910-9e18-8f8db5995f80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:42:46.219335 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:46.219322    4930 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11d7b556-9fe7-4910-9e18-8f8db5995f80-plugins-config-volume\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:46.219376 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:46.219338    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6bhv\" (UniqueName: \"kubernetes.io/projected/97b6f511-4996-4752-887c-75bee107d98b-kube-api-access-z6bhv\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:46.219376 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:46.219349    4930 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b6f511-4996-4752-887c-75bee107d98b-plugins-config-volume\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:46.219376 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:46.219358    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qbn2\" (UniqueName: \"kubernetes.io/projected/11d7b556-9fe7-4910-9e18-8f8db5995f80-kube-api-access-5qbn2\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:42:46.296038 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:46.295980    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:46.433693 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:46.433610    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-686153-76d64ff869-c69fk"]
Nov 01 03:42:46.437930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:46.437906    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-686153-76d64ff869-c69fk"]
Nov 01 03:42:46.583366 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:46.583328    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-686153-78c947b5ff-hcd4s"]
Nov 01 03:42:46.590306 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:46.590275    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-686153-78c947b5ff-hcd4s"]
Nov 01 03:42:46.725901 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:46.725803    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:46.937148 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:46.935455    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:47.206766 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:47.206611    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:47.425581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:47.421264    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:47.623630 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:47.623596    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11d7b556-9fe7-4910-9e18-8f8db5995f80" path="/var/lib/kubelet/pods/11d7b556-9fe7-4910-9e18-8f8db5995f80/volumes"
Nov 01 03:42:47.624255 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:47.624234    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97b6f511-4996-4752-887c-75bee107d98b" path="/var/lib/kubelet/pods/97b6f511-4996-4752-887c-75bee107d98b/volumes"
Nov 01 03:42:48.394279 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:48.394243    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:48.553605 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:48.553568    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:48.818661 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:48.818534    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:49.122474 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:42:49.122441    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:42:51.891264 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:51.891226    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:52.101328 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:52.101295    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:52.256830 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:52.256743    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:53.497902 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:53.497856    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:53.681931 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:53.680377    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:53.837272 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:53.837235    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:53.951944 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:53.951908    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-686153-t2d8c"]
Nov 01 03:42:54.001440 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:54.001406    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-686153-t2d8c"]
Nov 01 03:42:54.008021 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:54.007989    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:54.367583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:54.367531    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:42:55.632779 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:55.632745    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="814f4d6b-537e-4fe3-8c66-c1606e87d162" path="/var/lib/kubelet/pods/814f4d6b-537e-4fe3-8c66-c1606e87d162/volumes"
Nov 01 03:42:59.280328 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:42:59.280284    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:43:01.902001 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:01.901963    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:43:02.105600 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:02.105568    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:43:02.258494 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:02.258398    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:43:02.259234 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:02.259202    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"]
Nov 01 03:43:02.259516 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:02.259475    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8" podUID="b941e0fe-6b17-4e30-917c-4d3f4382c545" containerName="istio-proxy" containerID="cri-o://531d96c3fec86de40cc71a6cb46e25e1287e0820759da0190f74ec879ee8f3af" gracePeriod=30
Nov 01 03:43:02.719394 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:02.719357    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:43:03.259754 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:03.259722    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:43:03.432798 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:03.432755    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:43:03.691602 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:03.691570    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e8c789-frgzj"]
Nov 01 03:43:03.692197 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:03.692181    4930 cpu_manager.go:414] "containerMap: removing container" podUID="a30e9e3f-be06-449e-84a9-e1e1b8be4d1d" containerName="istio-proxy"
Nov 01 03:43:03.692252 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:03.692199    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a30e9e3f-be06-449e-84a9-e1e1b8be4d1d" containerName="istio-proxy"
Nov 01 03:43:03.692252 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:03.692213    4930 cpu_manager.go:414] "containerMap: removing container" podUID="814f4d6b-537e-4fe3-8c66-c1606e87d162" containerName="revoke-keys"
Nov 01 03:43:03.692252 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:03.692218    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="814f4d6b-537e-4fe3-8c66-c1606e87d162" containerName="revoke-keys"
Nov 01 03:43:03.692252 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:03.692238    4930 cpu_manager.go:414] "containerMap: removing container" podUID="a0d37273-4961-471d-a3e6-3b48c1cc0acb" containerName="istio-proxy"
Nov 01 03:43:03.692252 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:03.692244    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0d37273-4961-471d-a3e6-3b48c1cc0acb" containerName="istio-proxy"
Nov 01 03:43:03.692252 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:03.692253    4930 cpu_manager.go:414] "containerMap: removing container" podUID="97b6f511-4996-4752-887c-75bee107d98b" containerName="payload-processing"
Nov 01 03:43:03.692478 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:03.692261    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97b6f511-4996-4752-887c-75bee107d98b" containerName="payload-processing"
Nov 01 03:43:03.692478 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:03.692274    4930 cpu_manager.go:414] "containerMap: removing container" podUID="11d7b556-9fe7-4910-9e18-8f8db5995f80" containerName="payload-pre-processing"
Nov 01 03:43:03.692478 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:03.692279    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11d7b556-9fe7-4910-9e18-8f8db5995f80" containerName="payload-pre-processing"
Nov 01 03:43:03.692478 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:03.692380    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="a30e9e3f-be06-449e-84a9-e1e1b8be4d1d" containerName="istio-proxy"
Nov 01 03:43:03.692478 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:03.692389    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="814f4d6b-537e-4fe3-8c66-c1606e87d162" containerName="revoke-keys"
Nov 01 03:43:03.692478 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:03.692400    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0d37273-4961-471d-a3e6-3b48c1cc0acb" containerName="istio-proxy"
Nov 01 03:43:03.692478 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:03.692407    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="97b6f511-4996-4752-887c-75bee107d98b" containerName="payload-processing"
Nov 01 03:43:03.692478 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:03.692414    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="11d7b556-9fe7-4910-9e18-8f8db5995f80" containerName="payload-pre-processing"
Nov 01 03:43:03.696060 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:03.696039    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e8c789-frgzj"
Nov 01 03:43:03.748570 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:03.748505    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:43:03.782136 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:03.763492    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61310c85-114a-46f0-9a4d-daa629c4f4d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e8c789-frgzj\" (UID: \"61310c85-114a-46f0-9a4d-daa629c4f4d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e8c789-frgzj"
Nov 01 03:43:03.782136 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:03.763587    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61310c85-114a-46f0-9a4d-daa629c4f4d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e8c789-frgzj\" (UID: \"61310c85-114a-46f0-9a4d-daa629c4f4d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e8c789-frgzj"
Nov 01 03:43:03.782136 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:03.763751    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs69q\" (UniqueName: \"kubernetes.io/projected/61310c85-114a-46f0-9a4d-daa629c4f4d4-kube-api-access-vs69q\") pod \"maas-api-revoke-keys-e2e-worker-w4-e8c789-frgzj\" (UID: \"61310c85-114a-46f0-9a4d-daa629c4f4d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e8c789-frgzj"
Nov 01 03:43:03.813579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:03.804873    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:43:03.813579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:03.805030    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:43:03.813579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:03.805086    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:43:03.849580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:03.842161    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e8c789-frgzj"]
Nov 01 03:43:03.905588 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:03.905051    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs69q\" (UniqueName: \"kubernetes.io/projected/61310c85-114a-46f0-9a4d-daa629c4f4d4-kube-api-access-vs69q\") pod \"maas-api-revoke-keys-e2e-worker-w4-e8c789-frgzj\" (UID: \"61310c85-114a-46f0-9a4d-daa629c4f4d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e8c789-frgzj"
Nov 01 03:43:03.905588 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:03.905248    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61310c85-114a-46f0-9a4d-daa629c4f4d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e8c789-frgzj\" (UID: \"61310c85-114a-46f0-9a4d-daa629c4f4d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e8c789-frgzj"
Nov 01 03:43:03.905588 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:03.905287    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61310c85-114a-46f0-9a4d-daa629c4f4d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e8c789-frgzj\" (UID: \"61310c85-114a-46f0-9a4d-daa629c4f4d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e8c789-frgzj"
Nov 01 03:43:03.923314 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:03.906137    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61310c85-114a-46f0-9a4d-daa629c4f4d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e8c789-frgzj\" (UID: \"61310c85-114a-46f0-9a4d-daa629c4f4d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e8c789-frgzj"
Nov 01 03:43:03.933152 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:03.933118    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61310c85-114a-46f0-9a4d-daa629c4f4d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e8c789-frgzj\" (UID: \"61310c85-114a-46f0-9a4d-daa629c4f4d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e8c789-frgzj"
Nov 01 03:43:03.967700 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:03.967626    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs69q\" (UniqueName: \"kubernetes.io/projected/61310c85-114a-46f0-9a4d-daa629c4f4d4-kube-api-access-vs69q\") pod \"maas-api-revoke-keys-e2e-worker-w4-e8c789-frgzj\" (UID: \"61310c85-114a-46f0-9a4d-daa629c4f4d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e8c789-frgzj"
Nov 01 03:43:04.011528 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:04.011502    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e8c789-frgzj"
Nov 01 03:43:04.395836 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:43:04.395796    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61310c85_114a_46f0_9a4d_daa629c4f4d4.slice/crio-efed1e14638fb1ee5b43089a737f1ba1cffb22dd8566e5dc2f5306a279e5fb57 WatchSource:0}: Error finding container efed1e14638fb1ee5b43089a737f1ba1cffb22dd8566e5dc2f5306a279e5fb57: Status 404 returned error can't find the container with id efed1e14638fb1ee5b43089a737f1ba1cffb22dd8566e5dc2f5306a279e5fb57
Nov 01 03:43:04.396293 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:04.396043    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e8c789-frgzj"]
Nov 01 03:43:04.645761 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:04.645674    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:43:05.372470 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:05.372440    4930 generic.go:356] "Generic (PLEG): container finished" podID="61310c85-114a-46f0-9a4d-daa629c4f4d4" containerID="4868f7063b8326bd6f853bf6be38c43d6c1ae43d4f17cee2bc2ade3a2bb3ee7a" exitCode=0
Nov 01 03:43:05.372675 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:05.372524    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e8c789-frgzj" event={"ID":"61310c85-114a-46f0-9a4d-daa629c4f4d4","Type":"ContainerDied","Data":"4868f7063b8326bd6f853bf6be38c43d6c1ae43d4f17cee2bc2ade3a2bb3ee7a"}
Nov 01 03:43:05.372675 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:05.372632    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e8c789-frgzj" event={"ID":"61310c85-114a-46f0-9a4d-daa629c4f4d4","Type":"ContainerStarted","Data":"efed1e14638fb1ee5b43089a737f1ba1cffb22dd8566e5dc2f5306a279e5fb57"}
Nov 01 03:43:06.557130 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:06.557103    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e8c789-frgzj"
Nov 01 03:43:06.640509 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:06.640429    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61310c85-114a-46f0-9a4d-daa629c4f4d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61310c85-114a-46f0-9a4d-daa629c4f4d4-maas-api-xks-ca\") pod \"61310c85-114a-46f0-9a4d-daa629c4f4d4\" (UID: \"61310c85-114a-46f0-9a4d-daa629c4f4d4\") "
Nov 01 03:43:06.640677 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:06.640533    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61310c85-114a-46f0-9a4d-daa629c4f4d4-kube-api-access-vs69q\" (UniqueName: \"kubernetes.io/projected/61310c85-114a-46f0-9a4d-daa629c4f4d4-kube-api-access-vs69q\") pod \"61310c85-114a-46f0-9a4d-daa629c4f4d4\" (UID: \"61310c85-114a-46f0-9a4d-daa629c4f4d4\") "
Nov 01 03:43:06.640677 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:06.640627    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61310c85-114a-46f0-9a4d-daa629c4f4d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61310c85-114a-46f0-9a4d-daa629c4f4d4-maas-api-openshift-ca\") pod \"61310c85-114a-46f0-9a4d-daa629c4f4d4\" (UID: \"61310c85-114a-46f0-9a4d-daa629c4f4d4\") "
Nov 01 03:43:06.640962 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:06.640939    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61310c85-114a-46f0-9a4d-daa629c4f4d4-maas-api-openshift-ca" pod "61310c85-114a-46f0-9a4d-daa629c4f4d4" (UID: "61310c85-114a-46f0-9a4d-daa629c4f4d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:43:06.641294 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:06.641275    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61310c85-114a-46f0-9a4d-daa629c4f4d4-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:43:06.642761 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:06.642729    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61310c85-114a-46f0-9a4d-daa629c4f4d4-maas-api-xks-ca" pod "61310c85-114a-46f0-9a4d-daa629c4f4d4" (UID: "61310c85-114a-46f0-9a4d-daa629c4f4d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:43:06.642876 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:06.642818    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61310c85-114a-46f0-9a4d-daa629c4f4d4-kube-api-access-vs69q" pod "61310c85-114a-46f0-9a4d-daa629c4f4d4" (UID: "61310c85-114a-46f0-9a4d-daa629c4f4d4"). InnerVolumeSpecName "kube-api-access-vs69q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:43:06.742379 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:06.742349    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs69q\" (UniqueName: \"kubernetes.io/projected/61310c85-114a-46f0-9a4d-daa629c4f4d4-kube-api-access-vs69q\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:43:06.742379 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:06.742378    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61310c85-114a-46f0-9a4d-daa629c4f4d4-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:43:07.390532 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:07.390500    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e8c789-frgzj"
Nov 01 03:43:07.390715 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:07.390510    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e8c789-frgzj" event={"ID":"61310c85-114a-46f0-9a4d-daa629c4f4d4","Type":"ContainerDied","Data":"efed1e14638fb1ee5b43089a737f1ba1cffb22dd8566e5dc2f5306a279e5fb57"}
Nov 01 03:43:07.390715 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:07.390620    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efed1e14638fb1ee5b43089a737f1ba1cffb22dd8566e5dc2f5306a279e5fb57"
Nov 01 03:43:07.402049 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:07.402017    4930 generic.go:356] "Generic (PLEG): container finished" podID="b941e0fe-6b17-4e30-917c-4d3f4382c545" containerID="531d96c3fec86de40cc71a6cb46e25e1287e0820759da0190f74ec879ee8f3af" exitCode=0
Nov 01 03:43:07.402190 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:07.402125    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8" event={"ID":"b941e0fe-6b17-4e30-917c-4d3f4382c545","Type":"ContainerDied","Data":"531d96c3fec86de40cc71a6cb46e25e1287e0820759da0190f74ec879ee8f3af"}
Nov 01 03:43:07.402190 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:07.402179    4930 scope.go:122] "RemoveContainer" containerID="a1681e86692725e01cda7745ec555ae99eb41f125053fd0d010bb18028603f3f"
Nov 01 03:43:07.550543 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:07.550516    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"
Nov 01 03:43:07.654979 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:07.654950    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b941e0fe-6b17-4e30-917c-4d3f4382c545-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b941e0fe-6b17-4e30-917c-4d3f4382c545-istio-data\") pod \"b941e0fe-6b17-4e30-917c-4d3f4382c545\" (UID: \"b941e0fe-6b17-4e30-917c-4d3f4382c545\") "
Nov 01 03:43:07.655440 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:07.655001    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b941e0fe-6b17-4e30-917c-4d3f4382c545-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b941e0fe-6b17-4e30-917c-4d3f4382c545-workload-certs\") pod \"b941e0fe-6b17-4e30-917c-4d3f4382c545\" (UID: \"b941e0fe-6b17-4e30-917c-4d3f4382c545\") "
Nov 01 03:43:07.655440 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:07.655041    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b941e0fe-6b17-4e30-917c-4d3f4382c545-kube-api-access-nfvq5\" (UniqueName: \"kubernetes.io/projected/b941e0fe-6b17-4e30-917c-4d3f4382c545-kube-api-access-nfvq5\") pod \"b941e0fe-6b17-4e30-917c-4d3f4382c545\" (UID: \"b941e0fe-6b17-4e30-917c-4d3f4382c545\") "
Nov 01 03:43:07.655440 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:07.655097    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b941e0fe-6b17-4e30-917c-4d3f4382c545-istio-token\" (UniqueName: \"kubernetes.io/projected/b941e0fe-6b17-4e30-917c-4d3f4382c545-istio-token\") pod \"b941e0fe-6b17-4e30-917c-4d3f4382c545\" (UID: \"b941e0fe-6b17-4e30-917c-4d3f4382c545\") "
Nov 01 03:43:07.655440 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:07.655265    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b941e0fe-6b17-4e30-917c-4d3f4382c545-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b941e0fe-6b17-4e30-917c-4d3f4382c545-istio-envoy\") pod \"b941e0fe-6b17-4e30-917c-4d3f4382c545\" (UID: \"b941e0fe-6b17-4e30-917c-4d3f4382c545\") "
Nov 01 03:43:07.655440 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:07.655282    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b941e0fe-6b17-4e30-917c-4d3f4382c545-istio-data" pod "b941e0fe-6b17-4e30-917c-4d3f4382c545" (UID: "b941e0fe-6b17-4e30-917c-4d3f4382c545"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:43:07.655440 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:07.655314    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b941e0fe-6b17-4e30-917c-4d3f4382c545-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b941e0fe-6b17-4e30-917c-4d3f4382c545-credential-socket\") pod \"b941e0fe-6b17-4e30-917c-4d3f4382c545\" (UID: \"b941e0fe-6b17-4e30-917c-4d3f4382c545\") "
Nov 01 03:43:07.655440 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:07.655347    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b941e0fe-6b17-4e30-917c-4d3f4382c545-workload-certs" pod "b941e0fe-6b17-4e30-917c-4d3f4382c545" (UID: "b941e0fe-6b17-4e30-917c-4d3f4382c545"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:43:07.655440 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:07.655396    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b941e0fe-6b17-4e30-917c-4d3f4382c545-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b941e0fe-6b17-4e30-917c-4d3f4382c545-workload-socket\") pod \"b941e0fe-6b17-4e30-917c-4d3f4382c545\" (UID: \"b941e0fe-6b17-4e30-917c-4d3f4382c545\") "
Nov 01 03:43:07.655920 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:07.655507    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b941e0fe-6b17-4e30-917c-4d3f4382c545-credential-socket" pod "b941e0fe-6b17-4e30-917c-4d3f4382c545" (UID: "b941e0fe-6b17-4e30-917c-4d3f4382c545"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:43:07.655920 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:07.655639    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b941e0fe-6b17-4e30-917c-4d3f4382c545-workload-socket" pod "b941e0fe-6b17-4e30-917c-4d3f4382c545" (UID: "b941e0fe-6b17-4e30-917c-4d3f4382c545"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:43:07.655920 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:07.655729    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b941e0fe-6b17-4e30-917c-4d3f4382c545-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b941e0fe-6b17-4e30-917c-4d3f4382c545-istiod-ca-cert\") pod \"b941e0fe-6b17-4e30-917c-4d3f4382c545\" (UID: \"b941e0fe-6b17-4e30-917c-4d3f4382c545\") "
Nov 01 03:43:07.655920 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:07.655776    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b941e0fe-6b17-4e30-917c-4d3f4382c545-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b941e0fe-6b17-4e30-917c-4d3f4382c545-istio-podinfo\") pod \"b941e0fe-6b17-4e30-917c-4d3f4382c545\" (UID: \"b941e0fe-6b17-4e30-917c-4d3f4382c545\") "
Nov 01 03:43:07.656121 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:07.656069    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b941e0fe-6b17-4e30-917c-4d3f4382c545-istiod-ca-cert" pod "b941e0fe-6b17-4e30-917c-4d3f4382c545" (UID: "b941e0fe-6b17-4e30-917c-4d3f4382c545"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:43:07.656415 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:07.656393    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b941e0fe-6b17-4e30-917c-4d3f4382c545-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:43:07.656506 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:07.656419    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b941e0fe-6b17-4e30-917c-4d3f4382c545-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:43:07.656506 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:07.656434    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b941e0fe-6b17-4e30-917c-4d3f4382c545-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:43:07.656506 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:07.656449    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b941e0fe-6b17-4e30-917c-4d3f4382c545-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:43:07.656506 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:07.656462    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b941e0fe-6b17-4e30-917c-4d3f4382c545-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:43:07.657526 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:07.657505    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b941e0fe-6b17-4e30-917c-4d3f4382c545-istio-envoy" pod "b941e0fe-6b17-4e30-917c-4d3f4382c545" (UID: "b941e0fe-6b17-4e30-917c-4d3f4382c545"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:43:07.657870 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:07.657842    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b941e0fe-6b17-4e30-917c-4d3f4382c545-istio-podinfo" pod "b941e0fe-6b17-4e30-917c-4d3f4382c545" (UID: "b941e0fe-6b17-4e30-917c-4d3f4382c545"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:43:07.658015 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:07.657997    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b941e0fe-6b17-4e30-917c-4d3f4382c545-kube-api-access-nfvq5" pod "b941e0fe-6b17-4e30-917c-4d3f4382c545" (UID: "b941e0fe-6b17-4e30-917c-4d3f4382c545"). InnerVolumeSpecName "kube-api-access-nfvq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:43:07.658091 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:07.658075    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b941e0fe-6b17-4e30-917c-4d3f4382c545-istio-token" pod "b941e0fe-6b17-4e30-917c-4d3f4382c545" (UID: "b941e0fe-6b17-4e30-917c-4d3f4382c545"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:43:07.757572 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:07.757522    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b941e0fe-6b17-4e30-917c-4d3f4382c545-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:43:07.757572 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:07.757570    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b941e0fe-6b17-4e30-917c-4d3f4382c545-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:43:07.757822 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:07.757585    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfvq5\" (UniqueName: \"kubernetes.io/projected/b941e0fe-6b17-4e30-917c-4d3f4382c545-kube-api-access-nfvq5\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:43:07.757822 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:07.757598    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b941e0fe-6b17-4e30-917c-4d3f4382c545-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:43:07.836655 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:07.836627    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:43:08.412538 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:08.412506    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"
Nov 01 03:43:08.412742 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:08.412512    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8" event={"ID":"b941e0fe-6b17-4e30-917c-4d3f4382c545","Type":"ContainerDied","Data":"a50bd657a5288886ee2fac8513badf68f831be4dd3e5737a760d637e9a29a96c"}
Nov 01 03:43:08.412742 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:08.412666    4930 scope.go:122] "RemoveContainer" containerID="531d96c3fec86de40cc71a6cb46e25e1287e0820759da0190f74ec879ee8f3af"
Nov 01 03:43:08.583308 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:08.583270    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"]
Nov 01 03:43:08.630345 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:08.630311    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-686153-openshift-default-7ccffcf78c-5gwh8"]
Nov 01 03:43:09.620926 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:09.620896    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b941e0fe-6b17-4e30-917c-4d3f4382c545" path="/var/lib/kubelet/pods/b941e0fe-6b17-4e30-917c-4d3f4382c545/volumes"
Nov 01 03:43:09.931360 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:09.931275    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:43:14.954507 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:14.954471    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e8c789-5b848f6fc5-9s298"]
Nov 01 03:43:15.044110 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:15.044066    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:43:15.322262 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:15.322171    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:43:15.522743 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:15.522659    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:43:19.124232 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:43:19.124146    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:43:19.831305 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:19.831268    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:43:19.955020 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:19.954976    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-e8c789-5b848f6fc5-9s298" podUID="1f72415b-cf6a-40c8-b9ab-33763ebbf66a" containerName="payload-processing" containerID="cri-o://386ac520879aefb3585d501e641f74490a85ed70c0e2cf5c08a68d442a1b8ef4" gracePeriod=25
Nov 01 03:43:21.348051 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:21.348014    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-e8c789-5b848f6fc5-9s298" podUID="1f72415b-cf6a-40c8-b9ab-33763ebbf66a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 03:43:24.755166 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:24.755116    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:43:29.696878 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:29.696838    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:43:31.348042 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:31.348005    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-e8c789-5b848f6fc5-9s298" podUID="1f72415b-cf6a-40c8-b9ab-33763ebbf66a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 03:43:31.655021 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:31.654985    4930 generic.go:356] "Generic (PLEG): container finished" podID="27e7c281-1391-4406-9dfa-d770c821678e" containerID="8ea181d9ef9afc7b4206e5a29ea369170cb7f6f25bf1a8fce106fa4f75966d9b" exitCode=0
Nov 01 03:43:31.655245 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:31.655062    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw" event={"ID":"27e7c281-1391-4406-9dfa-d770c821678e","Type":"ContainerDied","Data":"8ea181d9ef9afc7b4206e5a29ea369170cb7f6f25bf1a8fce106fa4f75966d9b"}
Nov 01 03:43:31.655490 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:31.655471    4930 scope.go:122] "RemoveContainer" containerID="8ea181d9ef9afc7b4206e5a29ea369170cb7f6f25bf1a8fce106fa4f75966d9b"
Nov 01 03:43:31.657436 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:31.657375    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:43:31.657610 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:31.657587    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:43:31.657716 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:31.657698    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:43:32.690175 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:32.690133    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw" event={"ID":"27e7c281-1391-4406-9dfa-d770c821678e","Type":"ContainerStarted","Data":"c6aea9e714e25329afaff81b3f7cf329000a41fb084c7167a37f59b1f12f6e8a"}
Nov 01 03:43:32.923216 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:32.923182    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"
Nov 01 03:43:32.923537 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:32.923393    4930 patch_prober.go:29] interesting pod/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Nov 01 03:43:32.923537 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:32.923452    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw" podUID="27e7c281-1391-4406-9dfa-d770c821678e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Nov 01 03:43:33.923208 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:33.923172    4930 patch_prober.go:29] interesting pod/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Nov 01 03:43:33.923629 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:33.923240    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw" podUID="27e7c281-1391-4406-9dfa-d770c821678e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Nov 01 03:43:34.655972 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:34.655938    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:43:34.923382 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:34.923301    4930 patch_prober.go:29] interesting pod/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Nov 01 03:43:34.923382 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:34.923360    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw" podUID="27e7c281-1391-4406-9dfa-d770c821678e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Nov 01 03:43:35.924129 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:35.924096    4930 patch_prober.go:29] interesting pod/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Nov 01 03:43:35.924518 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:35.924148    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw" podUID="27e7c281-1391-4406-9dfa-d770c821678e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Nov 01 03:43:36.924019 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:36.923987    4930 patch_prober.go:29] interesting pod/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Nov 01 03:43:36.924200 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:36.924039    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw" podUID="27e7c281-1391-4406-9dfa-d770c821678e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Nov 01 03:43:37.943593 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:37.943526    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"
Nov 01 03:43:37.944343 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:37.944307    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"
Nov 01 03:43:37.945241 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:37.945215    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"
Nov 01 03:43:39.021727 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.021693    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43005547-kh5lj"]
Nov 01 03:43:39.022913 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.022890    4930 cpu_manager.go:414] "containerMap: removing container" podUID="61310c85-114a-46f0-9a4d-daa629c4f4d4" containerName="revoke-keys"
Nov 01 03:43:39.022913 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.022912    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61310c85-114a-46f0-9a4d-daa629c4f4d4" containerName="revoke-keys"
Nov 01 03:43:39.023092 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.022937    4930 cpu_manager.go:414] "containerMap: removing container" podUID="b941e0fe-6b17-4e30-917c-4d3f4382c545" containerName="istio-proxy"
Nov 01 03:43:39.023092 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.022943    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b941e0fe-6b17-4e30-917c-4d3f4382c545" containerName="istio-proxy"
Nov 01 03:43:39.023092 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.022963    4930 cpu_manager.go:414] "containerMap: removing container" podUID="b941e0fe-6b17-4e30-917c-4d3f4382c545" containerName="istio-proxy"
Nov 01 03:43:39.023092 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.022968    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b941e0fe-6b17-4e30-917c-4d3f4382c545" containerName="istio-proxy"
Nov 01 03:43:39.023092 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.022976    4930 cpu_manager.go:414] "containerMap: removing container" podUID="a30e9e3f-be06-449e-84a9-e1e1b8be4d1d" containerName="istio-proxy"
Nov 01 03:43:39.023092 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.022980    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a30e9e3f-be06-449e-84a9-e1e1b8be4d1d" containerName="istio-proxy"
Nov 01 03:43:39.023092 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.023078    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="b941e0fe-6b17-4e30-917c-4d3f4382c545" containerName="istio-proxy"
Nov 01 03:43:39.023092 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.023091    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="b941e0fe-6b17-4e30-917c-4d3f4382c545" containerName="istio-proxy"
Nov 01 03:43:39.023495 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.023098    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="a30e9e3f-be06-449e-84a9-e1e1b8be4d1d" containerName="istio-proxy"
Nov 01 03:43:39.023495 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.023109    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="61310c85-114a-46f0-9a4d-daa629c4f4d4" containerName="revoke-keys"
Nov 01 03:43:39.040912 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.040883    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43005547-kh5lj"
Nov 01 03:43:39.063452 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.063425    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f855682-z69dn"]
Nov 01 03:43:39.067665 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.067645    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f855682-z69dn"
Nov 01 03:43:39.084423 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.084401    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43005547-kh5lj"]
Nov 01 03:43:39.094596 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.093541    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:43:39.094596 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.093544    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:43:39.094596 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.093597    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:43:39.094596 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.093635    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:43:39.128141 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.128113    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f855682-z69dn"]
Nov 01 03:43:39.209134 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.209103    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn99r\" (UniqueName: \"kubernetes.io/projected/28a14d04-9448-4306-9af5-3a3259ce2beb-kube-api-access-dn99r\") pod \"maas-api-revoke-sub-1f855682-z69dn\" (UID: \"28a14d04-9448-4306-9af5-3a3259ce2beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f855682-z69dn"
Nov 01 03:43:39.209282 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.209217    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28a14d04-9448-4306-9af5-3a3259ce2beb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f855682-z69dn\" (UID: \"28a14d04-9448-4306-9af5-3a3259ce2beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f855682-z69dn"
Nov 01 03:43:39.209282 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.209266    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/002037cc-a7d9-4eb3-af38-f3c45f47583b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43005547-kh5lj\" (UID: \"002037cc-a7d9-4eb3-af38-f3c45f47583b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43005547-kh5lj"
Nov 01 03:43:39.209421 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.209408    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28a14d04-9448-4306-9af5-3a3259ce2beb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f855682-z69dn\" (UID: \"28a14d04-9448-4306-9af5-3a3259ce2beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f855682-z69dn"
Nov 01 03:43:39.209458 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.209432    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkmsp\" (UniqueName: \"kubernetes.io/projected/002037cc-a7d9-4eb3-af38-f3c45f47583b-kube-api-access-xkmsp\") pod \"maas-api-revoke-sub-43005547-kh5lj\" (UID: \"002037cc-a7d9-4eb3-af38-f3c45f47583b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43005547-kh5lj"
Nov 01 03:43:39.209524 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.209507    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/002037cc-a7d9-4eb3-af38-f3c45f47583b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43005547-kh5lj\" (UID: \"002037cc-a7d9-4eb3-af38-f3c45f47583b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43005547-kh5lj"
Nov 01 03:43:39.311230 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.311155    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn99r\" (UniqueName: \"kubernetes.io/projected/28a14d04-9448-4306-9af5-3a3259ce2beb-kube-api-access-dn99r\") pod \"maas-api-revoke-sub-1f855682-z69dn\" (UID: \"28a14d04-9448-4306-9af5-3a3259ce2beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f855682-z69dn"
Nov 01 03:43:39.311370 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.311235    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28a14d04-9448-4306-9af5-3a3259ce2beb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f855682-z69dn\" (UID: \"28a14d04-9448-4306-9af5-3a3259ce2beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f855682-z69dn"
Nov 01 03:43:39.311370 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.311276    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/002037cc-a7d9-4eb3-af38-f3c45f47583b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43005547-kh5lj\" (UID: \"002037cc-a7d9-4eb3-af38-f3c45f47583b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43005547-kh5lj"
Nov 01 03:43:39.311494 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.311400    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28a14d04-9448-4306-9af5-3a3259ce2beb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f855682-z69dn\" (UID: \"28a14d04-9448-4306-9af5-3a3259ce2beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f855682-z69dn"
Nov 01 03:43:39.311494 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.311437    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkmsp\" (UniqueName: \"kubernetes.io/projected/002037cc-a7d9-4eb3-af38-f3c45f47583b-kube-api-access-xkmsp\") pod \"maas-api-revoke-sub-43005547-kh5lj\" (UID: \"002037cc-a7d9-4eb3-af38-f3c45f47583b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43005547-kh5lj"
Nov 01 03:43:39.311494 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.311480    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/002037cc-a7d9-4eb3-af38-f3c45f47583b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43005547-kh5lj\" (UID: \"002037cc-a7d9-4eb3-af38-f3c45f47583b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43005547-kh5lj"
Nov 01 03:43:39.312183 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.312156    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28a14d04-9448-4306-9af5-3a3259ce2beb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f855682-z69dn\" (UID: \"28a14d04-9448-4306-9af5-3a3259ce2beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f855682-z69dn"
Nov 01 03:43:39.312305 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.312234    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/002037cc-a7d9-4eb3-af38-f3c45f47583b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43005547-kh5lj\" (UID: \"002037cc-a7d9-4eb3-af38-f3c45f47583b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43005547-kh5lj"
Nov 01 03:43:39.313831 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.313804    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/002037cc-a7d9-4eb3-af38-f3c45f47583b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43005547-kh5lj\" (UID: \"002037cc-a7d9-4eb3-af38-f3c45f47583b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43005547-kh5lj"
Nov 01 03:43:39.313926 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.313839    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28a14d04-9448-4306-9af5-3a3259ce2beb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f855682-z69dn\" (UID: \"28a14d04-9448-4306-9af5-3a3259ce2beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f855682-z69dn"
Nov 01 03:43:39.343853 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.343827    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:43:39.387403 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.387381    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkmsp\" (UniqueName: \"kubernetes.io/projected/002037cc-a7d9-4eb3-af38-f3c45f47583b-kube-api-access-xkmsp\") pod \"maas-api-revoke-sub-43005547-kh5lj\" (UID: \"002037cc-a7d9-4eb3-af38-f3c45f47583b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43005547-kh5lj"
Nov 01 03:43:39.387518 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.387383    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn99r\" (UniqueName: \"kubernetes.io/projected/28a14d04-9448-4306-9af5-3a3259ce2beb-kube-api-access-dn99r\") pod \"maas-api-revoke-sub-1f855682-z69dn\" (UID: \"28a14d04-9448-4306-9af5-3a3259ce2beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f855682-z69dn"
Nov 01 03:43:39.655618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.655592    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43005547-kh5lj"
Nov 01 03:43:39.681055 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.681033    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f855682-z69dn"
Nov 01 03:43:39.856015 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:39.855982    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:43:40.039048 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:40.039009    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43005547-kh5lj"]
Nov 01 03:43:40.076661 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:43:40.076632    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod002037cc_a7d9_4eb3_af38_f3c45f47583b.slice/crio-fc4f30c0400a4152c83c58e20d371cce113ff27df3b8e3cd84b84dcd7922402e WatchSource:0}: Error finding container fc4f30c0400a4152c83c58e20d371cce113ff27df3b8e3cd84b84dcd7922402e: Status 404 returned error can't find the container with id fc4f30c0400a4152c83c58e20d371cce113ff27df3b8e3cd84b84dcd7922402e
Nov 01 03:43:40.089183 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:40.089148    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f855682-z69dn"]
Nov 01 03:43:40.091304 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:43:40.091279    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28a14d04_9448_4306_9af5_3a3259ce2beb.slice/crio-b230d7e46da75219e749d590784c94334ac2ce7246a169768dc9282420f44cfe WatchSource:0}: Error finding container b230d7e46da75219e749d590784c94334ac2ce7246a169768dc9282420f44cfe: Status 404 returned error can't find the container with id b230d7e46da75219e749d590784c94334ac2ce7246a169768dc9282420f44cfe
Nov 01 03:43:40.220127 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:40.220062    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-e8c789-5b848f6fc5-9s298"
Nov 01 03:43:40.333503 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:40.333477    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f72415b-cf6a-40c8-b9ab-33763ebbf66a-kube-api-access-fxmwn\" (UniqueName: \"kubernetes.io/projected/1f72415b-cf6a-40c8-b9ab-33763ebbf66a-kube-api-access-fxmwn\") pod \"1f72415b-cf6a-40c8-b9ab-33763ebbf66a\" (UID: \"1f72415b-cf6a-40c8-b9ab-33763ebbf66a\") "
Nov 01 03:43:40.333657 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:40.333516    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f72415b-cf6a-40c8-b9ab-33763ebbf66a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f72415b-cf6a-40c8-b9ab-33763ebbf66a-plugins-config-volume\") pod \"1f72415b-cf6a-40c8-b9ab-33763ebbf66a\" (UID: \"1f72415b-cf6a-40c8-b9ab-33763ebbf66a\") "
Nov 01 03:43:40.334026 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:40.334002    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f72415b-cf6a-40c8-b9ab-33763ebbf66a-plugins-config-volume" pod "1f72415b-cf6a-40c8-b9ab-33763ebbf66a" (UID: "1f72415b-cf6a-40c8-b9ab-33763ebbf66a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:43:40.334114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:40.334094    4930 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f72415b-cf6a-40c8-b9ab-33763ebbf66a-plugins-config-volume\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:43:40.335770 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:40.335713    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f72415b-cf6a-40c8-b9ab-33763ebbf66a-kube-api-access-fxmwn" pod "1f72415b-cf6a-40c8-b9ab-33763ebbf66a" (UID: "1f72415b-cf6a-40c8-b9ab-33763ebbf66a"). InnerVolumeSpecName "kube-api-access-fxmwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:43:40.434969 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:40.434937    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxmwn\" (UniqueName: \"kubernetes.io/projected/1f72415b-cf6a-40c8-b9ab-33763ebbf66a-kube-api-access-fxmwn\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:43:40.777216 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:40.777130    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f855682-z69dn" event={"ID":"28a14d04-9448-4306-9af5-3a3259ce2beb","Type":"ContainerStarted","Data":"e91ffd6b25481f961a4ba7d3c33a46664e02aa7f7d5feac51b586a758cd6d21d"}
Nov 01 03:43:40.777216 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:40.777196    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f855682-z69dn" event={"ID":"28a14d04-9448-4306-9af5-3a3259ce2beb","Type":"ContainerStarted","Data":"b230d7e46da75219e749d590784c94334ac2ce7246a169768dc9282420f44cfe"}
Nov 01 03:43:40.778902 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:40.778864    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43005547-kh5lj" event={"ID":"002037cc-a7d9-4eb3-af38-f3c45f47583b","Type":"ContainerStarted","Data":"bd1d5108a790a24b8b2cfda23be55d7fe39898f4f1f0522dfd08a9d9d03da88d"}
Nov 01 03:43:40.779067 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:40.778917    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43005547-kh5lj" event={"ID":"002037cc-a7d9-4eb3-af38-f3c45f47583b","Type":"ContainerStarted","Data":"fc4f30c0400a4152c83c58e20d371cce113ff27df3b8e3cd84b84dcd7922402e"}
Nov 01 03:43:40.780130 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:40.780107    4930 generic.go:356] "Generic (PLEG): container finished" podID="1f72415b-cf6a-40c8-b9ab-33763ebbf66a" containerID="386ac520879aefb3585d501e641f74490a85ed70c0e2cf5c08a68d442a1b8ef4" exitCode=0
Nov 01 03:43:40.780250 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:40.780137    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-e8c789-5b848f6fc5-9s298"
Nov 01 03:43:40.780250 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:40.780144    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-e8c789-5b848f6fc5-9s298" event={"ID":"1f72415b-cf6a-40c8-b9ab-33763ebbf66a","Type":"ContainerDied","Data":"386ac520879aefb3585d501e641f74490a85ed70c0e2cf5c08a68d442a1b8ef4"}
Nov 01 03:43:40.780250 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:40.780232    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-e8c789-5b848f6fc5-9s298" event={"ID":"1f72415b-cf6a-40c8-b9ab-33763ebbf66a","Type":"ContainerDied","Data":"6d33f77984dbf9af35ae99d225dddabe4f52581a0e2fc3771d319229e8f628c9"}
Nov 01 03:43:40.780356 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:40.780258    4930 scope.go:122] "RemoveContainer" containerID="386ac520879aefb3585d501e641f74490a85ed70c0e2cf5c08a68d442a1b8ef4"
Nov 01 03:43:40.791689 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:40.791668    4930 scope.go:122] "RemoveContainer" containerID="386ac520879aefb3585d501e641f74490a85ed70c0e2cf5c08a68d442a1b8ef4"
Nov 01 03:43:40.791943 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:43:40.791924    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"386ac520879aefb3585d501e641f74490a85ed70c0e2cf5c08a68d442a1b8ef4\": container with ID starting with 386ac520879aefb3585d501e641f74490a85ed70c0e2cf5c08a68d442a1b8ef4 not found: ID does not exist" containerID="386ac520879aefb3585d501e641f74490a85ed70c0e2cf5c08a68d442a1b8ef4"
Nov 01 03:43:40.792009 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:40.791950    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"386ac520879aefb3585d501e641f74490a85ed70c0e2cf5c08a68d442a1b8ef4"} err="failed to get container status \"386ac520879aefb3585d501e641f74490a85ed70c0e2cf5c08a68d442a1b8ef4\": rpc error: code = NotFound desc = could not find container \"386ac520879aefb3585d501e641f74490a85ed70c0e2cf5c08a68d442a1b8ef4\": container with ID starting with 386ac520879aefb3585d501e641f74490a85ed70c0e2cf5c08a68d442a1b8ef4 not found: ID does not exist"
Nov 01 03:43:40.893940 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:40.893888    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f855682-z69dn" podStartSLOduration=1.893874638 podStartE2EDuration="1.893874638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:43:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:43:40.892739066 +0000 UTC m=+2225.415745655" watchObservedRunningTime="2026-11-01 03:43:40.893874638 +0000 UTC m=+2225.416881225"
Nov 01 03:43:41.013199 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:41.013148    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43005547-kh5lj" podStartSLOduration=3.013134463 podStartE2EDuration="3.013134463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:43:41.01276797 +0000 UTC m=+2225.535774560" watchObservedRunningTime="2026-11-01 03:43:41.013134463 +0000 UTC m=+2225.536141052"
Nov 01 03:43:41.126285 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:41.126249    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e8c789-5b848f6fc5-9s298"]
Nov 01 03:43:41.130346 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:41.130317    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e8c789-5b848f6fc5-9s298"]
Nov 01 03:43:41.626551 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:41.626520    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f72415b-cf6a-40c8-b9ab-33763ebbf66a" path="/var/lib/kubelet/pods/1f72415b-cf6a-40c8-b9ab-33763ebbf66a/volumes"
Nov 01 03:43:41.789188 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:41.789151    4930 generic.go:356] "Generic (PLEG): container finished" podID="28a14d04-9448-4306-9af5-3a3259ce2beb" containerID="e91ffd6b25481f961a4ba7d3c33a46664e02aa7f7d5feac51b586a758cd6d21d" exitCode=0
Nov 01 03:43:41.789372 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:41.789232    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f855682-z69dn" event={"ID":"28a14d04-9448-4306-9af5-3a3259ce2beb","Type":"ContainerDied","Data":"e91ffd6b25481f961a4ba7d3c33a46664e02aa7f7d5feac51b586a758cd6d21d"}
Nov 01 03:43:41.791006 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:41.790985    4930 generic.go:356] "Generic (PLEG): container finished" podID="002037cc-a7d9-4eb3-af38-f3c45f47583b" containerID="bd1d5108a790a24b8b2cfda23be55d7fe39898f4f1f0522dfd08a9d9d03da88d" exitCode=0
Nov 01 03:43:41.791113 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:41.791018    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43005547-kh5lj" event={"ID":"002037cc-a7d9-4eb3-af38-f3c45f47583b","Type":"ContainerDied","Data":"bd1d5108a790a24b8b2cfda23be55d7fe39898f4f1f0522dfd08a9d9d03da88d"}
Nov 01 03:43:42.999357 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:42.999332    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43005547-kh5lj"
Nov 01 03:43:43.002005 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:43.001986    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f855682-z69dn"
Nov 01 03:43:43.079389 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:43.079322    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28a14d04-9448-4306-9af5-3a3259ce2beb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28a14d04-9448-4306-9af5-3a3259ce2beb-maas-api-openshift-ca\") pod \"28a14d04-9448-4306-9af5-3a3259ce2beb\" (UID: \"28a14d04-9448-4306-9af5-3a3259ce2beb\") "
Nov 01 03:43:43.079517 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:43.079434    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/002037cc-a7d9-4eb3-af38-f3c45f47583b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/002037cc-a7d9-4eb3-af38-f3c45f47583b-maas-api-xks-ca\") pod \"002037cc-a7d9-4eb3-af38-f3c45f47583b\" (UID: \"002037cc-a7d9-4eb3-af38-f3c45f47583b\") "
Nov 01 03:43:43.079517 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:43.079483    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28a14d04-9448-4306-9af5-3a3259ce2beb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28a14d04-9448-4306-9af5-3a3259ce2beb-maas-api-xks-ca\") pod \"28a14d04-9448-4306-9af5-3a3259ce2beb\" (UID: \"28a14d04-9448-4306-9af5-3a3259ce2beb\") "
Nov 01 03:43:43.079684 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:43.079536    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/002037cc-a7d9-4eb3-af38-f3c45f47583b-kube-api-access-xkmsp\" (UniqueName: \"kubernetes.io/projected/002037cc-a7d9-4eb3-af38-f3c45f47583b-kube-api-access-xkmsp\") pod \"002037cc-a7d9-4eb3-af38-f3c45f47583b\" (UID: \"002037cc-a7d9-4eb3-af38-f3c45f47583b\") "
Nov 01 03:43:43.079684 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:43.079624    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28a14d04-9448-4306-9af5-3a3259ce2beb-kube-api-access-dn99r\" (UniqueName: \"kubernetes.io/projected/28a14d04-9448-4306-9af5-3a3259ce2beb-kube-api-access-dn99r\") pod \"28a14d04-9448-4306-9af5-3a3259ce2beb\" (UID: \"28a14d04-9448-4306-9af5-3a3259ce2beb\") "
Nov 01 03:43:43.079684 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:43.079669    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/002037cc-a7d9-4eb3-af38-f3c45f47583b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/002037cc-a7d9-4eb3-af38-f3c45f47583b-maas-api-openshift-ca\") pod \"002037cc-a7d9-4eb3-af38-f3c45f47583b\" (UID: \"002037cc-a7d9-4eb3-af38-f3c45f47583b\") "
Nov 01 03:43:43.079852 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:43.079690    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28a14d04-9448-4306-9af5-3a3259ce2beb-maas-api-openshift-ca" pod "28a14d04-9448-4306-9af5-3a3259ce2beb" (UID: "28a14d04-9448-4306-9af5-3a3259ce2beb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:43:43.080131 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:43.080102    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/002037cc-a7d9-4eb3-af38-f3c45f47583b-maas-api-openshift-ca" pod "002037cc-a7d9-4eb3-af38-f3c45f47583b" (UID: "002037cc-a7d9-4eb3-af38-f3c45f47583b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:43:43.080345 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:43.080322    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/002037cc-a7d9-4eb3-af38-f3c45f47583b-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:43:43.080424 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:43.080350    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28a14d04-9448-4306-9af5-3a3259ce2beb-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:43:43.081819 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:43.081792    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28a14d04-9448-4306-9af5-3a3259ce2beb-maas-api-xks-ca" pod "28a14d04-9448-4306-9af5-3a3259ce2beb" (UID: "28a14d04-9448-4306-9af5-3a3259ce2beb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:43:43.081917 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:43.081894    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/002037cc-a7d9-4eb3-af38-f3c45f47583b-kube-api-access-xkmsp" pod "002037cc-a7d9-4eb3-af38-f3c45f47583b" (UID: "002037cc-a7d9-4eb3-af38-f3c45f47583b"). InnerVolumeSpecName "kube-api-access-xkmsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:43:43.082130 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:43.082108    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28a14d04-9448-4306-9af5-3a3259ce2beb-kube-api-access-dn99r" pod "28a14d04-9448-4306-9af5-3a3259ce2beb" (UID: "28a14d04-9448-4306-9af5-3a3259ce2beb"). InnerVolumeSpecName "kube-api-access-dn99r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:43:43.082176 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:43.082119    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/002037cc-a7d9-4eb3-af38-f3c45f47583b-maas-api-xks-ca" pod "002037cc-a7d9-4eb3-af38-f3c45f47583b" (UID: "002037cc-a7d9-4eb3-af38-f3c45f47583b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:43:43.181856 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:43.181823    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/002037cc-a7d9-4eb3-af38-f3c45f47583b-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:43:43.181856 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:43.181851    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28a14d04-9448-4306-9af5-3a3259ce2beb-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:43:43.181856 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:43.181862    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkmsp\" (UniqueName: \"kubernetes.io/projected/002037cc-a7d9-4eb3-af38-f3c45f47583b-kube-api-access-xkmsp\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:43:43.182071 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:43.181871    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn99r\" (UniqueName: \"kubernetes.io/projected/28a14d04-9448-4306-9af5-3a3259ce2beb-kube-api-access-dn99r\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:43:43.807312 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:43.807277    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43005547-kh5lj"
Nov 01 03:43:43.807496 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:43.807311    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43005547-kh5lj" event={"ID":"002037cc-a7d9-4eb3-af38-f3c45f47583b","Type":"ContainerDied","Data":"fc4f30c0400a4152c83c58e20d371cce113ff27df3b8e3cd84b84dcd7922402e"}
Nov 01 03:43:43.807496 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:43.807355    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc4f30c0400a4152c83c58e20d371cce113ff27df3b8e3cd84b84dcd7922402e"
Nov 01 03:43:43.809074 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:43.809053    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f855682-z69dn"
Nov 01 03:43:43.809074 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:43.809064    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f855682-z69dn" event={"ID":"28a14d04-9448-4306-9af5-3a3259ce2beb","Type":"ContainerDied","Data":"b230d7e46da75219e749d590784c94334ac2ce7246a169768dc9282420f44cfe"}
Nov 01 03:43:43.809236 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:43.809103    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b230d7e46da75219e749d590784c94334ac2ce7246a169768dc9282420f44cfe"
Nov 01 03:43:44.723708 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:44.723669    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:43:44.921364 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:44.921326    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:43:45.704331 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:45.704296    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:43:45.907964 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:45.907932    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:43:46.728707 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:46.728673    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:43:48.784500 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:48.784472    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:43:49.122912 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:43:49.122862    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:43:49.830164 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:49.830121    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:43:53.410819 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:53.410755    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:43:53.593609 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:53.593546    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:43:54.275985 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:54.275950    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:43:54.477975 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:54.477937    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:43:54.958443 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:54.958402    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:43:59.701162 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:59.701125    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:43:59.853699 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:43:59.853666    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:44:00.025751 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:00.025669    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:44:01.407625 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:01.404334    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:44:02.240577 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:02.238279    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:44:02.889577 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:02.887177    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:44:03.303371 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:03.303282    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:44:04.685240 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:04.685209    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:44:06.471865 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:06.471825    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:44:06.693191 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:06.693156    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:44:06.894184 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:06.894151    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:44:07.076518 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:07.076484    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f2383c-jlfkt"]
Nov 01 03:44:07.122958 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:07.122922    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f2383c-jlfkt"]
Nov 01 03:44:07.417048 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:07.417013    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:44:07.588721 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:07.588690    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:44:07.622391 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:07.622359    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="714d732d-0c4f-410f-986b-5ea4b6807902" path="/var/lib/kubelet/pods/714d732d-0c4f-410f-986b-5ea4b6807902/volumes"
Nov 01 03:44:07.735908 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:07.735826    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:44:12.172339 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:12.172302    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:44:12.465880 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:12.465793    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:44:12.669533 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:12.669495    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:44:12.971054 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:12.971016    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:44:15.316840 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:15.316809    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:44:15.499290 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:15.499251    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:44:15.675285 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:15.675246    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:44:16.514405 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:16.514368    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4f30-lnvvf"]
Nov 01 03:44:16.515212 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:16.515189    4930 cpu_manager.go:414] "containerMap: removing container" podUID="28a14d04-9448-4306-9af5-3a3259ce2beb" containerName="revoke-keys"
Nov 01 03:44:16.515324 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:16.515213    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28a14d04-9448-4306-9af5-3a3259ce2beb" containerName="revoke-keys"
Nov 01 03:44:16.515324 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:16.515230    4930 cpu_manager.go:414] "containerMap: removing container" podUID="1f72415b-cf6a-40c8-b9ab-33763ebbf66a" containerName="payload-processing"
Nov 01 03:44:16.515324 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:16.515239    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f72415b-cf6a-40c8-b9ab-33763ebbf66a" containerName="payload-processing"
Nov 01 03:44:16.515324 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:16.515276    4930 cpu_manager.go:414] "containerMap: removing container" podUID="002037cc-a7d9-4eb3-af38-f3c45f47583b" containerName="revoke-keys"
Nov 01 03:44:16.515324 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:16.515283    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="002037cc-a7d9-4eb3-af38-f3c45f47583b" containerName="revoke-keys"
Nov 01 03:44:16.515643 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:16.515483    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="28a14d04-9448-4306-9af5-3a3259ce2beb" containerName="revoke-keys"
Nov 01 03:44:16.515643 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:16.515497    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f72415b-cf6a-40c8-b9ab-33763ebbf66a" containerName="payload-processing"
Nov 01 03:44:16.515643 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:16.515516    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="002037cc-a7d9-4eb3-af38-f3c45f47583b" containerName="revoke-keys"
Nov 01 03:44:16.520309 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:16.520288    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4f30-lnvvf"
Nov 01 03:44:16.585598 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:16.585536    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4f30-lnvvf"]
Nov 01 03:44:16.624129 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:16.624097    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkq8d\" (UniqueName: \"kubernetes.io/projected/695decaa-433d-4b80-92f0-761191c7c66b-kube-api-access-fkq8d\") pod \"maas-api-revoke-keys-e2e-api-b-da4f30-lnvvf\" (UID: \"695decaa-433d-4b80-92f0-761191c7c66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4f30-lnvvf"
Nov 01 03:44:16.624284 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:16.624168    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/695decaa-433d-4b80-92f0-761191c7c66b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-da4f30-lnvvf\" (UID: \"695decaa-433d-4b80-92f0-761191c7c66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4f30-lnvvf"
Nov 01 03:44:16.624333 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:16.624309    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/695decaa-433d-4b80-92f0-761191c7c66b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-da4f30-lnvvf\" (UID: \"695decaa-433d-4b80-92f0-761191c7c66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4f30-lnvvf"
Nov 01 03:44:16.629590 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:16.629533    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:44:16.629711 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:16.629624    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:44:16.632654 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:16.632631    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:44:16.632922 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:16.632909    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:44:16.726069 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:16.726038    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/695decaa-433d-4b80-92f0-761191c7c66b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-da4f30-lnvvf\" (UID: \"695decaa-433d-4b80-92f0-761191c7c66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4f30-lnvvf"
Nov 01 03:44:16.726265 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:16.726170    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/695decaa-433d-4b80-92f0-761191c7c66b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-da4f30-lnvvf\" (UID: \"695decaa-433d-4b80-92f0-761191c7c66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4f30-lnvvf"
Nov 01 03:44:16.726384 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:16.726368    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkq8d\" (UniqueName: \"kubernetes.io/projected/695decaa-433d-4b80-92f0-761191c7c66b-kube-api-access-fkq8d\") pod \"maas-api-revoke-keys-e2e-api-b-da4f30-lnvvf\" (UID: \"695decaa-433d-4b80-92f0-761191c7c66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4f30-lnvvf"
Nov 01 03:44:16.726916 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:16.726893    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/695decaa-433d-4b80-92f0-761191c7c66b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-da4f30-lnvvf\" (UID: \"695decaa-433d-4b80-92f0-761191c7c66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4f30-lnvvf"
Nov 01 03:44:16.728460 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:16.728434    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/695decaa-433d-4b80-92f0-761191c7c66b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-da4f30-lnvvf\" (UID: \"695decaa-433d-4b80-92f0-761191c7c66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4f30-lnvvf"
Nov 01 03:44:16.787587 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:16.787491    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkq8d\" (UniqueName: \"kubernetes.io/projected/695decaa-433d-4b80-92f0-761191c7c66b-kube-api-access-fkq8d\") pod \"maas-api-revoke-keys-e2e-api-b-da4f30-lnvvf\" (UID: \"695decaa-433d-4b80-92f0-761191c7c66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4f30-lnvvf"
Nov 01 03:44:16.835170 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:16.835136    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4f30-lnvvf"
Nov 01 03:44:17.229229 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:17.229197    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4f30-lnvvf"]
Nov 01 03:44:17.229952 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:44:17.229926    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod695decaa_433d_4b80_92f0_761191c7c66b.slice/crio-70179280acf3f37fee4e0cccd40adb0bf6faac4cb66c9d482280857e6b28db90 WatchSource:0}: Error finding container 70179280acf3f37fee4e0cccd40adb0bf6faac4cb66c9d482280857e6b28db90: Status 404 returned error can't find the container with id 70179280acf3f37fee4e0cccd40adb0bf6faac4cb66c9d482280857e6b28db90
Nov 01 03:44:17.913131 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:17.913099    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:44:18.214478 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:18.214438    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4f30-lnvvf" event={"ID":"695decaa-433d-4b80-92f0-761191c7c66b","Type":"ContainerStarted","Data":"4ac4caa3b804083627610a790dcedfca430ff1b898c4dac090cd93df22347b15"}
Nov 01 03:44:18.214686 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:18.214495    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4f30-lnvvf" event={"ID":"695decaa-433d-4b80-92f0-761191c7c66b","Type":"ContainerStarted","Data":"70179280acf3f37fee4e0cccd40adb0bf6faac4cb66c9d482280857e6b28db90"}
Nov 01 03:44:18.338367 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:18.338314    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4f30-lnvvf" podStartSLOduration=2.338301045 podStartE2EDuration="2.338301045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:44:18.334801659 +0000 UTC m=+2262.857808260" watchObservedRunningTime="2026-11-01 03:44:18.338301045 +0000 UTC m=+2262.861307634"
Nov 01 03:44:19.122141 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:44:19.122099    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:44:19.222983 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:19.222952    4930 generic.go:356] "Generic (PLEG): container finished" podID="695decaa-433d-4b80-92f0-761191c7c66b" containerID="4ac4caa3b804083627610a790dcedfca430ff1b898c4dac090cd93df22347b15" exitCode=0
Nov 01 03:44:19.223183 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:19.223037    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4f30-lnvvf" event={"ID":"695decaa-433d-4b80-92f0-761191c7c66b","Type":"ContainerDied","Data":"4ac4caa3b804083627610a790dcedfca430ff1b898c4dac090cd93df22347b15"}
Nov 01 03:44:20.419743 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:20.419718    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4f30-lnvvf"
Nov 01 03:44:20.585835 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:20.585810    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/695decaa-433d-4b80-92f0-761191c7c66b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/695decaa-433d-4b80-92f0-761191c7c66b-maas-api-openshift-ca\") pod \"695decaa-433d-4b80-92f0-761191c7c66b\" (UID: \"695decaa-433d-4b80-92f0-761191c7c66b\") "
Nov 01 03:44:20.586002 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:20.585901    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/695decaa-433d-4b80-92f0-761191c7c66b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/695decaa-433d-4b80-92f0-761191c7c66b-maas-api-xks-ca\") pod \"695decaa-433d-4b80-92f0-761191c7c66b\" (UID: \"695decaa-433d-4b80-92f0-761191c7c66b\") "
Nov 01 03:44:20.586148 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:20.586130    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/695decaa-433d-4b80-92f0-761191c7c66b-kube-api-access-fkq8d\" (UniqueName: \"kubernetes.io/projected/695decaa-433d-4b80-92f0-761191c7c66b-kube-api-access-fkq8d\") pod \"695decaa-433d-4b80-92f0-761191c7c66b\" (UID: \"695decaa-433d-4b80-92f0-761191c7c66b\") "
Nov 01 03:44:20.586208 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:20.586156    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/695decaa-433d-4b80-92f0-761191c7c66b-maas-api-openshift-ca" pod "695decaa-433d-4b80-92f0-761191c7c66b" (UID: "695decaa-433d-4b80-92f0-761191c7c66b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:44:20.586609 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:20.586582    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/695decaa-433d-4b80-92f0-761191c7c66b-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:44:20.588132 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:20.588108    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/695decaa-433d-4b80-92f0-761191c7c66b-maas-api-xks-ca" pod "695decaa-433d-4b80-92f0-761191c7c66b" (UID: "695decaa-433d-4b80-92f0-761191c7c66b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:44:20.588274 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:20.588253    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/695decaa-433d-4b80-92f0-761191c7c66b-kube-api-access-fkq8d" pod "695decaa-433d-4b80-92f0-761191c7c66b" (UID: "695decaa-433d-4b80-92f0-761191c7c66b"). InnerVolumeSpecName "kube-api-access-fkq8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:44:20.688526 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:20.688484    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/695decaa-433d-4b80-92f0-761191c7c66b-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:44:20.688526 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:20.688513    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkq8d\" (UniqueName: \"kubernetes.io/projected/695decaa-433d-4b80-92f0-761191c7c66b-kube-api-access-fkq8d\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:44:21.240970 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:21.240901    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4f30-lnvvf" event={"ID":"695decaa-433d-4b80-92f0-761191c7c66b","Type":"ContainerDied","Data":"70179280acf3f37fee4e0cccd40adb0bf6faac4cb66c9d482280857e6b28db90"}
Nov 01 03:44:21.241437 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:21.241365    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70179280acf3f37fee4e0cccd40adb0bf6faac4cb66c9d482280857e6b28db90"
Nov 01 03:44:21.241551 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:21.241479    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4f30-lnvvf"
Nov 01 03:44:23.199841 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:23.199795    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:44:23.405941 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:23.405909    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fdbdfefd-9qs47"]
Nov 01 03:44:23.406529 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:23.406515    4930 cpu_manager.go:414] "containerMap: removing container" podUID="695decaa-433d-4b80-92f0-761191c7c66b" containerName="revoke-keys"
Nov 01 03:44:23.406626 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:23.406531    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="695decaa-433d-4b80-92f0-761191c7c66b" containerName="revoke-keys"
Nov 01 03:44:23.406678 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:23.406668    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="695decaa-433d-4b80-92f0-761191c7c66b" containerName="revoke-keys"
Nov 01 03:44:23.410271 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:23.410248    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fdbdfefd-9qs47"
Nov 01 03:44:23.465815 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:23.465679    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:44:23.515029 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:23.515001    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:44:23.515246 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:23.515224    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:44:23.515387 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:23.515369    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:44:23.519696 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:23.519677    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fdbdfefd-9qs47"]
Nov 01 03:44:23.526259 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:23.526237    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztjhf\" (UniqueName: \"kubernetes.io/projected/1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad-kube-api-access-ztjhf\") pod \"maas-api-revoke-keys-e2e-mt-fdbdfefd-9qs47\" (UID: \"1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fdbdfefd-9qs47"
Nov 01 03:44:23.526390 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:23.526371    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-fdbdfefd-9qs47\" (UID: \"1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fdbdfefd-9qs47"
Nov 01 03:44:23.526509 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:23.526488    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-fdbdfefd-9qs47\" (UID: \"1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fdbdfefd-9qs47"
Nov 01 03:44:23.627929 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:23.627899    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-fdbdfefd-9qs47\" (UID: \"1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fdbdfefd-9qs47"
Nov 01 03:44:23.628050 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:23.627972    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-fdbdfefd-9qs47\" (UID: \"1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fdbdfefd-9qs47"
Nov 01 03:44:23.628100 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:23.628083    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztjhf\" (UniqueName: \"kubernetes.io/projected/1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad-kube-api-access-ztjhf\") pod \"maas-api-revoke-keys-e2e-mt-fdbdfefd-9qs47\" (UID: \"1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fdbdfefd-9qs47"
Nov 01 03:44:23.628636 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:23.628616    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-fdbdfefd-9qs47\" (UID: \"1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fdbdfefd-9qs47"
Nov 01 03:44:23.630293 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:23.630266    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-fdbdfefd-9qs47\" (UID: \"1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fdbdfefd-9qs47"
Nov 01 03:44:23.689394 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:23.689366    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztjhf\" (UniqueName: \"kubernetes.io/projected/1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad-kube-api-access-ztjhf\") pod \"maas-api-revoke-keys-e2e-mt-fdbdfefd-9qs47\" (UID: \"1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fdbdfefd-9qs47"
Nov 01 03:44:23.724081 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:23.724014    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fdbdfefd-9qs47"
Nov 01 03:44:24.134594 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:44:24.134542    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b7551de_ee98_4dd5_ba0e_0886cfa7a2ad.slice/crio-2febc952a3d78a1f9ff3fb3763bfb211e1276ce9d611288fd3bf3edcfc579a45 WatchSource:0}: Error finding container 2febc952a3d78a1f9ff3fb3763bfb211e1276ce9d611288fd3bf3edcfc579a45: Status 404 returned error can't find the container with id 2febc952a3d78a1f9ff3fb3763bfb211e1276ce9d611288fd3bf3edcfc579a45
Nov 01 03:44:24.134820 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:24.134795    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fdbdfefd-9qs47"]
Nov 01 03:44:24.271703 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:24.271667    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fdbdfefd-9qs47" event={"ID":"1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad","Type":"ContainerStarted","Data":"2febc952a3d78a1f9ff3fb3763bfb211e1276ce9d611288fd3bf3edcfc579a45"}
Nov 01 03:44:25.282310 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:25.282273    4930 generic.go:356] "Generic (PLEG): container finished" podID="1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad" containerID="eead74419348327a647a938771156a9cea748891cd0efe1e8c209977f555aa20" exitCode=0
Nov 01 03:44:25.282729 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:25.282342    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fdbdfefd-9qs47" event={"ID":"1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad","Type":"ContainerDied","Data":"eead74419348327a647a938771156a9cea748891cd0efe1e8c209977f555aa20"}
Nov 01 03:44:26.443992 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:26.443969    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fdbdfefd-9qs47"
Nov 01 03:44:26.566358 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:26.566282    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad-maas-api-xks-ca\") pod \"1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad\" (UID: \"1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad\") "
Nov 01 03:44:26.566513 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:26.566381    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad-kube-api-access-ztjhf\" (UniqueName: \"kubernetes.io/projected/1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad-kube-api-access-ztjhf\") pod \"1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad\" (UID: \"1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad\") "
Nov 01 03:44:26.566513 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:26.566432    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad-maas-api-openshift-ca\") pod \"1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad\" (UID: \"1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad\") "
Nov 01 03:44:26.566848 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:26.566814    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad-maas-api-openshift-ca" pod "1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad" (UID: "1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:44:26.567113 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:26.567087    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:44:26.568424 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:26.568399    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad-maas-api-xks-ca" pod "1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad" (UID: "1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:44:26.574742 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:26.574714    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad-kube-api-access-ztjhf" pod "1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad" (UID: "1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad"). InnerVolumeSpecName "kube-api-access-ztjhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:44:26.668430 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:26.668402    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:44:26.668430 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:26.668432    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztjhf\" (UniqueName: \"kubernetes.io/projected/1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad-kube-api-access-ztjhf\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:44:27.299201 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:27.299173    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fdbdfefd-9qs47"
Nov 01 03:44:27.299398 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:27.299212    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fdbdfefd-9qs47" event={"ID":"1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad","Type":"ContainerDied","Data":"2febc952a3d78a1f9ff3fb3763bfb211e1276ce9d611288fd3bf3edcfc579a45"}
Nov 01 03:44:27.299398 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:27.299269    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2febc952a3d78a1f9ff3fb3763bfb211e1276ce9d611288fd3bf3edcfc579a45"
Nov 01 03:44:28.074993 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:28.074955    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-da4f30-646b9f5d65-qhxwx"]
Nov 01 03:44:28.197179 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:28.197144    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-da4f30-647ffb7887-skpkm"]
Nov 01 03:44:28.483890 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:28.483835    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:44:28.627701 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:28.627608    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:44:33.076209 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:33.076167    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-da4f30-646b9f5d65-qhxwx" podUID="08c53d46-b65c-4799-98f7-9bd64e561ebc" containerName="payload-processing" containerID="cri-o://ac37a90661a61da657110f8ee03e900f4cb61a39587b955536ef5266da847f66" gracePeriod=25
Nov 01 03:44:33.198352 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:33.198319    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da4f30-647ffb7887-skpkm" podUID="3077cb58-8c2b-48fd-a777-dafaecbce99e" containerName="payload-pre-processing" containerID="cri-o://bb02cd4d37f900d46d6afe8a6bc10998fd656ca63a82422db6841282a80ce2d9" gracePeriod=25
Nov 01 03:44:33.363433 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:33.363322    4930 generic.go:356] "Generic (PLEG): container finished" podID="08c53d46-b65c-4799-98f7-9bd64e561ebc" containerID="ac37a90661a61da657110f8ee03e900f4cb61a39587b955536ef5266da847f66" exitCode=0
Nov 01 03:44:33.363433 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:33.363392    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-da4f30-646b9f5d65-qhxwx" event={"ID":"08c53d46-b65c-4799-98f7-9bd64e561ebc","Type":"ContainerDied","Data":"ac37a90661a61da657110f8ee03e900f4cb61a39587b955536ef5266da847f66"}
Nov 01 03:44:33.363638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:33.363448    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-da4f30-646b9f5d65-qhxwx" event={"ID":"08c53d46-b65c-4799-98f7-9bd64e561ebc","Type":"ContainerDied","Data":"0f09ae6e899ecb55b9db6751dd6640924a1dc59915f3e1f92bb544ae688537a8"}
Nov 01 03:44:33.363638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:33.363471    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f09ae6e899ecb55b9db6751dd6640924a1dc59915f3e1f92bb544ae688537a8"
Nov 01 03:44:33.365335 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:33.365311    4930 generic.go:356] "Generic (PLEG): container finished" podID="3077cb58-8c2b-48fd-a777-dafaecbce99e" containerID="bb02cd4d37f900d46d6afe8a6bc10998fd656ca63a82422db6841282a80ce2d9" exitCode=0
Nov 01 03:44:33.365446 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:33.365433    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da4f30-647ffb7887-skpkm" event={"ID":"3077cb58-8c2b-48fd-a777-dafaecbce99e","Type":"ContainerDied","Data":"bb02cd4d37f900d46d6afe8a6bc10998fd656ca63a82422db6841282a80ce2d9"}
Nov 01 03:44:33.379581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:33.379541    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-da4f30-646b9f5d65-qhxwx"
Nov 01 03:44:33.464055 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:33.463916    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08c53d46-b65c-4799-98f7-9bd64e561ebc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08c53d46-b65c-4799-98f7-9bd64e561ebc-plugins-config-volume\") pod \"08c53d46-b65c-4799-98f7-9bd64e561ebc\" (UID: \"08c53d46-b65c-4799-98f7-9bd64e561ebc\") "
Nov 01 03:44:33.464055 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:33.463977    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08c53d46-b65c-4799-98f7-9bd64e561ebc-kube-api-access-zwnxn\" (UniqueName: \"kubernetes.io/projected/08c53d46-b65c-4799-98f7-9bd64e561ebc-kube-api-access-zwnxn\") pod \"08c53d46-b65c-4799-98f7-9bd64e561ebc\" (UID: \"08c53d46-b65c-4799-98f7-9bd64e561ebc\") "
Nov 01 03:44:33.465006 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:33.464975    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08c53d46-b65c-4799-98f7-9bd64e561ebc-plugins-config-volume" pod "08c53d46-b65c-4799-98f7-9bd64e561ebc" (UID: "08c53d46-b65c-4799-98f7-9bd64e561ebc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:44:33.467697 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:33.467653    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c53d46-b65c-4799-98f7-9bd64e561ebc-kube-api-access-zwnxn" pod "08c53d46-b65c-4799-98f7-9bd64e561ebc" (UID: "08c53d46-b65c-4799-98f7-9bd64e561ebc"). InnerVolumeSpecName "kube-api-access-zwnxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:44:33.469302 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:33.469273    4930 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08c53d46-b65c-4799-98f7-9bd64e561ebc-plugins-config-volume\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:44:33.469426 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:33.469308    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwnxn\" (UniqueName: \"kubernetes.io/projected/08c53d46-b65c-4799-98f7-9bd64e561ebc-kube-api-access-zwnxn\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:44:33.497725 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:33.497704    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da4f30-647ffb7887-skpkm"
Nov 01 03:44:33.571370 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:33.571340    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3077cb58-8c2b-48fd-a777-dafaecbce99e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3077cb58-8c2b-48fd-a777-dafaecbce99e-plugins-config-volume\") pod \"3077cb58-8c2b-48fd-a777-dafaecbce99e\" (UID: \"3077cb58-8c2b-48fd-a777-dafaecbce99e\") "
Nov 01 03:44:33.571533 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:33.571383    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3077cb58-8c2b-48fd-a777-dafaecbce99e-kube-api-access-rsmsc\" (UniqueName: \"kubernetes.io/projected/3077cb58-8c2b-48fd-a777-dafaecbce99e-kube-api-access-rsmsc\") pod \"3077cb58-8c2b-48fd-a777-dafaecbce99e\" (UID: \"3077cb58-8c2b-48fd-a777-dafaecbce99e\") "
Nov 01 03:44:33.571834 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:33.571803    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3077cb58-8c2b-48fd-a777-dafaecbce99e-plugins-config-volume" pod "3077cb58-8c2b-48fd-a777-dafaecbce99e" (UID: "3077cb58-8c2b-48fd-a777-dafaecbce99e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:44:33.572034 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:33.572012    4930 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3077cb58-8c2b-48fd-a777-dafaecbce99e-plugins-config-volume\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:44:33.573507 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:33.573475    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3077cb58-8c2b-48fd-a777-dafaecbce99e-kube-api-access-rsmsc" pod "3077cb58-8c2b-48fd-a777-dafaecbce99e" (UID: "3077cb58-8c2b-48fd-a777-dafaecbce99e"). InnerVolumeSpecName "kube-api-access-rsmsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:44:33.674714 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:33.674684    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsmsc\" (UniqueName: \"kubernetes.io/projected/3077cb58-8c2b-48fd-a777-dafaecbce99e-kube-api-access-rsmsc\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:44:33.780863 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:33.780825    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:44:34.022680 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:34.022582    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fdbdfefd-6bcb9c8569-tp459"]
Nov 01 03:44:34.134353 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:34.134317    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-79txj"]
Nov 01 03:44:34.374522 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:34.374495    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da4f30-647ffb7887-skpkm"
Nov 01 03:44:34.374522 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:34.374513    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-da4f30-646b9f5d65-qhxwx"
Nov 01 03:44:34.374522 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:34.374513    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-da4f30-647ffb7887-skpkm" event={"ID":"3077cb58-8c2b-48fd-a777-dafaecbce99e","Type":"ContainerDied","Data":"bd4630d06284298bcce64e9c8d8b07c755094ae463c9ff569f4a6b3ac28ae8d3"}
Nov 01 03:44:34.374890 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:34.374599    4930 scope.go:122] "RemoveContainer" containerID="bb02cd4d37f900d46d6afe8a6bc10998fd656ca63a82422db6841282a80ce2d9"
Nov 01 03:44:34.403449 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:34.403422    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:44:34.545741 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:34.545694    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-da4f30-647ffb7887-skpkm"]
Nov 01 03:44:34.550856 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:34.550827    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:44:34.552522 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:34.552497    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-da4f30-647ffb7887-skpkm"]
Nov 01 03:44:34.678382 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:34.678300    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-da4f30-646b9f5d65-qhxwx"]
Nov 01 03:44:34.683263 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:34.683236    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-da4f30-646b9f5d65-qhxwx"]
Nov 01 03:44:35.679581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:35.678768    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08c53d46-b65c-4799-98f7-9bd64e561ebc" path="/var/lib/kubelet/pods/08c53d46-b65c-4799-98f7-9bd64e561ebc/volumes"
Nov 01 03:44:35.679581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:35.679396    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3077cb58-8c2b-48fd-a777-dafaecbce99e" path="/var/lib/kubelet/pods/3077cb58-8c2b-48fd-a777-dafaecbce99e/volumes"
Nov 01 03:44:38.681807 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:38.681774    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:44:39.023015 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:39.022910    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-fdbdfefd-6bcb9c8569-tp459" podUID="d6670d91-12e9-4621-8f9c-627f6d722be1" containerName="payload-processing" containerID="cri-o://b5cc9d605899ff707c98dc66e9ffc287c1b0eabe8a4d5f55e5a6f3050f64e723" gracePeriod=25
Nov 01 03:44:39.075568 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:44:39.075525    4930 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6670d91_12e9_4621_8f9c_627f6d722be1.slice/crio-b5cc9d605899ff707c98dc66e9ffc287c1b0eabe8a4d5f55e5a6f3050f64e723.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6670d91_12e9_4621_8f9c_627f6d722be1.slice/crio-conmon-b5cc9d605899ff707c98dc66e9ffc287c1b0eabe8a4d5f55e5a6f3050f64e723.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 03:44:39.134814 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:39.134782    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-79txj" podUID="088b8dfa-d4c7-4667-a118-f3f5cabde5e3" containerName="payload-pre-processing" containerID="cri-o://4275033917f7d0eb7a2870cdb89104c159d4a6f82382970f4dda5e10d396e8fa" gracePeriod=25
Nov 01 03:44:39.305331 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:39.304946    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fdbdfefd-6bcb9c8569-tp459"
Nov 01 03:44:39.365636 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:39.360400    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6670d91-12e9-4621-8f9c-627f6d722be1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6670d91-12e9-4621-8f9c-627f6d722be1-plugins-config-volume\") pod \"d6670d91-12e9-4621-8f9c-627f6d722be1\" (UID: \"d6670d91-12e9-4621-8f9c-627f6d722be1\") "
Nov 01 03:44:39.365636 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:39.360475    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6670d91-12e9-4621-8f9c-627f6d722be1-kube-api-access-tpz4c\" (UniqueName: \"kubernetes.io/projected/d6670d91-12e9-4621-8f9c-627f6d722be1-kube-api-access-tpz4c\") pod \"d6670d91-12e9-4621-8f9c-627f6d722be1\" (UID: \"d6670d91-12e9-4621-8f9c-627f6d722be1\") "
Nov 01 03:44:39.365636 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:39.362024    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6670d91-12e9-4621-8f9c-627f6d722be1-plugins-config-volume" pod "d6670d91-12e9-4621-8f9c-627f6d722be1" (UID: "d6670d91-12e9-4621-8f9c-627f6d722be1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:44:39.375617 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:39.374727    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6670d91-12e9-4621-8f9c-627f6d722be1-kube-api-access-tpz4c" pod "d6670d91-12e9-4621-8f9c-627f6d722be1" (UID: "d6670d91-12e9-4621-8f9c-627f6d722be1"). InnerVolumeSpecName "kube-api-access-tpz4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:44:39.416170 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:39.416145    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-79txj"
Nov 01 03:44:39.419601 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:39.419575    4930 generic.go:356] "Generic (PLEG): container finished" podID="088b8dfa-d4c7-4667-a118-f3f5cabde5e3" containerID="4275033917f7d0eb7a2870cdb89104c159d4a6f82382970f4dda5e10d396e8fa" exitCode=0
Nov 01 03:44:39.419729 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:39.419610    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-79txj" event={"ID":"088b8dfa-d4c7-4667-a118-f3f5cabde5e3","Type":"ContainerDied","Data":"4275033917f7d0eb7a2870cdb89104c159d4a6f82382970f4dda5e10d396e8fa"}
Nov 01 03:44:39.419729 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:39.419664    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-79txj" event={"ID":"088b8dfa-d4c7-4667-a118-f3f5cabde5e3","Type":"ContainerDied","Data":"19bd7da35c14b72314953aa01abd9dfdc6cb0cb821181c51cf3c57fb64be6e0a"}
Nov 01 03:44:39.419729 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:39.419614    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-79txj"
Nov 01 03:44:39.419881 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:39.419693    4930 scope.go:122] "RemoveContainer" containerID="4275033917f7d0eb7a2870cdb89104c159d4a6f82382970f4dda5e10d396e8fa"
Nov 01 03:44:39.421165 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:39.421139    4930 generic.go:356] "Generic (PLEG): container finished" podID="d6670d91-12e9-4621-8f9c-627f6d722be1" containerID="b5cc9d605899ff707c98dc66e9ffc287c1b0eabe8a4d5f55e5a6f3050f64e723" exitCode=0
Nov 01 03:44:39.421290 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:39.421176    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fdbdfefd-6bcb9c8569-tp459"
Nov 01 03:44:39.421290 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:39.421215    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fdbdfefd-6bcb9c8569-tp459" event={"ID":"d6670d91-12e9-4621-8f9c-627f6d722be1","Type":"ContainerDied","Data":"b5cc9d605899ff707c98dc66e9ffc287c1b0eabe8a4d5f55e5a6f3050f64e723"}
Nov 01 03:44:39.421290 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:39.421254    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fdbdfefd-6bcb9c8569-tp459" event={"ID":"d6670d91-12e9-4621-8f9c-627f6d722be1","Type":"ContainerDied","Data":"7eb32af0f2ea95600728e24da73de58e3a7034a196268aba8eb3bf0926b488d6"}
Nov 01 03:44:39.433236 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:39.433214    4930 scope.go:122] "RemoveContainer" containerID="4275033917f7d0eb7a2870cdb89104c159d4a6f82382970f4dda5e10d396e8fa"
Nov 01 03:44:39.433530 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:44:39.433511    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4275033917f7d0eb7a2870cdb89104c159d4a6f82382970f4dda5e10d396e8fa\": container with ID starting with 4275033917f7d0eb7a2870cdb89104c159d4a6f82382970f4dda5e10d396e8fa not found: ID does not exist" containerID="4275033917f7d0eb7a2870cdb89104c159d4a6f82382970f4dda5e10d396e8fa"
Nov 01 03:44:39.433633 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:39.433540    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4275033917f7d0eb7a2870cdb89104c159d4a6f82382970f4dda5e10d396e8fa"} err="failed to get container status \"4275033917f7d0eb7a2870cdb89104c159d4a6f82382970f4dda5e10d396e8fa\": rpc error: code = NotFound desc = could not find container \"4275033917f7d0eb7a2870cdb89104c159d4a6f82382970f4dda5e10d396e8fa\": container with ID starting with 4275033917f7d0eb7a2870cdb89104c159d4a6f82382970f4dda5e10d396e8fa not found: ID does not exist"
Nov 01 03:44:39.433633 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:39.433581    4930 scope.go:122] "RemoveContainer" containerID="b5cc9d605899ff707c98dc66e9ffc287c1b0eabe8a4d5f55e5a6f3050f64e723"
Nov 01 03:44:39.445167 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:39.445145    4930 scope.go:122] "RemoveContainer" containerID="b5cc9d605899ff707c98dc66e9ffc287c1b0eabe8a4d5f55e5a6f3050f64e723"
Nov 01 03:44:39.445474 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:44:39.445448    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5cc9d605899ff707c98dc66e9ffc287c1b0eabe8a4d5f55e5a6f3050f64e723\": container with ID starting with b5cc9d605899ff707c98dc66e9ffc287c1b0eabe8a4d5f55e5a6f3050f64e723 not found: ID does not exist" containerID="b5cc9d605899ff707c98dc66e9ffc287c1b0eabe8a4d5f55e5a6f3050f64e723"
Nov 01 03:44:39.445547 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:39.445483    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5cc9d605899ff707c98dc66e9ffc287c1b0eabe8a4d5f55e5a6f3050f64e723"} err="failed to get container status \"b5cc9d605899ff707c98dc66e9ffc287c1b0eabe8a4d5f55e5a6f3050f64e723\": rpc error: code = NotFound desc = could not find container \"b5cc9d605899ff707c98dc66e9ffc287c1b0eabe8a4d5f55e5a6f3050f64e723\": container with ID starting with b5cc9d605899ff707c98dc66e9ffc287c1b0eabe8a4d5f55e5a6f3050f64e723 not found: ID does not exist"
Nov 01 03:44:39.464813 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:39.462337    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/088b8dfa-d4c7-4667-a118-f3f5cabde5e3-kube-api-access-nphp9\" (UniqueName: \"kubernetes.io/projected/088b8dfa-d4c7-4667-a118-f3f5cabde5e3-kube-api-access-nphp9\") pod \"088b8dfa-d4c7-4667-a118-f3f5cabde5e3\" (UID: \"088b8dfa-d4c7-4667-a118-f3f5cabde5e3\") "
Nov 01 03:44:39.464813 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:39.462668    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/088b8dfa-d4c7-4667-a118-f3f5cabde5e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/088b8dfa-d4c7-4667-a118-f3f5cabde5e3-plugins-config-volume\") pod \"088b8dfa-d4c7-4667-a118-f3f5cabde5e3\" (UID: \"088b8dfa-d4c7-4667-a118-f3f5cabde5e3\") "
Nov 01 03:44:39.464813 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:39.463190    4930 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6670d91-12e9-4621-8f9c-627f6d722be1-plugins-config-volume\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:44:39.464813 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:39.463211    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpz4c\" (UniqueName: \"kubernetes.io/projected/d6670d91-12e9-4621-8f9c-627f6d722be1-kube-api-access-tpz4c\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:44:39.464813 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:39.463611    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/088b8dfa-d4c7-4667-a118-f3f5cabde5e3-plugins-config-volume" pod "088b8dfa-d4c7-4667-a118-f3f5cabde5e3" (UID: "088b8dfa-d4c7-4667-a118-f3f5cabde5e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:44:39.471688 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:39.471658    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/088b8dfa-d4c7-4667-a118-f3f5cabde5e3-kube-api-access-nphp9" pod "088b8dfa-d4c7-4667-a118-f3f5cabde5e3" (UID: "088b8dfa-d4c7-4667-a118-f3f5cabde5e3"). InnerVolumeSpecName "kube-api-access-nphp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:44:39.480373 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:39.480293    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:44:39.564464 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:39.564391    4930 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/088b8dfa-d4c7-4667-a118-f3f5cabde5e3-plugins-config-volume\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:44:39.564464 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:39.564425    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nphp9\" (UniqueName: \"kubernetes.io/projected/088b8dfa-d4c7-4667-a118-f3f5cabde5e3-kube-api-access-nphp9\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:44:39.694922 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:39.694865    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:44:39.712326 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:39.712288    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fdbdfefd-6bcb9c8569-tp459"]
Nov 01 03:44:39.716293 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:39.716268    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fdbdfefd-6bcb9c8569-tp459"]
Nov 01 03:44:39.837221 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:39.837192    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-79txj"]
Nov 01 03:44:39.841043 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:39.841020    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fdbdfefd-8468f4575f-79txj"]
Nov 01 03:44:41.621245 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:41.621214    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="088b8dfa-d4c7-4667-a118-f3f5cabde5e3" path="/var/lib/kubelet/pods/088b8dfa-d4c7-4667-a118-f3f5cabde5e3/volumes"
Nov 01 03:44:41.621732 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:41.621714    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6670d91-12e9-4621-8f9c-627f6d722be1" path="/var/lib/kubelet/pods/d6670d91-12e9-4621-8f9c-627f6d722be1/volumes"
Nov 01 03:44:43.598425 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:43.598376    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:44:48.885661 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:48.883409    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:44:49.123154 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:44:49.123109    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:44:49.128432 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:49.128395    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:44:50.488101 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:50.488068    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:44:50.675563 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:50.675521    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:44:53.922656 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:53.922620    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:44:59.000040 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:44:59.000000    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:00.256260 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.256220    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c22dd8-29891745-vltrn"]
Nov 01 03:45:00.256921 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.256900    4930 cpu_manager.go:414] "containerMap: removing container" podUID="d6670d91-12e9-4621-8f9c-627f6d722be1" containerName="payload-processing"
Nov 01 03:45:00.256988 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.256923    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6670d91-12e9-4621-8f9c-627f6d722be1" containerName="payload-processing"
Nov 01 03:45:00.256988 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.256964    4930 cpu_manager.go:414] "containerMap: removing container" podUID="088b8dfa-d4c7-4667-a118-f3f5cabde5e3" containerName="payload-pre-processing"
Nov 01 03:45:00.256988 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.256970    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="088b8dfa-d4c7-4667-a118-f3f5cabde5e3" containerName="payload-pre-processing"
Nov 01 03:45:00.256988 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.256981    4930 cpu_manager.go:414] "containerMap: removing container" podUID="3077cb58-8c2b-48fd-a777-dafaecbce99e" containerName="payload-pre-processing"
Nov 01 03:45:00.256988 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.256986    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3077cb58-8c2b-48fd-a777-dafaecbce99e" containerName="payload-pre-processing"
Nov 01 03:45:00.257155 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.257001    4930 cpu_manager.go:414] "containerMap: removing container" podUID="08c53d46-b65c-4799-98f7-9bd64e561ebc" containerName="payload-processing"
Nov 01 03:45:00.257155 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.257006    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08c53d46-b65c-4799-98f7-9bd64e561ebc" containerName="payload-processing"
Nov 01 03:45:00.257155 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.257015    4930 cpu_manager.go:414] "containerMap: removing container" podUID="1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad" containerName="revoke-keys"
Nov 01 03:45:00.257155 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.257019    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad" containerName="revoke-keys"
Nov 01 03:45:00.257155 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.257106    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="088b8dfa-d4c7-4667-a118-f3f5cabde5e3" containerName="payload-pre-processing"
Nov 01 03:45:00.257155 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.257113    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="3077cb58-8c2b-48fd-a777-dafaecbce99e" containerName="payload-pre-processing"
Nov 01 03:45:00.257155 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.257122    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="08c53d46-b65c-4799-98f7-9bd64e561ebc" containerName="payload-processing"
Nov 01 03:45:00.257155 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.257128    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad" containerName="revoke-keys"
Nov 01 03:45:00.257155 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.257135    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6670d91-12e9-4621-8f9c-627f6d722be1" containerName="payload-processing"
Nov 01 03:45:00.262599 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.262578    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c22dd8-29891745-vltrn"
Nov 01 03:45:00.293419 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.293381    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22e11b-29891745-mpj4g"]
Nov 01 03:45:00.302798 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.302776    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-19faa5-29891745-shv69"]
Nov 01 03:45:00.302938 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.302905    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22e11b-29891745-mpj4g"
Nov 01 03:45:00.307618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.307596    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891745-k78n5"]
Nov 01 03:45:00.307748 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.307727    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-19faa5-29891745-shv69"
Nov 01 03:45:00.311491 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.311470    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-fdbdfefd-29891745-xpl9r"]
Nov 01 03:45:00.311635 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.311575    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891745-k78n5"
Nov 01 03:45:00.317117 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.317089    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891745-g6s6m"]
Nov 01 03:45:00.324752 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.324727    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891745-g6s6m"
Nov 01 03:45:00.324860 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.324762    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-fdbdfefd-29891745-xpl9r"
Nov 01 03:45:00.329373 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.329348    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891745-fdqkt"]
Nov 01 03:45:00.329551 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.329531    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:45:00.329687 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.329661    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:45:00.333486 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.333465    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891745-fdqkt"
Nov 01 03:45:00.353467 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.353443    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2c8373-29891745-m2hl2"]
Nov 01 03:45:00.357912 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.357891    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2c8373-29891745-m2hl2"
Nov 01 03:45:00.358501 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.358477    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-da4f30-29891745-8trrg"]
Nov 01 03:45:00.358645 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.358523    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx2nn\" (UniqueName: \"kubernetes.io/projected/77c03a7b-ad42-424e-851f-fa2bcd3e1b5e-kube-api-access-zx2nn\") pod \"maas-api-key-cleanup-e2e-worker-w6-19faa5-29891745-shv69\" (UID: \"77c03a7b-ad42-424e-851f-fa2bcd3e1b5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-19faa5-29891745-shv69"
Nov 01 03:45:00.358727 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.358648    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gk9w\" (UniqueName: \"kubernetes.io/projected/b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950-kube-api-access-4gk9w\") pod \"maas-api-key-cleanup-e2e-worker-w5-c22dd8-29891745-vltrn\" (UID: \"b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c22dd8-29891745-vltrn"
Nov 01 03:45:00.358727 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.358699    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/146c7163-c4db-4c71-85a1-3536ebdeec82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891745-k78n5\" (UID: \"146c7163-c4db-4c71-85a1-3536ebdeec82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891745-k78n5"
Nov 01 03:45:00.358923 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.358727    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97sb2\" (UniqueName: \"kubernetes.io/projected/146c7163-c4db-4c71-85a1-3536ebdeec82-kube-api-access-97sb2\") pod \"maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891745-k78n5\" (UID: \"146c7163-c4db-4c71-85a1-3536ebdeec82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891745-k78n5"
Nov 01 03:45:00.358923 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.358840    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77c03a7b-ad42-424e-851f-fa2bcd3e1b5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-19faa5-29891745-shv69\" (UID: \"77c03a7b-ad42-424e-851f-fa2bcd3e1b5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-19faa5-29891745-shv69"
Nov 01 03:45:00.359046 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.358956    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08723624-3a83-4821-8d9a-716ca8847aff-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891745-g6s6m\" (UID: \"08723624-3a83-4821-8d9a-716ca8847aff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891745-g6s6m"
Nov 01 03:45:00.359046 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.358983    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3876a597-4092-43ab-b4a5-c89cb1f1aa46-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-22e11b-29891745-mpj4g\" (UID: \"3876a597-4092-43ab-b4a5-c89cb1f1aa46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22e11b-29891745-mpj4g"
Nov 01 03:45:00.359046 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.359031    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dqnq\" (UniqueName: \"kubernetes.io/projected/3876a597-4092-43ab-b4a5-c89cb1f1aa46-kube-api-access-7dqnq\") pod \"maas-api-key-cleanup-e2e-inf-a-22e11b-29891745-mpj4g\" (UID: \"3876a597-4092-43ab-b4a5-c89cb1f1aa46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22e11b-29891745-mpj4g"
Nov 01 03:45:00.359220 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.359116    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x9pt\" (UniqueName: \"kubernetes.io/projected/f8b3a0cd-6f72-4296-ae23-c03fc6e42613-kube-api-access-6x9pt\") pod \"maas-api-key-cleanup-e2e-mt-fdbdfefd-29891745-xpl9r\" (UID: \"f8b3a0cd-6f72-4296-ae23-c03fc6e42613\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-fdbdfefd-29891745-xpl9r"
Nov 01 03:45:00.359220 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.359152    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rpgv\" (UniqueName: \"kubernetes.io/projected/08723624-3a83-4821-8d9a-716ca8847aff-kube-api-access-2rpgv\") pod \"maas-api-key-cleanup-29891745-g6s6m\" (UID: \"08723624-3a83-4821-8d9a-716ca8847aff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891745-g6s6m"
Nov 01 03:45:00.359220 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.359188    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8b3a0cd-6f72-4296-ae23-c03fc6e42613-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-fdbdfefd-29891745-xpl9r\" (UID: \"f8b3a0cd-6f72-4296-ae23-c03fc6e42613\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-fdbdfefd-29891745-xpl9r"
Nov 01 03:45:00.359408 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.359221    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-c22dd8-29891745-vltrn\" (UID: \"b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c22dd8-29891745-vltrn"
Nov 01 03:45:00.363302 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.363269    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-da4f30-29891745-8trrg"
Nov 01 03:45:00.371998 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.371973    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891745-k78n5"]
Nov 01 03:45:00.373527 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.372921    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:45:00.380005 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.379983    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22e11b-29891745-mpj4g"]
Nov 01 03:45:00.393761 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.388407    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c22dd8-29891745-vltrn"]
Nov 01 03:45:00.393761 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.388920    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-7cd000-29891745-r2xxc"]
Nov 01 03:45:00.394293 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.394274    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-7cd000-29891745-r2xxc"
Nov 01 03:45:00.423778 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.423751    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891745-g6s6m"]
Nov 01 03:45:00.430524 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.430490    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-fdbdfefd-29891745-xpl9r"]
Nov 01 03:45:00.440130 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.439059    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891745-fdqkt"]
Nov 01 03:45:00.443385 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.443355    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-da4f30-29891745-8trrg"]
Nov 01 03:45:00.449435 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.448626    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-7cd000-29891745-r2xxc"]
Nov 01 03:45:00.475579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.460301    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8b3a0cd-6f72-4296-ae23-c03fc6e42613-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-fdbdfefd-29891745-xpl9r\" (UID: \"f8b3a0cd-6f72-4296-ae23-c03fc6e42613\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-fdbdfefd-29891745-xpl9r"
Nov 01 03:45:00.475579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.460353    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-c22dd8-29891745-vltrn\" (UID: \"b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c22dd8-29891745-vltrn"
Nov 01 03:45:00.475579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.460466    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf8xm\" (UniqueName: \"kubernetes.io/projected/5aa6bb82-d7e1-410c-9118-e7feed83bad8-kube-api-access-tf8xm\") pod \"maas-api-key-cleanup-e2e-api-b-da4f30-29891745-8trrg\" (UID: \"5aa6bb82-d7e1-410c-9118-e7feed83bad8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-da4f30-29891745-8trrg"
Nov 01 03:45:00.475579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.460579    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drj6x\" (UniqueName: \"kubernetes.io/projected/190a288b-699e-4719-949e-ec24a8dd155b-kube-api-access-drj6x\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-7cd000-29891745-r2xxc\" (UID: \"190a288b-699e-4719-949e-ec24a8dd155b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-7cd000-29891745-r2xxc"
Nov 01 03:45:00.475579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.460621    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx2nn\" (UniqueName: \"kubernetes.io/projected/77c03a7b-ad42-424e-851f-fa2bcd3e1b5e-kube-api-access-zx2nn\") pod \"maas-api-key-cleanup-e2e-worker-w6-19faa5-29891745-shv69\" (UID: \"77c03a7b-ad42-424e-851f-fa2bcd3e1b5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-19faa5-29891745-shv69"
Nov 01 03:45:00.475579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.460688    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gk9w\" (UniqueName: \"kubernetes.io/projected/b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950-kube-api-access-4gk9w\") pod \"maas-api-key-cleanup-e2e-worker-w5-c22dd8-29891745-vltrn\" (UID: \"b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c22dd8-29891745-vltrn"
Nov 01 03:45:00.475579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.460724    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2bq4\" (UniqueName: \"kubernetes.io/projected/5df89ff2-793b-4eb5-9920-236b2674f026-kube-api-access-f2bq4\") pod \"maas-api-key-cleanup-e2e-api-a-2c8373-29891745-m2hl2\" (UID: \"5df89ff2-793b-4eb5-9920-236b2674f026\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2c8373-29891745-m2hl2"
Nov 01 03:45:00.475579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.460755    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/146c7163-c4db-4c71-85a1-3536ebdeec82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891745-k78n5\" (UID: \"146c7163-c4db-4c71-85a1-3536ebdeec82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891745-k78n5"
Nov 01 03:45:00.475579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.460777    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97sb2\" (UniqueName: \"kubernetes.io/projected/146c7163-c4db-4c71-85a1-3536ebdeec82-kube-api-access-97sb2\") pod \"maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891745-k78n5\" (UID: \"146c7163-c4db-4c71-85a1-3536ebdeec82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891745-k78n5"
Nov 01 03:45:00.475579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.460817    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aa6bb82-d7e1-410c-9118-e7feed83bad8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-da4f30-29891745-8trrg\" (UID: \"5aa6bb82-d7e1-410c-9118-e7feed83bad8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-da4f30-29891745-8trrg"
Nov 01 03:45:00.475579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.460889    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77c03a7b-ad42-424e-851f-fa2bcd3e1b5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-19faa5-29891745-shv69\" (UID: \"77c03a7b-ad42-424e-851f-fa2bcd3e1b5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-19faa5-29891745-shv69"
Nov 01 03:45:00.475579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.460984    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83bd0b30-2532-40b5-b5ab-e417f56cbe19-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d7f535-29891745-fdqkt\" (UID: \"83bd0b30-2532-40b5-b5ab-e417f56cbe19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891745-fdqkt"
Nov 01 03:45:00.475579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.461048    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08723624-3a83-4821-8d9a-716ca8847aff-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891745-g6s6m\" (UID: \"08723624-3a83-4821-8d9a-716ca8847aff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891745-g6s6m"
Nov 01 03:45:00.475579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.461095    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3876a597-4092-43ab-b4a5-c89cb1f1aa46-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-22e11b-29891745-mpj4g\" (UID: \"3876a597-4092-43ab-b4a5-c89cb1f1aa46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22e11b-29891745-mpj4g"
Nov 01 03:45:00.475579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.461153    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dqnq\" (UniqueName: \"kubernetes.io/projected/3876a597-4092-43ab-b4a5-c89cb1f1aa46-kube-api-access-7dqnq\") pod \"maas-api-key-cleanup-e2e-inf-a-22e11b-29891745-mpj4g\" (UID: \"3876a597-4092-43ab-b4a5-c89cb1f1aa46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22e11b-29891745-mpj4g"
Nov 01 03:45:00.475579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.461169    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8b3a0cd-6f72-4296-ae23-c03fc6e42613-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-fdbdfefd-29891745-xpl9r\" (UID: \"f8b3a0cd-6f72-4296-ae23-c03fc6e42613\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-fdbdfefd-29891745-xpl9r"
Nov 01 03:45:00.475579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.461239    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/190a288b-699e-4719-949e-ec24a8dd155b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-7cd000-29891745-r2xxc\" (UID: \"190a288b-699e-4719-949e-ec24a8dd155b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-7cd000-29891745-r2xxc"
Nov 01 03:45:00.475579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.461300    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df89ff2-793b-4eb5-9920-236b2674f026-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-2c8373-29891745-m2hl2\" (UID: \"5df89ff2-793b-4eb5-9920-236b2674f026\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2c8373-29891745-m2hl2"
Nov 01 03:45:00.475579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.461354    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x9pt\" (UniqueName: \"kubernetes.io/projected/f8b3a0cd-6f72-4296-ae23-c03fc6e42613-kube-api-access-6x9pt\") pod \"maas-api-key-cleanup-e2e-mt-fdbdfefd-29891745-xpl9r\" (UID: \"f8b3a0cd-6f72-4296-ae23-c03fc6e42613\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-fdbdfefd-29891745-xpl9r"
Nov 01 03:45:00.475579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.461405    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rpgv\" (UniqueName: \"kubernetes.io/projected/08723624-3a83-4821-8d9a-716ca8847aff-kube-api-access-2rpgv\") pod \"maas-api-key-cleanup-29891745-g6s6m\" (UID: \"08723624-3a83-4821-8d9a-716ca8847aff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891745-g6s6m"
Nov 01 03:45:00.475579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.461443    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrc9l\" (UniqueName: \"kubernetes.io/projected/83bd0b30-2532-40b5-b5ab-e417f56cbe19-kube-api-access-jrc9l\") pod \"maas-api-key-cleanup-e2e-worker-w3-d7f535-29891745-fdqkt\" (UID: \"83bd0b30-2532-40b5-b5ab-e417f56cbe19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891745-fdqkt"
Nov 01 03:45:00.475579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.462266    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-c22dd8-29891745-vltrn\" (UID: \"b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c22dd8-29891745-vltrn"
Nov 01 03:45:00.475579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.462731    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/146c7163-c4db-4c71-85a1-3536ebdeec82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891745-k78n5\" (UID: \"146c7163-c4db-4c71-85a1-3536ebdeec82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891745-k78n5"
Nov 01 03:45:00.475579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.463138    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77c03a7b-ad42-424e-851f-fa2bcd3e1b5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-19faa5-29891745-shv69\" (UID: \"77c03a7b-ad42-424e-851f-fa2bcd3e1b5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-19faa5-29891745-shv69"
Nov 01 03:45:00.475579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.463881    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08723624-3a83-4821-8d9a-716ca8847aff-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891745-g6s6m\" (UID: \"08723624-3a83-4821-8d9a-716ca8847aff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891745-g6s6m"
Nov 01 03:45:00.475579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.463927    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3876a597-4092-43ab-b4a5-c89cb1f1aa46-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-22e11b-29891745-mpj4g\" (UID: \"3876a597-4092-43ab-b4a5-c89cb1f1aa46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22e11b-29891745-mpj4g"
Nov 01 03:45:00.480624 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.480034    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-19faa5-29891745-shv69"]
Nov 01 03:45:00.497623 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.496139    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2c8373-29891745-m2hl2"]
Nov 01 03:45:00.530511 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.530349    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gk9w\" (UniqueName: \"kubernetes.io/projected/b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950-kube-api-access-4gk9w\") pod \"maas-api-key-cleanup-e2e-worker-w5-c22dd8-29891745-vltrn\" (UID: \"b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c22dd8-29891745-vltrn"
Nov 01 03:45:00.533282 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.533254    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx2nn\" (UniqueName: \"kubernetes.io/projected/77c03a7b-ad42-424e-851f-fa2bcd3e1b5e-kube-api-access-zx2nn\") pod \"maas-api-key-cleanup-e2e-worker-w6-19faa5-29891745-shv69\" (UID: \"77c03a7b-ad42-424e-851f-fa2bcd3e1b5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-19faa5-29891745-shv69"
Nov 01 03:45:00.544183 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.541140    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97sb2\" (UniqueName: \"kubernetes.io/projected/146c7163-c4db-4c71-85a1-3536ebdeec82-kube-api-access-97sb2\") pod \"maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891745-k78n5\" (UID: \"146c7163-c4db-4c71-85a1-3536ebdeec82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891745-k78n5"
Nov 01 03:45:00.544183 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.541809    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dqnq\" (UniqueName: \"kubernetes.io/projected/3876a597-4092-43ab-b4a5-c89cb1f1aa46-kube-api-access-7dqnq\") pod \"maas-api-key-cleanup-e2e-inf-a-22e11b-29891745-mpj4g\" (UID: \"3876a597-4092-43ab-b4a5-c89cb1f1aa46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22e11b-29891745-mpj4g"
Nov 01 03:45:00.544183 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.542451    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x9pt\" (UniqueName: \"kubernetes.io/projected/f8b3a0cd-6f72-4296-ae23-c03fc6e42613-kube-api-access-6x9pt\") pod \"maas-api-key-cleanup-e2e-mt-fdbdfefd-29891745-xpl9r\" (UID: \"f8b3a0cd-6f72-4296-ae23-c03fc6e42613\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-fdbdfefd-29891745-xpl9r"
Nov 01 03:45:00.544850 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.544821    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rpgv\" (UniqueName: \"kubernetes.io/projected/08723624-3a83-4821-8d9a-716ca8847aff-kube-api-access-2rpgv\") pod \"maas-api-key-cleanup-29891745-g6s6m\" (UID: \"08723624-3a83-4821-8d9a-716ca8847aff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891745-g6s6m"
Nov 01 03:45:00.563272 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.563241    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2bq4\" (UniqueName: \"kubernetes.io/projected/5df89ff2-793b-4eb5-9920-236b2674f026-kube-api-access-f2bq4\") pod \"maas-api-key-cleanup-e2e-api-a-2c8373-29891745-m2hl2\" (UID: \"5df89ff2-793b-4eb5-9920-236b2674f026\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2c8373-29891745-m2hl2"
Nov 01 03:45:00.563413 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.563298    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aa6bb82-d7e1-410c-9118-e7feed83bad8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-da4f30-29891745-8trrg\" (UID: \"5aa6bb82-d7e1-410c-9118-e7feed83bad8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-da4f30-29891745-8trrg"
Nov 01 03:45:00.563413 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.563346    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83bd0b30-2532-40b5-b5ab-e417f56cbe19-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d7f535-29891745-fdqkt\" (UID: \"83bd0b30-2532-40b5-b5ab-e417f56cbe19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891745-fdqkt"
Nov 01 03:45:00.563413 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.563403    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/190a288b-699e-4719-949e-ec24a8dd155b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-7cd000-29891745-r2xxc\" (UID: \"190a288b-699e-4719-949e-ec24a8dd155b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-7cd000-29891745-r2xxc"
Nov 01 03:45:00.563641 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.563424    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df89ff2-793b-4eb5-9920-236b2674f026-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-2c8373-29891745-m2hl2\" (UID: \"5df89ff2-793b-4eb5-9920-236b2674f026\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2c8373-29891745-m2hl2"
Nov 01 03:45:00.563641 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.563463    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrc9l\" (UniqueName: \"kubernetes.io/projected/83bd0b30-2532-40b5-b5ab-e417f56cbe19-kube-api-access-jrc9l\") pod \"maas-api-key-cleanup-e2e-worker-w3-d7f535-29891745-fdqkt\" (UID: \"83bd0b30-2532-40b5-b5ab-e417f56cbe19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891745-fdqkt"
Nov 01 03:45:00.563641 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.563514    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf8xm\" (UniqueName: \"kubernetes.io/projected/5aa6bb82-d7e1-410c-9118-e7feed83bad8-kube-api-access-tf8xm\") pod \"maas-api-key-cleanup-e2e-api-b-da4f30-29891745-8trrg\" (UID: \"5aa6bb82-d7e1-410c-9118-e7feed83bad8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-da4f30-29891745-8trrg"
Nov 01 03:45:00.563641 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.563548    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drj6x\" (UniqueName: \"kubernetes.io/projected/190a288b-699e-4719-949e-ec24a8dd155b-kube-api-access-drj6x\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-7cd000-29891745-r2xxc\" (UID: \"190a288b-699e-4719-949e-ec24a8dd155b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-7cd000-29891745-r2xxc"
Nov 01 03:45:00.565381 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.564324    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/190a288b-699e-4719-949e-ec24a8dd155b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-7cd000-29891745-r2xxc\" (UID: \"190a288b-699e-4719-949e-ec24a8dd155b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-7cd000-29891745-r2xxc"
Nov 01 03:45:00.565381 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.564967    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df89ff2-793b-4eb5-9920-236b2674f026-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-2c8373-29891745-m2hl2\" (UID: \"5df89ff2-793b-4eb5-9920-236b2674f026\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2c8373-29891745-m2hl2"
Nov 01 03:45:00.566016 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.565838    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83bd0b30-2532-40b5-b5ab-e417f56cbe19-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d7f535-29891745-fdqkt\" (UID: \"83bd0b30-2532-40b5-b5ab-e417f56cbe19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891745-fdqkt"
Nov 01 03:45:00.566492 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.566465    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aa6bb82-d7e1-410c-9118-e7feed83bad8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-da4f30-29891745-8trrg\" (UID: \"5aa6bb82-d7e1-410c-9118-e7feed83bad8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-da4f30-29891745-8trrg"
Nov 01 03:45:00.580522 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.580496    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c22dd8-29891745-vltrn"
Nov 01 03:45:00.625725 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.625697    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrc9l\" (UniqueName: \"kubernetes.io/projected/83bd0b30-2532-40b5-b5ab-e417f56cbe19-kube-api-access-jrc9l\") pod \"maas-api-key-cleanup-e2e-worker-w3-d7f535-29891745-fdqkt\" (UID: \"83bd0b30-2532-40b5-b5ab-e417f56cbe19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891745-fdqkt"
Nov 01 03:45:00.626221 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.626195    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drj6x\" (UniqueName: \"kubernetes.io/projected/190a288b-699e-4719-949e-ec24a8dd155b-kube-api-access-drj6x\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-7cd000-29891745-r2xxc\" (UID: \"190a288b-699e-4719-949e-ec24a8dd155b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-7cd000-29891745-r2xxc"
Nov 01 03:45:00.630157 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.630102    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf8xm\" (UniqueName: \"kubernetes.io/projected/5aa6bb82-d7e1-410c-9118-e7feed83bad8-kube-api-access-tf8xm\") pod \"maas-api-key-cleanup-e2e-api-b-da4f30-29891745-8trrg\" (UID: \"5aa6bb82-d7e1-410c-9118-e7feed83bad8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-da4f30-29891745-8trrg"
Nov 01 03:45:00.630423 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.630405    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2bq4\" (UniqueName: \"kubernetes.io/projected/5df89ff2-793b-4eb5-9920-236b2674f026-kube-api-access-f2bq4\") pod \"maas-api-key-cleanup-e2e-api-a-2c8373-29891745-m2hl2\" (UID: \"5df89ff2-793b-4eb5-9920-236b2674f026\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2c8373-29891745-m2hl2"
Nov 01 03:45:00.634034 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.634010    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22e11b-29891745-mpj4g"
Nov 01 03:45:00.643199 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.642931    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-19faa5-29891745-shv69"
Nov 01 03:45:00.650811 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.648906    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891745-k78n5"
Nov 01 03:45:00.658395 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.657770    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891745-g6s6m"
Nov 01 03:45:00.675000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.674799    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-fdbdfefd-29891745-xpl9r"
Nov 01 03:45:00.699416 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.699241    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891745-fdqkt"
Nov 01 03:45:00.729131 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.728909    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2c8373-29891745-m2hl2"
Nov 01 03:45:00.731880 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.731825    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-da4f30-29891745-8trrg"
Nov 01 03:45:00.762111 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.761863    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-7cd000-29891745-r2xxc"
Nov 01 03:45:00.977103 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:00.976711    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c22dd8-29891745-vltrn"]
Nov 01 03:45:01.289667 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:01.289638    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22e11b-29891745-mpj4g"]
Nov 01 03:45:01.290128 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:45:01.290097    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3876a597_4092_43ab_b4a5_c89cb1f1aa46.slice/crio-53ad6d4d333820b051b654f3be236fca0337ea1c348203e258ed415c51d39323 WatchSource:0}: Error finding container 53ad6d4d333820b051b654f3be236fca0337ea1c348203e258ed415c51d39323: Status 404 returned error can't find the container with id 53ad6d4d333820b051b654f3be236fca0337ea1c348203e258ed415c51d39323
Nov 01 03:45:01.333675 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:45:01.333642    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77c03a7b_ad42_424e_851f_fa2bcd3e1b5e.slice/crio-eb7affa6a4770b087be5497e1f269a494e5cfe6c6f75cde4bbdff7150ac1fcb0 WatchSource:0}: Error finding container eb7affa6a4770b087be5497e1f269a494e5cfe6c6f75cde4bbdff7150ac1fcb0: Status 404 returned error can't find the container with id eb7affa6a4770b087be5497e1f269a494e5cfe6c6f75cde4bbdff7150ac1fcb0
Nov 01 03:45:01.334015 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:01.333972    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-19faa5-29891745-shv69"]
Nov 01 03:45:01.338455 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:45:01.338422    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146c7163_c4db_4c71_85a1_3536ebdeec82.slice/crio-1bad95cacae8c1ba50a279aee607454bbb1854059975d5aadcfb8d380a2e6794 WatchSource:0}: Error finding container 1bad95cacae8c1ba50a279aee607454bbb1854059975d5aadcfb8d380a2e6794: Status 404 returned error can't find the container with id 1bad95cacae8c1ba50a279aee607454bbb1854059975d5aadcfb8d380a2e6794
Nov 01 03:45:01.339325 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:01.339302    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891745-k78n5"]
Nov 01 03:45:01.419541 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:01.419513    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891745-fdqkt"]
Nov 01 03:45:01.474667 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:45:01.474624    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83bd0b30_2532_40b5_b5ab_e417f56cbe19.slice/crio-a1d63623e1e272b349b99c0ac5f3a56e66abaa58da2525967a075c9a8270eb92 WatchSource:0}: Error finding container a1d63623e1e272b349b99c0ac5f3a56e66abaa58da2525967a075c9a8270eb92: Status 404 returned error can't find the container with id a1d63623e1e272b349b99c0ac5f3a56e66abaa58da2525967a075c9a8270eb92
Nov 01 03:45:01.492158 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:45:01.491974    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aa6bb82_d7e1_410c_9118_e7feed83bad8.slice/crio-e50a4b040a4324cb45d4b970511e7ac6ad3573660a67300cbe958fa43647dde3 WatchSource:0}: Error finding container e50a4b040a4324cb45d4b970511e7ac6ad3573660a67300cbe958fa43647dde3: Status 404 returned error can't find the container with id e50a4b040a4324cb45d4b970511e7ac6ad3573660a67300cbe958fa43647dde3
Nov 01 03:45:01.498814 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:01.492311    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891745-g6s6m"]
Nov 01 03:45:01.507631 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:01.507597    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-da4f30-29891745-8trrg"]
Nov 01 03:45:01.514795 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:01.513715    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2c8373-29891745-m2hl2"]
Nov 01 03:45:01.514795 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:45:01.513983    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5df89ff2_793b_4eb5_9920_236b2674f026.slice/crio-458bae37fa90c2e300b9b70f3c7687b3e33d59f08039f3a9c077c99a87d1a760 WatchSource:0}: Error finding container 458bae37fa90c2e300b9b70f3c7687b3e33d59f08039f3a9c077c99a87d1a760: Status 404 returned error can't find the container with id 458bae37fa90c2e300b9b70f3c7687b3e33d59f08039f3a9c077c99a87d1a760
Nov 01 03:45:01.517850 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:45:01.517825    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8b3a0cd_6f72_4296_ae23_c03fc6e42613.slice/crio-a7f7183e72cd28f1f93e7a1a25da735e353114331787dd6e8f909c57ac0437a3 WatchSource:0}: Error finding container a7f7183e72cd28f1f93e7a1a25da735e353114331787dd6e8f909c57ac0437a3: Status 404 returned error can't find the container with id a7f7183e72cd28f1f93e7a1a25da735e353114331787dd6e8f909c57ac0437a3
Nov 01 03:45:01.520611 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:01.520544    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-fdbdfefd-29891745-xpl9r"]
Nov 01 03:45:01.549235 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:01.549215    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-7cd000-29891745-r2xxc"]
Nov 01 03:45:01.565117 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:45:01.565093    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod190a288b_699e_4719_949e_ec24a8dd155b.slice/crio-55082a85f42a5b8ae08f4fdc826e2f1b66dcb956bc4faeaebbb1949b086ede4d WatchSource:0}: Error finding container 55082a85f42a5b8ae08f4fdc826e2f1b66dcb956bc4faeaebbb1949b086ede4d: Status 404 returned error can't find the container with id 55082a85f42a5b8ae08f4fdc826e2f1b66dcb956bc4faeaebbb1949b086ede4d
Nov 01 03:45:01.610666 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:01.610631    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891745-g6s6m" event={"ID":"08723624-3a83-4821-8d9a-716ca8847aff","Type":"ContainerStarted","Data":"bc35b93bcf6a65b0451472f4d4470e759dad5b5c3d57d758003d8242ad7b0956"}
Nov 01 03:45:01.612236 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:01.612209    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c22dd8-29891745-vltrn" event={"ID":"b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950","Type":"ContainerStarted","Data":"da2e2470f38ab1a8685d460a1b738a70e9dc180f401486d00af1c1350a69464f"}
Nov 01 03:45:01.612336 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:01.612286    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c22dd8-29891745-vltrn" event={"ID":"b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950","Type":"ContainerStarted","Data":"10cb9d5125d6b439d284e903486843aae45a781b7d2e1b2635c174c001041f0a"}
Nov 01 03:45:01.628715 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:01.628687    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-da4f30-29891745-8trrg" event={"ID":"5aa6bb82-d7e1-410c-9118-e7feed83bad8","Type":"ContainerStarted","Data":"e50a4b040a4324cb45d4b970511e7ac6ad3573660a67300cbe958fa43647dde3"}
Nov 01 03:45:01.628808 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:01.628733    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-19faa5-29891745-shv69" event={"ID":"77c03a7b-ad42-424e-851f-fa2bcd3e1b5e","Type":"ContainerStarted","Data":"eb7affa6a4770b087be5497e1f269a494e5cfe6c6f75cde4bbdff7150ac1fcb0"}
Nov 01 03:45:01.628808 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:01.628751    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891745-k78n5" event={"ID":"146c7163-c4db-4c71-85a1-3536ebdeec82","Type":"ContainerStarted","Data":"1bad95cacae8c1ba50a279aee607454bbb1854059975d5aadcfb8d380a2e6794"}
Nov 01 03:45:01.629735 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:01.629714    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-fdbdfefd-29891745-xpl9r" event={"ID":"f8b3a0cd-6f72-4296-ae23-c03fc6e42613","Type":"ContainerStarted","Data":"a7f7183e72cd28f1f93e7a1a25da735e353114331787dd6e8f909c57ac0437a3"}
Nov 01 03:45:01.630692 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:01.630673    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891745-fdqkt" event={"ID":"83bd0b30-2532-40b5-b5ab-e417f56cbe19","Type":"ContainerStarted","Data":"a1d63623e1e272b349b99c0ac5f3a56e66abaa58da2525967a075c9a8270eb92"}
Nov 01 03:45:01.631572 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:01.631542    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-7cd000-29891745-r2xxc" event={"ID":"190a288b-699e-4719-949e-ec24a8dd155b","Type":"ContainerStarted","Data":"55082a85f42a5b8ae08f4fdc826e2f1b66dcb956bc4faeaebbb1949b086ede4d"}
Nov 01 03:45:01.632883 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:01.632860    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2c8373-29891745-m2hl2" event={"ID":"5df89ff2-793b-4eb5-9920-236b2674f026","Type":"ContainerStarted","Data":"458bae37fa90c2e300b9b70f3c7687b3e33d59f08039f3a9c077c99a87d1a760"}
Nov 01 03:45:01.634823 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:01.634791    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22e11b-29891745-mpj4g" event={"ID":"3876a597-4092-43ab-b4a5-c89cb1f1aa46","Type":"ContainerStarted","Data":"53ad6d4d333820b051b654f3be236fca0337ea1c348203e258ed415c51d39323"}
Nov 01 03:45:01.742098 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:01.742051    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c22dd8-29891745-vltrn" podStartSLOduration=1.742036932 podStartE2EDuration="1.742036932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:45:01.737781786 +0000 UTC m=+2306.260788373" watchObservedRunningTime="2026-11-01 03:45:01.742036932 +0000 UTC m=+2306.265043557"
Nov 01 03:45:02.647861 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:02.647829    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-7cd000-29891745-r2xxc" event={"ID":"190a288b-699e-4719-949e-ec24a8dd155b","Type":"ContainerStarted","Data":"5ee1eebcce77f49f3527da43ea0fff6ac857c971112600dc7a34d6c18770518b"}
Nov 01 03:45:02.649598 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:02.649542    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2c8373-29891745-m2hl2" event={"ID":"5df89ff2-793b-4eb5-9920-236b2674f026","Type":"ContainerStarted","Data":"935c4d98837fba4a2df2f4b49553d044dca3471acc1722c204f15ca1502f6184"}
Nov 01 03:45:02.651207 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:02.651183    4930 generic.go:356] "Generic (PLEG): container finished" podID="3876a597-4092-43ab-b4a5-c89cb1f1aa46" containerID="667b21841c0be75934078216ad22f55f6fabb600ca79388a50e86b2080cb5f52" exitCode=60
Nov 01 03:45:02.651332 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:02.651256    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22e11b-29891745-mpj4g" event={"ID":"3876a597-4092-43ab-b4a5-c89cb1f1aa46","Type":"ContainerDied","Data":"667b21841c0be75934078216ad22f55f6fabb600ca79388a50e86b2080cb5f52"}
Nov 01 03:45:02.651432 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:02.651416    4930 scope.go:122] "RemoveContainer" containerID="667b21841c0be75934078216ad22f55f6fabb600ca79388a50e86b2080cb5f52"
Nov 01 03:45:02.653229 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:02.653209    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891745-g6s6m" event={"ID":"08723624-3a83-4821-8d9a-716ca8847aff","Type":"ContainerStarted","Data":"8333ca5244e1c116e6014d7bbfaef25d2231138b3eb0b3ba9162c14fa112d01b"}
Nov 01 03:45:02.655053 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:02.655029    4930 generic.go:356] "Generic (PLEG): container finished" podID="b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950" containerID="da2e2470f38ab1a8685d460a1b738a70e9dc180f401486d00af1c1350a69464f" exitCode=60
Nov 01 03:45:02.655164 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:02.655140    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c22dd8-29891745-vltrn" event={"ID":"b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950","Type":"ContainerDied","Data":"da2e2470f38ab1a8685d460a1b738a70e9dc180f401486d00af1c1350a69464f"}
Nov 01 03:45:02.655312 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:02.655295    4930 scope.go:122] "RemoveContainer" containerID="da2e2470f38ab1a8685d460a1b738a70e9dc180f401486d00af1c1350a69464f"
Nov 01 03:45:02.656872 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:02.656852    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-da4f30-29891745-8trrg" event={"ID":"5aa6bb82-d7e1-410c-9118-e7feed83bad8","Type":"ContainerStarted","Data":"4e62bfc621793cb3de41ecf96788ef5fe5ea14f1ba2bc7b53bee62ce0ab66a85"}
Nov 01 03:45:02.658492 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:02.658467    4930 generic.go:356] "Generic (PLEG): container finished" podID="77c03a7b-ad42-424e-851f-fa2bcd3e1b5e" containerID="43e9a4678a79383dce2fb05c1aead8c723d628f344d3803a75d3c9b722a4e107" exitCode=60
Nov 01 03:45:02.658591 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:02.658539    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-19faa5-29891745-shv69" event={"ID":"77c03a7b-ad42-424e-851f-fa2bcd3e1b5e","Type":"ContainerDied","Data":"43e9a4678a79383dce2fb05c1aead8c723d628f344d3803a75d3c9b722a4e107"}
Nov 01 03:45:02.658781 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:02.658750    4930 scope.go:122] "RemoveContainer" containerID="43e9a4678a79383dce2fb05c1aead8c723d628f344d3803a75d3c9b722a4e107"
Nov 01 03:45:02.662505 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:02.662482    4930 generic.go:356] "Generic (PLEG): container finished" podID="146c7163-c4db-4c71-85a1-3536ebdeec82" containerID="4dc63996d1a6d3908e4a374f3deeedc15399284217c7a53658e6a377c5c8c409" exitCode=60
Nov 01 03:45:02.662656 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:02.662634    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891745-k78n5" event={"ID":"146c7163-c4db-4c71-85a1-3536ebdeec82","Type":"ContainerDied","Data":"4dc63996d1a6d3908e4a374f3deeedc15399284217c7a53658e6a377c5c8c409"}
Nov 01 03:45:02.663252 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:02.662983    4930 scope.go:122] "RemoveContainer" containerID="4dc63996d1a6d3908e4a374f3deeedc15399284217c7a53658e6a377c5c8c409"
Nov 01 03:45:02.665903 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:02.665879    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-fdbdfefd-29891745-xpl9r" event={"ID":"f8b3a0cd-6f72-4296-ae23-c03fc6e42613","Type":"ContainerStarted","Data":"0815a5c621f0ba3c2f5d2826ae6e181668ee1b61e96961d5cb55c3f3784832f4"}
Nov 01 03:45:02.669448 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:02.669330    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891745-fdqkt" event={"ID":"83bd0b30-2532-40b5-b5ab-e417f56cbe19","Type":"ContainerStarted","Data":"acd431b678b9c183bed2d1df0b03c1a1307418152413577c25178f47a7e94a58"}
Nov 01 03:45:02.774633 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:02.774575    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-7cd000-29891745-r2xxc" podStartSLOduration=2.774547481 podStartE2EDuration="2.774547481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:45:02.763043357 +0000 UTC m=+2307.286049945" watchObservedRunningTime="2026-11-01 03:45:02.774547481 +0000 UTC m=+2307.297554066"
Nov 01 03:45:02.838252 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:02.838215    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:02.887703 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:02.887644    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891745-fdqkt" podStartSLOduration=2.887623911 podStartE2EDuration="2.887623911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:45:02.877909361 +0000 UTC m=+2307.400915951" watchObservedRunningTime="2026-11-01 03:45:02.887623911 +0000 UTC m=+2307.410630501"
Nov 01 03:45:03.003838 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:03.003790    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2c8373-29891745-m2hl2" podStartSLOduration=3.003776823 podStartE2EDuration="3.003776823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:45:03.001217487 +0000 UTC m=+2307.524224098" watchObservedRunningTime="2026-11-01 03:45:03.003776823 +0000 UTC m=+2307.526783411"
Nov 01 03:45:03.032566 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:03.032526    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:03.353997 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:03.353944    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891745-g6s6m" podStartSLOduration=3.353929368 podStartE2EDuration="3.353929368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:45:03.350817984 +0000 UTC m=+2307.873824571" watchObservedRunningTime="2026-11-01 03:45:03.353929368 +0000 UTC m=+2307.876935955"
Nov 01 03:45:03.593781 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:03.593716    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-fdbdfefd-29891745-xpl9r" podStartSLOduration=3.593697405 podStartE2EDuration="3.593697405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:45:03.592942408 +0000 UTC m=+2308.115949026" watchObservedRunningTime="2026-11-01 03:45:03.593697405 +0000 UTC m=+2308.116703997"
Nov 01 03:45:03.682071 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:03.681990    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-19faa5-29891745-shv69" event={"ID":"77c03a7b-ad42-424e-851f-fa2bcd3e1b5e","Type":"ContainerStarted","Data":"a02a860690af046bb374d8151f0a1437711a62e552de6ac1f6adf2f7c1ab8013"}
Nov 01 03:45:03.684604 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:03.684569    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891745-k78n5" event={"ID":"146c7163-c4db-4c71-85a1-3536ebdeec82","Type":"ContainerStarted","Data":"daa4475dc27d7ca0b7a8641879683b5712d6038796bc57324a2935536ffa2763"}
Nov 01 03:45:03.686471 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:03.686442    4930 generic.go:356] "Generic (PLEG): container finished" podID="f8b3a0cd-6f72-4296-ae23-c03fc6e42613" containerID="0815a5c621f0ba3c2f5d2826ae6e181668ee1b61e96961d5cb55c3f3784832f4" exitCode=60
Nov 01 03:45:03.686639 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:03.686567    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-fdbdfefd-29891745-xpl9r" event={"ID":"f8b3a0cd-6f72-4296-ae23-c03fc6e42613","Type":"ContainerDied","Data":"0815a5c621f0ba3c2f5d2826ae6e181668ee1b61e96961d5cb55c3f3784832f4"}
Nov 01 03:45:03.686765 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:03.686743    4930 scope.go:122] "RemoveContainer" containerID="0815a5c621f0ba3c2f5d2826ae6e181668ee1b61e96961d5cb55c3f3784832f4"
Nov 01 03:45:03.688439 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:03.688414    4930 generic.go:356] "Generic (PLEG): container finished" podID="83bd0b30-2532-40b5-b5ab-e417f56cbe19" containerID="acd431b678b9c183bed2d1df0b03c1a1307418152413577c25178f47a7e94a58" exitCode=60
Nov 01 03:45:03.688689 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:03.688565    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891745-fdqkt" event={"ID":"83bd0b30-2532-40b5-b5ab-e417f56cbe19","Type":"ContainerDied","Data":"acd431b678b9c183bed2d1df0b03c1a1307418152413577c25178f47a7e94a58"}
Nov 01 03:45:03.688758 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:03.688718    4930 scope.go:122] "RemoveContainer" containerID="acd431b678b9c183bed2d1df0b03c1a1307418152413577c25178f47a7e94a58"
Nov 01 03:45:03.694856 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:03.694833    4930 generic.go:356] "Generic (PLEG): container finished" podID="190a288b-699e-4719-949e-ec24a8dd155b" containerID="5ee1eebcce77f49f3527da43ea0fff6ac857c971112600dc7a34d6c18770518b" exitCode=60
Nov 01 03:45:03.694970 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:03.694914    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-7cd000-29891745-r2xxc" event={"ID":"190a288b-699e-4719-949e-ec24a8dd155b","Type":"ContainerDied","Data":"5ee1eebcce77f49f3527da43ea0fff6ac857c971112600dc7a34d6c18770518b"}
Nov 01 03:45:03.695162 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:03.695143    4930 scope.go:122] "RemoveContainer" containerID="5ee1eebcce77f49f3527da43ea0fff6ac857c971112600dc7a34d6c18770518b"
Nov 01 03:45:03.697329 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:03.697297    4930 generic.go:356] "Generic (PLEG): container finished" podID="5df89ff2-793b-4eb5-9920-236b2674f026" containerID="935c4d98837fba4a2df2f4b49553d044dca3471acc1722c204f15ca1502f6184" exitCode=60
Nov 01 03:45:03.697431 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:03.697371    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2c8373-29891745-m2hl2" event={"ID":"5df89ff2-793b-4eb5-9920-236b2674f026","Type":"ContainerDied","Data":"935c4d98837fba4a2df2f4b49553d044dca3471acc1722c204f15ca1502f6184"}
Nov 01 03:45:03.697657 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:03.697636    4930 scope.go:122] "RemoveContainer" containerID="935c4d98837fba4a2df2f4b49553d044dca3471acc1722c204f15ca1502f6184"
Nov 01 03:45:03.699866 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:03.699845    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22e11b-29891745-mpj4g" event={"ID":"3876a597-4092-43ab-b4a5-c89cb1f1aa46","Type":"ContainerStarted","Data":"76fc341b95344cb2e2c6def5db5535e912d506489575e6f7f78e22eb617f6dba"}
Nov 01 03:45:03.707169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:03.707143    4930 generic.go:356] "Generic (PLEG): container finished" podID="08723624-3a83-4821-8d9a-716ca8847aff" containerID="8333ca5244e1c116e6014d7bbfaef25d2231138b3eb0b3ba9162c14fa112d01b" exitCode=60
Nov 01 03:45:03.707244 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:03.707226    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891745-g6s6m" event={"ID":"08723624-3a83-4821-8d9a-716ca8847aff","Type":"ContainerDied","Data":"8333ca5244e1c116e6014d7bbfaef25d2231138b3eb0b3ba9162c14fa112d01b"}
Nov 01 03:45:03.707596 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:03.707545    4930 scope.go:122] "RemoveContainer" containerID="8333ca5244e1c116e6014d7bbfaef25d2231138b3eb0b3ba9162c14fa112d01b"
Nov 01 03:45:03.711720 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:03.711695    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c22dd8-29891745-vltrn" event={"ID":"b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950","Type":"ContainerStarted","Data":"88cb15daccd3515c530b07bfd5e66bb2fbbfdfc7a22f112e8b56fd2989c73201"}
Nov 01 03:45:03.713519 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:03.713498    4930 generic.go:356] "Generic (PLEG): container finished" podID="5aa6bb82-d7e1-410c-9118-e7feed83bad8" containerID="4e62bfc621793cb3de41ecf96788ef5fe5ea14f1ba2bc7b53bee62ce0ab66a85" exitCode=60
Nov 01 03:45:03.713632 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:03.713525    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-da4f30-29891745-8trrg" event={"ID":"5aa6bb82-d7e1-410c-9118-e7feed83bad8","Type":"ContainerDied","Data":"4e62bfc621793cb3de41ecf96788ef5fe5ea14f1ba2bc7b53bee62ce0ab66a85"}
Nov 01 03:45:03.713844 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:03.713798    4930 scope.go:122] "RemoveContainer" containerID="4e62bfc621793cb3de41ecf96788ef5fe5ea14f1ba2bc7b53bee62ce0ab66a85"
Nov 01 03:45:03.830906 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:03.830860    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-da4f30-29891745-8trrg" podStartSLOduration=3.83084473 podStartE2EDuration="3.83084473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:45:03.710764651 +0000 UTC m=+2308.233771252" watchObservedRunningTime="2026-11-01 03:45:03.83084473 +0000 UTC m=+2308.353851319"
Nov 01 03:45:04.118648 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:45:04.118610    4930 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb31c5cbc_0fb5_4a9c_a55a_c232bbd2f950.slice/crio-88cb15daccd3515c530b07bfd5e66bb2fbbfdfc7a22f112e8b56fd2989c73201.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 03:45:04.257892 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:04.257856    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:04.319287 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:04.319158    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-19faa5-29891745-shv69" podStartSLOduration=4.319135763 podStartE2EDuration="4.319135763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:45:04.313290655 +0000 UTC m=+2308.836297244" watchObservedRunningTime="2026-11-01 03:45:04.319135763 +0000 UTC m=+2308.842142378"
Nov 01 03:45:04.550234 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:04.550184    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891745-k78n5" podStartSLOduration=4.550168965 podStartE2EDuration="4.550168965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:45:04.54976524 +0000 UTC m=+2309.072771830" watchObservedRunningTime="2026-11-01 03:45:04.550168965 +0000 UTC m=+2309.073175552"
Nov 01 03:45:04.725057 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:04.725015    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891745-g6s6m" event={"ID":"08723624-3a83-4821-8d9a-716ca8847aff","Type":"ContainerStarted","Data":"7ce92d036c77f6e691133c936dd4f285c0027afbeb850c2dee32d96ecacea181"}
Nov 01 03:45:04.727041 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:04.727018    4930 generic.go:356] "Generic (PLEG): container finished" podID="b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950" containerID="88cb15daccd3515c530b07bfd5e66bb2fbbfdfc7a22f112e8b56fd2989c73201" exitCode=60
Nov 01 03:45:04.727187 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:04.727102    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c22dd8-29891745-vltrn" event={"ID":"b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950","Type":"ContainerDied","Data":"88cb15daccd3515c530b07bfd5e66bb2fbbfdfc7a22f112e8b56fd2989c73201"}
Nov 01 03:45:04.727187 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:04.727167    4930 scope.go:122] "RemoveContainer" containerID="da2e2470f38ab1a8685d460a1b738a70e9dc180f401486d00af1c1350a69464f"
Nov 01 03:45:04.727396 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:04.727374    4930 scope.go:122] "RemoveContainer" containerID="88cb15daccd3515c530b07bfd5e66bb2fbbfdfc7a22f112e8b56fd2989c73201"
Nov 01 03:45:04.727750 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:45:04.727729    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-c22dd8-29891745-vltrn_odh-ai-gateway-infra(b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c22dd8-29891745-vltrn" podUID="b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950"
Nov 01 03:45:04.729220 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:04.729196    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-da4f30-29891745-8trrg" event={"ID":"5aa6bb82-d7e1-410c-9118-e7feed83bad8","Type":"ContainerStarted","Data":"38089f5d8ea4760e1a3f15f280561eb6f09c905b0011ff07fa1cfd14742e48f7"}
Nov 01 03:45:04.731221 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:04.731200    4930 generic.go:356] "Generic (PLEG): container finished" podID="77c03a7b-ad42-424e-851f-fa2bcd3e1b5e" containerID="a02a860690af046bb374d8151f0a1437711a62e552de6ac1f6adf2f7c1ab8013" exitCode=60
Nov 01 03:45:04.731308 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:04.731229    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-19faa5-29891745-shv69" event={"ID":"77c03a7b-ad42-424e-851f-fa2bcd3e1b5e","Type":"ContainerDied","Data":"a02a860690af046bb374d8151f0a1437711a62e552de6ac1f6adf2f7c1ab8013"}
Nov 01 03:45:04.731594 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:04.731543    4930 scope.go:122] "RemoveContainer" containerID="a02a860690af046bb374d8151f0a1437711a62e552de6ac1f6adf2f7c1ab8013"
Nov 01 03:45:04.731938 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:45:04.731915    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-19faa5-29891745-shv69_odh-ai-gateway-infra(77c03a7b-ad42-424e-851f-fa2bcd3e1b5e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-19faa5-29891745-shv69" podUID="77c03a7b-ad42-424e-851f-fa2bcd3e1b5e"
Nov 01 03:45:04.733866 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:04.733622    4930 generic.go:356] "Generic (PLEG): container finished" podID="146c7163-c4db-4c71-85a1-3536ebdeec82" containerID="daa4475dc27d7ca0b7a8641879683b5712d6038796bc57324a2935536ffa2763" exitCode=60
Nov 01 03:45:04.733943 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:04.733875    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891745-k78n5" event={"ID":"146c7163-c4db-4c71-85a1-3536ebdeec82","Type":"ContainerDied","Data":"daa4475dc27d7ca0b7a8641879683b5712d6038796bc57324a2935536ffa2763"}
Nov 01 03:45:04.733984 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:04.733971    4930 scope.go:122] "RemoveContainer" containerID="daa4475dc27d7ca0b7a8641879683b5712d6038796bc57324a2935536ffa2763"
Nov 01 03:45:04.734227 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:45:04.734207    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891745-k78n5_odh-ai-gateway-infra(146c7163-c4db-4c71-85a1-3536ebdeec82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891745-k78n5" podUID="146c7163-c4db-4c71-85a1-3536ebdeec82"
Nov 01 03:45:04.735852 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:04.735830    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-fdbdfefd-29891745-xpl9r" event={"ID":"f8b3a0cd-6f72-4296-ae23-c03fc6e42613","Type":"ContainerStarted","Data":"e1597cce9593fccef7e34cb44e50db14eceda5ac9bf8a903f532a9c70a93c62d"}
Nov 01 03:45:04.737796 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:04.737768    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891745-fdqkt" event={"ID":"83bd0b30-2532-40b5-b5ab-e417f56cbe19","Type":"ContainerStarted","Data":"413a4a8f577367e279adaece0a70664e99025ceb9df4d02944cbf336c6bbc55c"}
Nov 01 03:45:04.739757 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:04.739731    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-7cd000-29891745-r2xxc" event={"ID":"190a288b-699e-4719-949e-ec24a8dd155b","Type":"ContainerStarted","Data":"6d0c2b9659151224cbd010bb6a5a70cdf680b9d7c64e2b6301f17dee5cfe39ea"}
Nov 01 03:45:04.741855 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:04.741823    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2c8373-29891745-m2hl2" event={"ID":"5df89ff2-793b-4eb5-9920-236b2674f026","Type":"ContainerStarted","Data":"5a01eaf4790714e84b5a36ab44d4bf85aebb65fe6bead23a155397083f0e9812"}
Nov 01 03:45:04.742497 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:04.742478    4930 scope.go:122] "RemoveContainer" containerID="43e9a4678a79383dce2fb05c1aead8c723d628f344d3803a75d3c9b722a4e107"
Nov 01 03:45:04.743917 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:04.743893    4930 generic.go:356] "Generic (PLEG): container finished" podID="3876a597-4092-43ab-b4a5-c89cb1f1aa46" containerID="76fc341b95344cb2e2c6def5db5535e912d506489575e6f7f78e22eb617f6dba" exitCode=60
Nov 01 03:45:04.744014 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:04.743929    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22e11b-29891745-mpj4g" event={"ID":"3876a597-4092-43ab-b4a5-c89cb1f1aa46","Type":"ContainerDied","Data":"76fc341b95344cb2e2c6def5db5535e912d506489575e6f7f78e22eb617f6dba"}
Nov 01 03:45:04.744125 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:04.744108    4930 scope.go:122] "RemoveContainer" containerID="76fc341b95344cb2e2c6def5db5535e912d506489575e6f7f78e22eb617f6dba"
Nov 01 03:45:04.744332 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:45:04.744315    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-22e11b-29891745-mpj4g_odh-ai-gateway-infra(3876a597-4092-43ab-b4a5-c89cb1f1aa46)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22e11b-29891745-mpj4g" podUID="3876a597-4092-43ab-b4a5-c89cb1f1aa46"
Nov 01 03:45:04.756114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:04.756092    4930 scope.go:122] "RemoveContainer" containerID="4dc63996d1a6d3908e4a374f3deeedc15399284217c7a53658e6a377c5c8c409"
Nov 01 03:45:04.776972 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:04.776947    4930 scope.go:122] "RemoveContainer" containerID="667b21841c0be75934078216ad22f55f6fabb600ca79388a50e86b2080cb5f52"
Nov 01 03:45:04.909656 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:04.909609    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22e11b-29891745-mpj4g" podStartSLOduration=4.909594287 podStartE2EDuration="4.909594287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:45:04.906887921 +0000 UTC m=+2309.429894510" watchObservedRunningTime="2026-11-01 03:45:04.909594287 +0000 UTC m=+2309.432600875"
Nov 01 03:45:05.755656 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:05.755611    4930 generic.go:356] "Generic (PLEG): container finished" podID="190a288b-699e-4719-949e-ec24a8dd155b" containerID="6d0c2b9659151224cbd010bb6a5a70cdf680b9d7c64e2b6301f17dee5cfe39ea" exitCode=60
Nov 01 03:45:05.756065 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:05.755695    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-7cd000-29891745-r2xxc" event={"ID":"190a288b-699e-4719-949e-ec24a8dd155b","Type":"ContainerDied","Data":"6d0c2b9659151224cbd010bb6a5a70cdf680b9d7c64e2b6301f17dee5cfe39ea"}
Nov 01 03:45:05.756065 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:05.755770    4930 scope.go:122] "RemoveContainer" containerID="5ee1eebcce77f49f3527da43ea0fff6ac857c971112600dc7a34d6c18770518b"
Nov 01 03:45:05.756065 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:05.756003    4930 scope.go:122] "RemoveContainer" containerID="6d0c2b9659151224cbd010bb6a5a70cdf680b9d7c64e2b6301f17dee5cfe39ea"
Nov 01 03:45:05.756409 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:45:05.756388    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-7cd000-29891745-r2xxc_odh-ai-gateway-infra(190a288b-699e-4719-949e-ec24a8dd155b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-7cd000-29891745-r2xxc" podUID="190a288b-699e-4719-949e-ec24a8dd155b"
Nov 01 03:45:05.758000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:05.757978    4930 generic.go:356] "Generic (PLEG): container finished" podID="5df89ff2-793b-4eb5-9920-236b2674f026" containerID="5a01eaf4790714e84b5a36ab44d4bf85aebb65fe6bead23a155397083f0e9812" exitCode=60
Nov 01 03:45:05.758122 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:05.758062    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2c8373-29891745-m2hl2" event={"ID":"5df89ff2-793b-4eb5-9920-236b2674f026","Type":"ContainerDied","Data":"5a01eaf4790714e84b5a36ab44d4bf85aebb65fe6bead23a155397083f0e9812"}
Nov 01 03:45:05.758334 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:05.758320    4930 scope.go:122] "RemoveContainer" containerID="5a01eaf4790714e84b5a36ab44d4bf85aebb65fe6bead23a155397083f0e9812"
Nov 01 03:45:05.758712 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:45:05.758691    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-2c8373-29891745-m2hl2_odh-ai-gateway-infra(5df89ff2-793b-4eb5-9920-236b2674f026)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2c8373-29891745-m2hl2" podUID="5df89ff2-793b-4eb5-9920-236b2674f026"
Nov 01 03:45:05.768427 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:05.768406    4930 scope.go:122] "RemoveContainer" containerID="76fc341b95344cb2e2c6def5db5535e912d506489575e6f7f78e22eb617f6dba"
Nov 01 03:45:05.768784 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:45:05.768762    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-22e11b-29891745-mpj4g_odh-ai-gateway-infra(3876a597-4092-43ab-b4a5-c89cb1f1aa46)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22e11b-29891745-mpj4g" podUID="3876a597-4092-43ab-b4a5-c89cb1f1aa46"
Nov 01 03:45:05.770689 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:05.770663    4930 generic.go:356] "Generic (PLEG): container finished" podID="08723624-3a83-4821-8d9a-716ca8847aff" containerID="7ce92d036c77f6e691133c936dd4f285c0027afbeb850c2dee32d96ecacea181" exitCode=60
Nov 01 03:45:05.770811 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:05.770724    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891745-g6s6m" event={"ID":"08723624-3a83-4821-8d9a-716ca8847aff","Type":"ContainerDied","Data":"7ce92d036c77f6e691133c936dd4f285c0027afbeb850c2dee32d96ecacea181"}
Nov 01 03:45:05.771007 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:05.770984    4930 scope.go:122] "RemoveContainer" containerID="7ce92d036c77f6e691133c936dd4f285c0027afbeb850c2dee32d96ecacea181"
Nov 01 03:45:05.771318 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:45:05.771297    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891745-g6s6m_odh-ai-gateway-infra(08723624-3a83-4821-8d9a-716ca8847aff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891745-g6s6m" podUID="08723624-3a83-4821-8d9a-716ca8847aff"
Nov 01 03:45:05.773087 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:05.773068    4930 scope.go:122] "RemoveContainer" containerID="88cb15daccd3515c530b07bfd5e66bb2fbbfdfc7a22f112e8b56fd2989c73201"
Nov 01 03:45:05.773392 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:45:05.773368    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-c22dd8-29891745-vltrn_odh-ai-gateway-infra(b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c22dd8-29891745-vltrn" podUID="b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950"
Nov 01 03:45:05.775115 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:05.774984    4930 generic.go:356] "Generic (PLEG): container finished" podID="5aa6bb82-d7e1-410c-9118-e7feed83bad8" containerID="38089f5d8ea4760e1a3f15f280561eb6f09c905b0011ff07fa1cfd14742e48f7" exitCode=60
Nov 01 03:45:05.775115 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:05.775070    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-da4f30-29891745-8trrg" event={"ID":"5aa6bb82-d7e1-410c-9118-e7feed83bad8","Type":"ContainerDied","Data":"38089f5d8ea4760e1a3f15f280561eb6f09c905b0011ff07fa1cfd14742e48f7"}
Nov 01 03:45:05.775377 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:05.775359    4930 scope.go:122] "RemoveContainer" containerID="38089f5d8ea4760e1a3f15f280561eb6f09c905b0011ff07fa1cfd14742e48f7"
Nov 01 03:45:05.775514 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:05.775498    4930 scope.go:122] "RemoveContainer" containerID="935c4d98837fba4a2df2f4b49553d044dca3471acc1722c204f15ca1502f6184"
Nov 01 03:45:05.775697 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:45:05.775673    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-da4f30-29891745-8trrg_odh-ai-gateway-infra(5aa6bb82-d7e1-410c-9118-e7feed83bad8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-da4f30-29891745-8trrg" podUID="5aa6bb82-d7e1-410c-9118-e7feed83bad8"
Nov 01 03:45:05.777323 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:05.777304    4930 scope.go:122] "RemoveContainer" containerID="a02a860690af046bb374d8151f0a1437711a62e552de6ac1f6adf2f7c1ab8013"
Nov 01 03:45:05.777603 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:45:05.777534    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-19faa5-29891745-shv69_odh-ai-gateway-infra(77c03a7b-ad42-424e-851f-fa2bcd3e1b5e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-19faa5-29891745-shv69" podUID="77c03a7b-ad42-424e-851f-fa2bcd3e1b5e"
Nov 01 03:45:05.779121 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:05.779103    4930 scope.go:122] "RemoveContainer" containerID="daa4475dc27d7ca0b7a8641879683b5712d6038796bc57324a2935536ffa2763"
Nov 01 03:45:05.779336 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:45:05.779310    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891745-k78n5_odh-ai-gateway-infra(146c7163-c4db-4c71-85a1-3536ebdeec82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891745-k78n5" podUID="146c7163-c4db-4c71-85a1-3536ebdeec82"
Nov 01 03:45:05.780924 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:05.780904    4930 generic.go:356] "Generic (PLEG): container finished" podID="f8b3a0cd-6f72-4296-ae23-c03fc6e42613" containerID="e1597cce9593fccef7e34cb44e50db14eceda5ac9bf8a903f532a9c70a93c62d" exitCode=60
Nov 01 03:45:05.781032 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:05.780988    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-fdbdfefd-29891745-xpl9r" event={"ID":"f8b3a0cd-6f72-4296-ae23-c03fc6e42613","Type":"ContainerDied","Data":"e1597cce9593fccef7e34cb44e50db14eceda5ac9bf8a903f532a9c70a93c62d"}
Nov 01 03:45:05.781190 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:05.781174    4930 scope.go:122] "RemoveContainer" containerID="e1597cce9593fccef7e34cb44e50db14eceda5ac9bf8a903f532a9c70a93c62d"
Nov 01 03:45:05.781417 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:45:05.781392    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-fdbdfefd-29891745-xpl9r_odh-ai-gateway-infra(f8b3a0cd-6f72-4296-ae23-c03fc6e42613)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-fdbdfefd-29891745-xpl9r" podUID="f8b3a0cd-6f72-4296-ae23-c03fc6e42613"
Nov 01 03:45:05.783312 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:05.783288    4930 generic.go:356] "Generic (PLEG): container finished" podID="83bd0b30-2532-40b5-b5ab-e417f56cbe19" containerID="413a4a8f577367e279adaece0a70664e99025ceb9df4d02944cbf336c6bbc55c" exitCode=60
Nov 01 03:45:05.783411 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:05.783339    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891745-fdqkt" event={"ID":"83bd0b30-2532-40b5-b5ab-e417f56cbe19","Type":"ContainerDied","Data":"413a4a8f577367e279adaece0a70664e99025ceb9df4d02944cbf336c6bbc55c"}
Nov 01 03:45:05.783633 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:05.783615    4930 scope.go:122] "RemoveContainer" containerID="413a4a8f577367e279adaece0a70664e99025ceb9df4d02944cbf336c6bbc55c"
Nov 01 03:45:05.783890 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:45:05.783867    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d7f535-29891745-fdqkt_odh-ai-gateway-infra(83bd0b30-2532-40b5-b5ab-e417f56cbe19)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891745-fdqkt" podUID="83bd0b30-2532-40b5-b5ab-e417f56cbe19"
Nov 01 03:45:05.792638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:05.792614    4930 scope.go:122] "RemoveContainer" containerID="8333ca5244e1c116e6014d7bbfaef25d2231138b3eb0b3ba9162c14fa112d01b"
Nov 01 03:45:05.806598 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:05.806579    4930 scope.go:122] "RemoveContainer" containerID="4e62bfc621793cb3de41ecf96788ef5fe5ea14f1ba2bc7b53bee62ce0ab66a85"
Nov 01 03:45:05.820331 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:05.820312    4930 scope.go:122] "RemoveContainer" containerID="0815a5c621f0ba3c2f5d2826ae6e181668ee1b61e96961d5cb55c3f3784832f4"
Nov 01 03:45:05.833415 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:05.833391    4930 scope.go:122] "RemoveContainer" containerID="acd431b678b9c183bed2d1df0b03c1a1307418152413577c25178f47a7e94a58"
Nov 01 03:45:06.794825 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:06.794794    4930 scope.go:122] "RemoveContainer" containerID="6d0c2b9659151224cbd010bb6a5a70cdf680b9d7c64e2b6301f17dee5cfe39ea"
Nov 01 03:45:06.795307 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:45:06.795114    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-7cd000-29891745-r2xxc_odh-ai-gateway-infra(190a288b-699e-4719-949e-ec24a8dd155b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-7cd000-29891745-r2xxc" podUID="190a288b-699e-4719-949e-ec24a8dd155b"
Nov 01 03:45:06.796383 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:06.796359    4930 scope.go:122] "RemoveContainer" containerID="5a01eaf4790714e84b5a36ab44d4bf85aebb65fe6bead23a155397083f0e9812"
Nov 01 03:45:06.796687 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:45:06.796664    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-2c8373-29891745-m2hl2_odh-ai-gateway-infra(5df89ff2-793b-4eb5-9920-236b2674f026)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2c8373-29891745-m2hl2" podUID="5df89ff2-793b-4eb5-9920-236b2674f026"
Nov 01 03:45:06.797953 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:06.797932    4930 scope.go:122] "RemoveContainer" containerID="7ce92d036c77f6e691133c936dd4f285c0027afbeb850c2dee32d96ecacea181"
Nov 01 03:45:06.798193 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:45:06.798171    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891745-g6s6m_odh-ai-gateway-infra(08723624-3a83-4821-8d9a-716ca8847aff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891745-g6s6m" podUID="08723624-3a83-4821-8d9a-716ca8847aff"
Nov 01 03:45:06.799611 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:06.799591    4930 scope.go:122] "RemoveContainer" containerID="38089f5d8ea4760e1a3f15f280561eb6f09c905b0011ff07fa1cfd14742e48f7"
Nov 01 03:45:06.799884 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:45:06.799859    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-da4f30-29891745-8trrg_odh-ai-gateway-infra(5aa6bb82-d7e1-410c-9118-e7feed83bad8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-da4f30-29891745-8trrg" podUID="5aa6bb82-d7e1-410c-9118-e7feed83bad8"
Nov 01 03:45:06.801186 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:06.801168    4930 scope.go:122] "RemoveContainer" containerID="e1597cce9593fccef7e34cb44e50db14eceda5ac9bf8a903f532a9c70a93c62d"
Nov 01 03:45:06.801413 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:45:06.801394    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-fdbdfefd-29891745-xpl9r_odh-ai-gateway-infra(f8b3a0cd-6f72-4296-ae23-c03fc6e42613)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-fdbdfefd-29891745-xpl9r" podUID="f8b3a0cd-6f72-4296-ae23-c03fc6e42613"
Nov 01 03:45:06.802905 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:06.802887    4930 scope.go:122] "RemoveContainer" containerID="413a4a8f577367e279adaece0a70664e99025ceb9df4d02944cbf336c6bbc55c"
Nov 01 03:45:06.803131 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:45:06.803108    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d7f535-29891745-fdqkt_odh-ai-gateway-infra(83bd0b30-2532-40b5-b5ab-e417f56cbe19)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891745-fdqkt" podUID="83bd0b30-2532-40b5-b5ab-e417f56cbe19"
Nov 01 03:45:08.048945 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:08.048905    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d91612c9-75kzp"]
Nov 01 03:45:08.054218 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:08.054193    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d91612c9-75kzp"]
Nov 01 03:45:08.593827 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:08.593795    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:09.622994 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:09.622962    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62aeb801-0607-477c-81b6-7a9f4f0d668b" path="/var/lib/kubelet/pods/62aeb801-0607-477c-81b6-7a9f4f0d668b/volumes"
Nov 01 03:45:10.125536 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:10.125496    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:10.341072 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:10.341038    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:10.493706 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:10.493615    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:10.625166 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:10.625135    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e8c789-frgzj"]
Nov 01 03:45:10.674399 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:10.674371    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e8c789-frgzj"]
Nov 01 03:45:11.018668 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:11.018622    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:11.622502 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:11.622470    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61310c85-114a-46f0-9a4d-daa629c4f4d4" path="/var/lib/kubelet/pods/61310c85-114a-46f0-9a4d-daa629c4f4d4/volumes"
Nov 01 03:45:14.613508 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:14.613464    4930 scope.go:122] "RemoveContainer" containerID="76fc341b95344cb2e2c6def5db5535e912d506489575e6f7f78e22eb617f6dba"
Nov 01 03:45:15.481060 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:15.481011    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:15.892628 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:15.892579    4930 generic.go:356] "Generic (PLEG): container finished" podID="3876a597-4092-43ab-b4a5-c89cb1f1aa46" containerID="ca1b31ab9f3d81ff35a4d084d8069928e744f11cffbbd556a91fb908d356c16c" exitCode=60
Nov 01 03:45:15.893051 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:15.892669    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22e11b-29891745-mpj4g" event={"ID":"3876a597-4092-43ab-b4a5-c89cb1f1aa46","Type":"ContainerDied","Data":"ca1b31ab9f3d81ff35a4d084d8069928e744f11cffbbd556a91fb908d356c16c"}
Nov 01 03:45:15.893051 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:15.892734    4930 scope.go:122] "RemoveContainer" containerID="76fc341b95344cb2e2c6def5db5535e912d506489575e6f7f78e22eb617f6dba"
Nov 01 03:45:15.893162 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:15.893071    4930 scope.go:122] "RemoveContainer" containerID="ca1b31ab9f3d81ff35a4d084d8069928e744f11cffbbd556a91fb908d356c16c"
Nov 01 03:45:15.893365 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:45:15.893344    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-22e11b-29891745-mpj4g_odh-ai-gateway-infra(3876a597-4092-43ab-b4a5-c89cb1f1aa46)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22e11b-29891745-mpj4g" podUID="3876a597-4092-43ab-b4a5-c89cb1f1aa46"
Nov 01 03:45:16.613080 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:16.613046    4930 scope.go:122] "RemoveContainer" containerID="6d0c2b9659151224cbd010bb6a5a70cdf680b9d7c64e2b6301f17dee5cfe39ea"
Nov 01 03:45:16.613280 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:16.613122    4930 scope.go:122] "RemoveContainer" containerID="88cb15daccd3515c530b07bfd5e66bb2fbbfdfc7a22f112e8b56fd2989c73201"
Nov 01 03:45:17.053021 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:17.052989    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22e11b-29891745-mpj4g"]
Nov 01 03:45:17.322429 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:17.322396    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22e11b-29891745-mpj4g"
Nov 01 03:45:17.486048 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:17.485958    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3876a597-4092-43ab-b4a5-c89cb1f1aa46-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3876a597-4092-43ab-b4a5-c89cb1f1aa46-maas-api-service-ca\") pod \"3876a597-4092-43ab-b4a5-c89cb1f1aa46\" (UID: \"3876a597-4092-43ab-b4a5-c89cb1f1aa46\") "
Nov 01 03:45:17.486226 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:17.486180    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3876a597-4092-43ab-b4a5-c89cb1f1aa46-kube-api-access-7dqnq\" (UniqueName: \"kubernetes.io/projected/3876a597-4092-43ab-b4a5-c89cb1f1aa46-kube-api-access-7dqnq\") pod \"3876a597-4092-43ab-b4a5-c89cb1f1aa46\" (UID: \"3876a597-4092-43ab-b4a5-c89cb1f1aa46\") "
Nov 01 03:45:17.487506 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:17.487473    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3876a597-4092-43ab-b4a5-c89cb1f1aa46-maas-api-service-ca" pod "3876a597-4092-43ab-b4a5-c89cb1f1aa46" (UID: "3876a597-4092-43ab-b4a5-c89cb1f1aa46"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:45:17.489506 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:17.489480    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3876a597-4092-43ab-b4a5-c89cb1f1aa46-kube-api-access-7dqnq" pod "3876a597-4092-43ab-b4a5-c89cb1f1aa46" (UID: "3876a597-4092-43ab-b4a5-c89cb1f1aa46"). InnerVolumeSpecName "kube-api-access-7dqnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:45:17.588085 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:17.588056    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dqnq\" (UniqueName: \"kubernetes.io/projected/3876a597-4092-43ab-b4a5-c89cb1f1aa46-kube-api-access-7dqnq\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:17.588085 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:17.588081    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3876a597-4092-43ab-b4a5-c89cb1f1aa46-maas-api-service-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:17.613776 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:17.613747    4930 scope.go:122] "RemoveContainer" containerID="a02a860690af046bb374d8151f0a1437711a62e552de6ac1f6adf2f7c1ab8013"
Nov 01 03:45:17.613955 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:17.613937    4930 scope.go:122] "RemoveContainer" containerID="7ce92d036c77f6e691133c936dd4f285c0027afbeb850c2dee32d96ecacea181"
Nov 01 03:45:17.614102 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:17.614001    4930 scope.go:122] "RemoveContainer" containerID="413a4a8f577367e279adaece0a70664e99025ceb9df4d02944cbf336c6bbc55c"
Nov 01 03:45:17.614163 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:17.614141    4930 scope.go:122] "RemoveContainer" containerID="daa4475dc27d7ca0b7a8641879683b5712d6038796bc57324a2935536ffa2763"
Nov 01 03:45:17.923121 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:17.923090    4930 generic.go:356] "Generic (PLEG): container finished" podID="190a288b-699e-4719-949e-ec24a8dd155b" containerID="1d0f46f579ce68966d9c5834ae789e72cfa77b6c9e5b31863b5a2d95c5246b66" exitCode=60
Nov 01 03:45:17.923296 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:17.923128    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-7cd000-29891745-r2xxc" event={"ID":"190a288b-699e-4719-949e-ec24a8dd155b","Type":"ContainerDied","Data":"1d0f46f579ce68966d9c5834ae789e72cfa77b6c9e5b31863b5a2d95c5246b66"}
Nov 01 03:45:17.923296 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:17.923196    4930 scope.go:122] "RemoveContainer" containerID="6d0c2b9659151224cbd010bb6a5a70cdf680b9d7c64e2b6301f17dee5cfe39ea"
Nov 01 03:45:17.923639 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:17.923619    4930 scope.go:122] "RemoveContainer" containerID="1d0f46f579ce68966d9c5834ae789e72cfa77b6c9e5b31863b5a2d95c5246b66"
Nov 01 03:45:17.924002 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:45:17.923978    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-7cd000-29891745-r2xxc_odh-ai-gateway-infra(190a288b-699e-4719-949e-ec24a8dd155b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-7cd000-29891745-r2xxc" podUID="190a288b-699e-4719-949e-ec24a8dd155b"
Nov 01 03:45:17.925051 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:17.925029    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22e11b-29891745-mpj4g" event={"ID":"3876a597-4092-43ab-b4a5-c89cb1f1aa46","Type":"ContainerDied","Data":"53ad6d4d333820b051b654f3be236fca0337ea1c348203e258ed415c51d39323"}
Nov 01 03:45:17.925174 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:17.925117    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22e11b-29891745-mpj4g"
Nov 01 03:45:17.927819 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:17.927798    4930 generic.go:356] "Generic (PLEG): container finished" podID="b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950" containerID="0b56c560424e97c7351f9ce39586b7564a63bcc90b4bc5363888391ea67cfab5" exitCode=60
Nov 01 03:45:17.927930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:17.927858    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c22dd8-29891745-vltrn" event={"ID":"b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950","Type":"ContainerDied","Data":"0b56c560424e97c7351f9ce39586b7564a63bcc90b4bc5363888391ea67cfab5"}
Nov 01 03:45:17.928205 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:17.928183    4930 scope.go:122] "RemoveContainer" containerID="0b56c560424e97c7351f9ce39586b7564a63bcc90b4bc5363888391ea67cfab5"
Nov 01 03:45:17.928482 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:45:17.928462    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-c22dd8-29891745-vltrn_odh-ai-gateway-infra(b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c22dd8-29891745-vltrn" podUID="b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950"
Nov 01 03:45:17.993284 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:17.993133    4930 scope.go:122] "RemoveContainer" containerID="ca1b31ab9f3d81ff35a4d084d8069928e744f11cffbbd556a91fb908d356c16c"
Nov 01 03:45:18.014193 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:18.014172    4930 scope.go:122] "RemoveContainer" containerID="88cb15daccd3515c530b07bfd5e66bb2fbbfdfc7a22f112e8b56fd2989c73201"
Nov 01 03:45:18.267573 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:18.267477    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22e11b-29891745-mpj4g"]
Nov 01 03:45:18.273952 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:18.273919    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-22e11b-29891745-mpj4g"]
Nov 01 03:45:18.466216 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:18.466181    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:18.613032 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:18.612996    4930 scope.go:122] "RemoveContainer" containerID="e1597cce9593fccef7e34cb44e50db14eceda5ac9bf8a903f532a9c70a93c62d"
Nov 01 03:45:18.613255 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:18.613235    4930 scope.go:122] "RemoveContainer" containerID="38089f5d8ea4760e1a3f15f280561eb6f09c905b0011ff07fa1cfd14742e48f7"
Nov 01 03:45:18.613722 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:18.613606    4930 scope.go:122] "RemoveContainer" containerID="5a01eaf4790714e84b5a36ab44d4bf85aebb65fe6bead23a155397083f0e9812"
Nov 01 03:45:18.619479 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:18.619437    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:18.742222 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:18.742195    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"]
Nov 01 03:45:18.742431 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:18.742404    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst" podUID="c515c2c3-7e65-4512-9209-7994209c12e5" containerName="istio-proxy" containerID="cri-o://e884a1d81198148e1d9e64c0b5d0028c48c9ea3898519ef56be32ab7a302dac6" gracePeriod=30
Nov 01 03:45:18.771601 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:18.771521    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:18.940239 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:18.940202    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:18.992581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:18.990645    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891745-g6s6m" event={"ID":"08723624-3a83-4821-8d9a-716ca8847aff","Type":"ContainerStarted","Data":"a41c2acd046aad5cf77c7abd6eb208a33a95edacf71b3a17a2b44d6b53894198"}
Nov 01 03:45:19.005581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:18.996796    4930 generic.go:356] "Generic (PLEG): container finished" podID="77c03a7b-ad42-424e-851f-fa2bcd3e1b5e" containerID="b0e9b38a302644bb7f0a0b66b623c6482bda7aeb88748cea918c68533fb6a806" exitCode=60
Nov 01 03:45:19.005581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:18.996895    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-19faa5-29891745-shv69" event={"ID":"77c03a7b-ad42-424e-851f-fa2bcd3e1b5e","Type":"ContainerDied","Data":"b0e9b38a302644bb7f0a0b66b623c6482bda7aeb88748cea918c68533fb6a806"}
Nov 01 03:45:19.005581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:18.996949    4930 scope.go:122] "RemoveContainer" containerID="a02a860690af046bb374d8151f0a1437711a62e552de6ac1f6adf2f7c1ab8013"
Nov 01 03:45:19.005581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:18.997399    4930 scope.go:122] "RemoveContainer" containerID="b0e9b38a302644bb7f0a0b66b623c6482bda7aeb88748cea918c68533fb6a806"
Nov 01 03:45:19.005581 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:45:18.997787    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-19faa5-29891745-shv69_odh-ai-gateway-infra(77c03a7b-ad42-424e-851f-fa2bcd3e1b5e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-19faa5-29891745-shv69" podUID="77c03a7b-ad42-424e-851f-fa2bcd3e1b5e"
Nov 01 03:45:19.022740 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:19.022709    4930 generic.go:356] "Generic (PLEG): container finished" podID="146c7163-c4db-4c71-85a1-3536ebdeec82" containerID="1008dd0dd551c873e1394799fb31acbc817d8596edb005af82c9262ab872cd1c" exitCode=60
Nov 01 03:45:19.023047 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:19.023027    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891745-k78n5" event={"ID":"146c7163-c4db-4c71-85a1-3536ebdeec82","Type":"ContainerDied","Data":"1008dd0dd551c873e1394799fb31acbc817d8596edb005af82c9262ab872cd1c"}
Nov 01 03:45:19.023615 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:19.023545    4930 scope.go:122] "RemoveContainer" containerID="1008dd0dd551c873e1394799fb31acbc817d8596edb005af82c9262ab872cd1c"
Nov 01 03:45:19.024016 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:45:19.023991    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891745-k78n5_odh-ai-gateway-infra(146c7163-c4db-4c71-85a1-3536ebdeec82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891745-k78n5" podUID="146c7163-c4db-4c71-85a1-3536ebdeec82"
Nov 01 03:45:19.026735 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:19.026713    4930 generic.go:356] "Generic (PLEG): container finished" podID="83bd0b30-2532-40b5-b5ab-e417f56cbe19" containerID="fee61de6c20a785c6bb872251cd11268e9ad08073bc9d4fc64ae95539c1a6030" exitCode=60
Nov 01 03:45:19.027017 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:19.027000    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891745-fdqkt" event={"ID":"83bd0b30-2532-40b5-b5ab-e417f56cbe19","Type":"ContainerDied","Data":"fee61de6c20a785c6bb872251cd11268e9ad08073bc9d4fc64ae95539c1a6030"}
Nov 01 03:45:19.027510 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:19.027478    4930 scope.go:122] "RemoveContainer" containerID="fee61de6c20a785c6bb872251cd11268e9ad08073bc9d4fc64ae95539c1a6030"
Nov 01 03:45:19.027904 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:45:19.027880    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d7f535-29891745-fdqkt_odh-ai-gateway-infra(83bd0b30-2532-40b5-b5ab-e417f56cbe19)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891745-fdqkt" podUID="83bd0b30-2532-40b5-b5ab-e417f56cbe19"
Nov 01 03:45:19.054635 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:19.053452    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-7cd000-29891745-r2xxc"]
Nov 01 03:45:19.110051 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:19.110010    4930 scope.go:122] "RemoveContainer" containerID="daa4475dc27d7ca0b7a8641879683b5712d6038796bc57324a2935536ffa2763"
Nov 01 03:45:19.132506 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:19.132363    4930 scope.go:122] "RemoveContainer" containerID="413a4a8f577367e279adaece0a70664e99025ceb9df4d02944cbf336c6bbc55c"
Nov 01 03:45:19.135052 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:45:19.134834    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:45:19.168236 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:19.168210    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c22dd8-29891745-vltrn"]
Nov 01 03:45:19.279056 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:19.279032    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-7cd000-29891745-r2xxc"
Nov 01 03:45:19.323313 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:19.323284    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/190a288b-699e-4719-949e-ec24a8dd155b-kube-api-access-drj6x\" (UniqueName: \"kubernetes.io/projected/190a288b-699e-4719-949e-ec24a8dd155b-kube-api-access-drj6x\") pod \"190a288b-699e-4719-949e-ec24a8dd155b\" (UID: \"190a288b-699e-4719-949e-ec24a8dd155b\") "
Nov 01 03:45:19.323443 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:19.323435    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/190a288b-699e-4719-949e-ec24a8dd155b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/190a288b-699e-4719-949e-ec24a8dd155b-maas-api-service-ca\") pod \"190a288b-699e-4719-949e-ec24a8dd155b\" (UID: \"190a288b-699e-4719-949e-ec24a8dd155b\") "
Nov 01 03:45:19.323806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:19.323782    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/190a288b-699e-4719-949e-ec24a8dd155b-maas-api-service-ca" pod "190a288b-699e-4719-949e-ec24a8dd155b" (UID: "190a288b-699e-4719-949e-ec24a8dd155b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:45:19.324095 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:19.324080    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/190a288b-699e-4719-949e-ec24a8dd155b-maas-api-service-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:19.326001 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:19.325973    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/190a288b-699e-4719-949e-ec24a8dd155b-kube-api-access-drj6x" pod "190a288b-699e-4719-949e-ec24a8dd155b" (UID: "190a288b-699e-4719-949e-ec24a8dd155b"). InnerVolumeSpecName "kube-api-access-drj6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:45:19.364432 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:19.364349    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c22dd8-29891745-vltrn"
Nov 01 03:45:19.425198 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:19.425125    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950-maas-api-service-ca\") pod \"b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950\" (UID: \"b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950\") "
Nov 01 03:45:19.425326 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:19.425295    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950-kube-api-access-4gk9w\" (UniqueName: \"kubernetes.io/projected/b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950-kube-api-access-4gk9w\") pod \"b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950\" (UID: \"b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950\") "
Nov 01 03:45:19.425544 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:19.425522    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950-maas-api-service-ca" pod "b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950" (UID: "b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:45:19.425799 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:19.425783    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950-maas-api-service-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:19.425843 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:19.425802    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drj6x\" (UniqueName: \"kubernetes.io/projected/190a288b-699e-4719-949e-ec24a8dd155b-kube-api-access-drj6x\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:19.427329 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:19.427309    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950-kube-api-access-4gk9w" pod "b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950" (UID: "b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950"). InnerVolumeSpecName "kube-api-access-4gk9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:45:19.527572 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:19.527530    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gk9w\" (UniqueName: \"kubernetes.io/projected/b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950-kube-api-access-4gk9w\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:19.600466 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:19.600427    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:19.622241 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:19.622205    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3876a597-4092-43ab-b4a5-c89cb1f1aa46" path="/var/lib/kubelet/pods/3876a597-4092-43ab-b4a5-c89cb1f1aa46/volumes"
Nov 01 03:45:20.040545 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.040519    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-7cd000-29891745-r2xxc"
Nov 01 03:45:20.040701 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.040548    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-7cd000-29891745-r2xxc" event={"ID":"190a288b-699e-4719-949e-ec24a8dd155b","Type":"ContainerDied","Data":"55082a85f42a5b8ae08f4fdc826e2f1b66dcb956bc4faeaebbb1949b086ede4d"}
Nov 01 03:45:20.040701 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.040624    4930 scope.go:122] "RemoveContainer" containerID="1d0f46f579ce68966d9c5834ae789e72cfa77b6c9e5b31863b5a2d95c5246b66"
Nov 01 03:45:20.043248 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.043228    4930 generic.go:356] "Generic (PLEG): container finished" podID="5df89ff2-793b-4eb5-9920-236b2674f026" containerID="9211be7588f93452d13c9bb851f9c066b27982bdd0a8b304c378d724459d01c1" exitCode=60
Nov 01 03:45:20.043378 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.043286    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2c8373-29891745-m2hl2" event={"ID":"5df89ff2-793b-4eb5-9920-236b2674f026","Type":"ContainerDied","Data":"9211be7588f93452d13c9bb851f9c066b27982bdd0a8b304c378d724459d01c1"}
Nov 01 03:45:20.043816 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.043797    4930 scope.go:122] "RemoveContainer" containerID="9211be7588f93452d13c9bb851f9c066b27982bdd0a8b304c378d724459d01c1"
Nov 01 03:45:20.044149 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:45:20.044121    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-2c8373-29891745-m2hl2_odh-ai-gateway-infra(5df89ff2-793b-4eb5-9920-236b2674f026)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2c8373-29891745-m2hl2" podUID="5df89ff2-793b-4eb5-9920-236b2674f026"
Nov 01 03:45:20.045659 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.045638    4930 generic.go:356] "Generic (PLEG): container finished" podID="08723624-3a83-4821-8d9a-716ca8847aff" containerID="a41c2acd046aad5cf77c7abd6eb208a33a95edacf71b3a17a2b44d6b53894198" exitCode=60
Nov 01 03:45:20.045766 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.045665    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891745-g6s6m" event={"ID":"08723624-3a83-4821-8d9a-716ca8847aff","Type":"ContainerDied","Data":"a41c2acd046aad5cf77c7abd6eb208a33a95edacf71b3a17a2b44d6b53894198"}
Nov 01 03:45:20.045998 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.045975    4930 scope.go:122] "RemoveContainer" containerID="a41c2acd046aad5cf77c7abd6eb208a33a95edacf71b3a17a2b44d6b53894198"
Nov 01 03:45:20.046281 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:45:20.046258    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891745-g6s6m_odh-ai-gateway-infra(08723624-3a83-4821-8d9a-716ca8847aff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891745-g6s6m" podUID="08723624-3a83-4821-8d9a-716ca8847aff"
Nov 01 03:45:20.047166 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.047150    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c22dd8-29891745-vltrn"
Nov 01 03:45:20.047266 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.047173    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c22dd8-29891745-vltrn" event={"ID":"b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950","Type":"ContainerDied","Data":"10cb9d5125d6b439d284e903486843aae45a781b7d2e1b2635c174c001041f0a"}
Nov 01 03:45:20.049232 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.049211    4930 generic.go:356] "Generic (PLEG): container finished" podID="5aa6bb82-d7e1-410c-9118-e7feed83bad8" containerID="a72d0a83b9e65b15d2005dacc02fd7c3f60a4cc638a93573e83fc9ee78caf093" exitCode=60
Nov 01 03:45:20.049340 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.049240    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-da4f30-29891745-8trrg" event={"ID":"5aa6bb82-d7e1-410c-9118-e7feed83bad8","Type":"ContainerDied","Data":"a72d0a83b9e65b15d2005dacc02fd7c3f60a4cc638a93573e83fc9ee78caf093"}
Nov 01 03:45:20.049694 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.049673    4930 scope.go:122] "RemoveContainer" containerID="a72d0a83b9e65b15d2005dacc02fd7c3f60a4cc638a93573e83fc9ee78caf093"
Nov 01 03:45:20.050119 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:45:20.049937    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-da4f30-29891745-8trrg_odh-ai-gateway-infra(5aa6bb82-d7e1-410c-9118-e7feed83bad8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-da4f30-29891745-8trrg" podUID="5aa6bb82-d7e1-410c-9118-e7feed83bad8"
Nov 01 03:45:20.053129 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.053105    4930 scope.go:122] "RemoveContainer" containerID="5a01eaf4790714e84b5a36ab44d4bf85aebb65fe6bead23a155397083f0e9812"
Nov 01 03:45:20.054702 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.054683    4930 generic.go:356] "Generic (PLEG): container finished" podID="f8b3a0cd-6f72-4296-ae23-c03fc6e42613" containerID="483f7fda84e3b479e036a9012cd1ef473faa398a9f0634d12393e73af58912e8" exitCode=60
Nov 01 03:45:20.054780 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.054739    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-fdbdfefd-29891745-xpl9r" event={"ID":"f8b3a0cd-6f72-4296-ae23-c03fc6e42613","Type":"ContainerDied","Data":"483f7fda84e3b479e036a9012cd1ef473faa398a9f0634d12393e73af58912e8"}
Nov 01 03:45:20.055067 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.055049    4930 scope.go:122] "RemoveContainer" containerID="483f7fda84e3b479e036a9012cd1ef473faa398a9f0634d12393e73af58912e8"
Nov 01 03:45:20.055303 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:45:20.055283    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-fdbdfefd-29891745-xpl9r_odh-ai-gateway-infra(f8b3a0cd-6f72-4296-ae23-c03fc6e42613)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-fdbdfefd-29891745-xpl9r" podUID="f8b3a0cd-6f72-4296-ae23-c03fc6e42613"
Nov 01 03:45:20.068252 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.068229    4930 scope.go:122] "RemoveContainer" containerID="7ce92d036c77f6e691133c936dd4f285c0027afbeb850c2dee32d96ecacea181"
Nov 01 03:45:20.082043 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.082025    4930 scope.go:122] "RemoveContainer" containerID="0b56c560424e97c7351f9ce39586b7564a63bcc90b4bc5363888391ea67cfab5"
Nov 01 03:45:20.093521 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.093499    4930 scope.go:122] "RemoveContainer" containerID="38089f5d8ea4760e1a3f15f280561eb6f09c905b0011ff07fa1cfd14742e48f7"
Nov 01 03:45:20.110805 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.110729    4930 scope.go:122] "RemoveContainer" containerID="e1597cce9593fccef7e34cb44e50db14eceda5ac9bf8a903f532a9c70a93c62d"
Nov 01 03:45:20.202293 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.202186    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891745-g6s6m"]
Nov 01 03:45:20.243944 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.243911    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-19faa5-29891745-shv69"]
Nov 01 03:45:20.358883 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.358059    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891745-k78n5"]
Nov 01 03:45:20.397159 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.397026    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-7cd000-29891745-r2xxc"]
Nov 01 03:45:20.399053 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.399020    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-7cd000-29891745-r2xxc"]
Nov 01 03:45:20.445338 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.445311    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-19faa5-29891745-shv69"
Nov 01 03:45:20.478191 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.474201    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891745-fdqkt"]
Nov 01 03:45:20.515590 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.505860    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c22dd8-29891745-vltrn"]
Nov 01 03:45:20.515590 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.511967    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-c22dd8-29891745-vltrn"]
Nov 01 03:45:20.540973 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.540947    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891745-k78n5"
Nov 01 03:45:20.542836 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.542810    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77c03a7b-ad42-424e-851f-fa2bcd3e1b5e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77c03a7b-ad42-424e-851f-fa2bcd3e1b5e-maas-api-service-ca\") pod \"77c03a7b-ad42-424e-851f-fa2bcd3e1b5e\" (UID: \"77c03a7b-ad42-424e-851f-fa2bcd3e1b5e\") "
Nov 01 03:45:20.542991 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.542974    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77c03a7b-ad42-424e-851f-fa2bcd3e1b5e-kube-api-access-zx2nn\" (UniqueName: \"kubernetes.io/projected/77c03a7b-ad42-424e-851f-fa2bcd3e1b5e-kube-api-access-zx2nn\") pod \"77c03a7b-ad42-424e-851f-fa2bcd3e1b5e\" (UID: \"77c03a7b-ad42-424e-851f-fa2bcd3e1b5e\") "
Nov 01 03:45:20.543367 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.543347    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77c03a7b-ad42-424e-851f-fa2bcd3e1b5e-maas-api-service-ca" pod "77c03a7b-ad42-424e-851f-fa2bcd3e1b5e" (UID: "77c03a7b-ad42-424e-851f-fa2bcd3e1b5e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:45:20.543531 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.543517    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77c03a7b-ad42-424e-851f-fa2bcd3e1b5e-maas-api-service-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:20.544983 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.544956    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77c03a7b-ad42-424e-851f-fa2bcd3e1b5e-kube-api-access-zx2nn" pod "77c03a7b-ad42-424e-851f-fa2bcd3e1b5e" (UID: "77c03a7b-ad42-424e-851f-fa2bcd3e1b5e"). InnerVolumeSpecName "kube-api-access-zx2nn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:45:20.637068 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.637038    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891745-fdqkt"
Nov 01 03:45:20.644169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.644142    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/146c7163-c4db-4c71-85a1-3536ebdeec82-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/146c7163-c4db-4c71-85a1-3536ebdeec82-maas-api-service-ca\") pod \"146c7163-c4db-4c71-85a1-3536ebdeec82\" (UID: \"146c7163-c4db-4c71-85a1-3536ebdeec82\") "
Nov 01 03:45:20.644316 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.644181    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/146c7163-c4db-4c71-85a1-3536ebdeec82-kube-api-access-97sb2\" (UniqueName: \"kubernetes.io/projected/146c7163-c4db-4c71-85a1-3536ebdeec82-kube-api-access-97sb2\") pod \"146c7163-c4db-4c71-85a1-3536ebdeec82\" (UID: \"146c7163-c4db-4c71-85a1-3536ebdeec82\") "
Nov 01 03:45:20.644534 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.644511    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/146c7163-c4db-4c71-85a1-3536ebdeec82-maas-api-service-ca" pod "146c7163-c4db-4c71-85a1-3536ebdeec82" (UID: "146c7163-c4db-4c71-85a1-3536ebdeec82"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:45:20.645230 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.645204    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/146c7163-c4db-4c71-85a1-3536ebdeec82-maas-api-service-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:20.645335 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.645237    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx2nn\" (UniqueName: \"kubernetes.io/projected/77c03a7b-ad42-424e-851f-fa2bcd3e1b5e-kube-api-access-zx2nn\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:20.646390 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.646368    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/146c7163-c4db-4c71-85a1-3536ebdeec82-kube-api-access-97sb2" pod "146c7163-c4db-4c71-85a1-3536ebdeec82" (UID: "146c7163-c4db-4c71-85a1-3536ebdeec82"). InnerVolumeSpecName "kube-api-access-97sb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:45:20.745994 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.745963    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83bd0b30-2532-40b5-b5ab-e417f56cbe19-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83bd0b30-2532-40b5-b5ab-e417f56cbe19-maas-api-service-ca\") pod \"83bd0b30-2532-40b5-b5ab-e417f56cbe19\" (UID: \"83bd0b30-2532-40b5-b5ab-e417f56cbe19\") "
Nov 01 03:45:20.746169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.746055    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83bd0b30-2532-40b5-b5ab-e417f56cbe19-kube-api-access-jrc9l\" (UniqueName: \"kubernetes.io/projected/83bd0b30-2532-40b5-b5ab-e417f56cbe19-kube-api-access-jrc9l\") pod \"83bd0b30-2532-40b5-b5ab-e417f56cbe19\" (UID: \"83bd0b30-2532-40b5-b5ab-e417f56cbe19\") "
Nov 01 03:45:20.746351 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.746328    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83bd0b30-2532-40b5-b5ab-e417f56cbe19-maas-api-service-ca" pod "83bd0b30-2532-40b5-b5ab-e417f56cbe19" (UID: "83bd0b30-2532-40b5-b5ab-e417f56cbe19"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:45:20.746482 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.746466    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97sb2\" (UniqueName: \"kubernetes.io/projected/146c7163-c4db-4c71-85a1-3536ebdeec82-kube-api-access-97sb2\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:20.746539 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.746488    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83bd0b30-2532-40b5-b5ab-e417f56cbe19-maas-api-service-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:20.748199 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.748178    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83bd0b30-2532-40b5-b5ab-e417f56cbe19-kube-api-access-jrc9l" pod "83bd0b30-2532-40b5-b5ab-e417f56cbe19" (UID: "83bd0b30-2532-40b5-b5ab-e417f56cbe19"). InnerVolumeSpecName "kube-api-access-jrc9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:45:20.848197 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.848157    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrc9l\" (UniqueName: \"kubernetes.io/projected/83bd0b30-2532-40b5-b5ab-e417f56cbe19-kube-api-access-jrc9l\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:20.885168 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:20.885133    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:21.070330 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:21.070302    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-19faa5-29891745-shv69"
Nov 01 03:45:21.070499 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:21.070300    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-19faa5-29891745-shv69" event={"ID":"77c03a7b-ad42-424e-851f-fa2bcd3e1b5e","Type":"ContainerDied","Data":"eb7affa6a4770b087be5497e1f269a494e5cfe6c6f75cde4bbdff7150ac1fcb0"}
Nov 01 03:45:21.070499 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:21.070444    4930 scope.go:122] "RemoveContainer" containerID="b0e9b38a302644bb7f0a0b66b623c6482bda7aeb88748cea918c68533fb6a806"
Nov 01 03:45:21.072021 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:21.071999    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891745-k78n5"
Nov 01 03:45:21.072366 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:21.072017    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891745-k78n5" event={"ID":"146c7163-c4db-4c71-85a1-3536ebdeec82","Type":"ContainerDied","Data":"1bad95cacae8c1ba50a279aee607454bbb1854059975d5aadcfb8d380a2e6794"}
Nov 01 03:45:21.085344 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:21.084221    4930 scope.go:122] "RemoveContainer" containerID="1008dd0dd551c873e1394799fb31acbc817d8596edb005af82c9262ab872cd1c"
Nov 01 03:45:21.086248 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:21.086221    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891745-fdqkt" event={"ID":"83bd0b30-2532-40b5-b5ab-e417f56cbe19","Type":"ContainerDied","Data":"a1d63623e1e272b349b99c0ac5f3a56e66abaa58da2525967a075c9a8270eb92"}
Nov 01 03:45:21.086355 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:21.086322    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891745-fdqkt"
Nov 01 03:45:21.109622 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:21.109592    4930 scope.go:122] "RemoveContainer" containerID="fee61de6c20a785c6bb872251cd11268e9ad08073bc9d4fc64ae95539c1a6030"
Nov 01 03:45:21.285364 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:21.285327    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2c8373-29891745-m2hl2"]
Nov 01 03:45:21.300151 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:21.300124    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891745-g6s6m"
Nov 01 03:45:21.328629 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:21.328482    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-19faa5-29891745-shv69"]
Nov 01 03:45:21.332232 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:21.332203    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-19faa5-29891745-shv69"]
Nov 01 03:45:21.356132 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:21.356103    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08723624-3a83-4821-8d9a-716ca8847aff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08723624-3a83-4821-8d9a-716ca8847aff-maas-api-service-ca\") pod \"08723624-3a83-4821-8d9a-716ca8847aff\" (UID: \"08723624-3a83-4821-8d9a-716ca8847aff\") "
Nov 01 03:45:21.356267 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:21.356235    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08723624-3a83-4821-8d9a-716ca8847aff-kube-api-access-2rpgv\" (UniqueName: \"kubernetes.io/projected/08723624-3a83-4821-8d9a-716ca8847aff-kube-api-access-2rpgv\") pod \"08723624-3a83-4821-8d9a-716ca8847aff\" (UID: \"08723624-3a83-4821-8d9a-716ca8847aff\") "
Nov 01 03:45:21.356656 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:21.356497    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08723624-3a83-4821-8d9a-716ca8847aff-maas-api-service-ca" pod "08723624-3a83-4821-8d9a-716ca8847aff" (UID: "08723624-3a83-4821-8d9a-716ca8847aff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:45:21.356943 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:21.356919    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08723624-3a83-4821-8d9a-716ca8847aff-maas-api-service-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:21.358626 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:21.358590    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08723624-3a83-4821-8d9a-716ca8847aff-kube-api-access-2rpgv" pod "08723624-3a83-4821-8d9a-716ca8847aff" (UID: "08723624-3a83-4821-8d9a-716ca8847aff"). InnerVolumeSpecName "kube-api-access-2rpgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:45:21.402395 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:21.402363    4930 patch_prober.go:29] interesting pod/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:45:21.402870 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:21.402424    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst" podUID="c515c2c3-7e65-4512-9209-7994209c12e5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:45:21.452036 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:21.452014    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2c8373-29891745-m2hl2"
Nov 01 03:45:21.458189 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:21.458166    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rpgv\" (UniqueName: \"kubernetes.io/projected/08723624-3a83-4821-8d9a-716ca8847aff-kube-api-access-2rpgv\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:21.557463 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:21.557423    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891745-k78n5"]
Nov 01 03:45:21.559084 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:21.559044    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5df89ff2-793b-4eb5-9920-236b2674f026-kube-api-access-f2bq4\" (UniqueName: \"kubernetes.io/projected/5df89ff2-793b-4eb5-9920-236b2674f026-kube-api-access-f2bq4\") pod \"5df89ff2-793b-4eb5-9920-236b2674f026\" (UID: \"5df89ff2-793b-4eb5-9920-236b2674f026\") "
Nov 01 03:45:21.559263 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:21.559176    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5df89ff2-793b-4eb5-9920-236b2674f026-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df89ff2-793b-4eb5-9920-236b2674f026-maas-api-service-ca\") pod \"5df89ff2-793b-4eb5-9920-236b2674f026\" (UID: \"5df89ff2-793b-4eb5-9920-236b2674f026\") "
Nov 01 03:45:21.561369 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:21.560092    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5df89ff2-793b-4eb5-9920-236b2674f026-maas-api-service-ca" pod "5df89ff2-793b-4eb5-9920-236b2674f026" (UID: "5df89ff2-793b-4eb5-9920-236b2674f026"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:45:21.561369 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:21.560509    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df89ff2-793b-4eb5-9920-236b2674f026-maas-api-service-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:21.562404 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:21.562376    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5df89ff2-793b-4eb5-9920-236b2674f026-kube-api-access-f2bq4" pod "5df89ff2-793b-4eb5-9920-236b2674f026" (UID: "5df89ff2-793b-4eb5-9920-236b2674f026"). InnerVolumeSpecName "kube-api-access-f2bq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:45:21.565131 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:21.565106    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c8cb9a-29891745-k78n5"]
Nov 01 03:45:21.621120 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:21.621097    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="146c7163-c4db-4c71-85a1-3536ebdeec82" path="/var/lib/kubelet/pods/146c7163-c4db-4c71-85a1-3536ebdeec82/volumes"
Nov 01 03:45:21.621603 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:21.621585    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="190a288b-699e-4719-949e-ec24a8dd155b" path="/var/lib/kubelet/pods/190a288b-699e-4719-949e-ec24a8dd155b/volumes"
Nov 01 03:45:21.622026 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:21.622013    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77c03a7b-ad42-424e-851f-fa2bcd3e1b5e" path="/var/lib/kubelet/pods/77c03a7b-ad42-424e-851f-fa2bcd3e1b5e/volumes"
Nov 01 03:45:21.622762 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:21.622747    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950" path="/var/lib/kubelet/pods/b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950/volumes"
Nov 01 03:45:21.662765 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:21.662493    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2bq4\" (UniqueName: \"kubernetes.io/projected/5df89ff2-793b-4eb5-9920-236b2674f026-kube-api-access-f2bq4\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:21.676393 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:21.676345    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891745-fdqkt"]
Nov 01 03:45:21.681875 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:21.681848    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7f535-29891745-fdqkt"]
Nov 01 03:45:21.747704 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:21.747678    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-fdbdfefd-29891745-xpl9r"]
Nov 01 03:45:21.862470 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:21.862432    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-da4f30-29891745-8trrg"]
Nov 01 03:45:21.970129 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:21.970101    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-fdbdfefd-29891745-xpl9r"
Nov 01 03:45:22.068612 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:22.068544    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8b3a0cd-6f72-4296-ae23-c03fc6e42613-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8b3a0cd-6f72-4296-ae23-c03fc6e42613-maas-api-service-ca\") pod \"f8b3a0cd-6f72-4296-ae23-c03fc6e42613\" (UID: \"f8b3a0cd-6f72-4296-ae23-c03fc6e42613\") "
Nov 01 03:45:22.068821 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:22.068801    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8b3a0cd-6f72-4296-ae23-c03fc6e42613-kube-api-access-6x9pt\" (UniqueName: \"kubernetes.io/projected/f8b3a0cd-6f72-4296-ae23-c03fc6e42613-kube-api-access-6x9pt\") pod \"f8b3a0cd-6f72-4296-ae23-c03fc6e42613\" (UID: \"f8b3a0cd-6f72-4296-ae23-c03fc6e42613\") "
Nov 01 03:45:22.068898 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:22.068880    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8b3a0cd-6f72-4296-ae23-c03fc6e42613-maas-api-service-ca" pod "f8b3a0cd-6f72-4296-ae23-c03fc6e42613" (UID: "f8b3a0cd-6f72-4296-ae23-c03fc6e42613"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:45:22.069276 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:22.069252    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8b3a0cd-6f72-4296-ae23-c03fc6e42613-maas-api-service-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:22.071084 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:22.071060    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8b3a0cd-6f72-4296-ae23-c03fc6e42613-kube-api-access-6x9pt" pod "f8b3a0cd-6f72-4296-ae23-c03fc6e42613" (UID: "f8b3a0cd-6f72-4296-ae23-c03fc6e42613"). InnerVolumeSpecName "kube-api-access-6x9pt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:45:22.085130 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:22.085105    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-da4f30-29891745-8trrg"
Nov 01 03:45:22.117511 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:22.117490    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2c8373-29891745-m2hl2"
Nov 01 03:45:22.117659 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:22.117527    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2c8373-29891745-m2hl2" event={"ID":"5df89ff2-793b-4eb5-9920-236b2674f026","Type":"ContainerDied","Data":"458bae37fa90c2e300b9b70f3c7687b3e33d59f08039f3a9c077c99a87d1a760"}
Nov 01 03:45:22.117659 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:22.117622    4930 scope.go:122] "RemoveContainer" containerID="9211be7588f93452d13c9bb851f9c066b27982bdd0a8b304c378d724459d01c1"
Nov 01 03:45:22.119260 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:22.119233    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891745-g6s6m"
Nov 01 03:45:22.119365 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:22.119261    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891745-g6s6m" event={"ID":"08723624-3a83-4821-8d9a-716ca8847aff","Type":"ContainerDied","Data":"bc35b93bcf6a65b0451472f4d4470e759dad5b5c3d57d758003d8242ad7b0956"}
Nov 01 03:45:22.120912 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:22.120894    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-da4f30-29891745-8trrg"
Nov 01 03:45:22.121022 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:22.120927    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-da4f30-29891745-8trrg" event={"ID":"5aa6bb82-d7e1-410c-9118-e7feed83bad8","Type":"ContainerDied","Data":"e50a4b040a4324cb45d4b970511e7ac6ad3573660a67300cbe958fa43647dde3"}
Nov 01 03:45:22.124707 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:22.124688    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-fdbdfefd-29891745-xpl9r" event={"ID":"f8b3a0cd-6f72-4296-ae23-c03fc6e42613","Type":"ContainerDied","Data":"a7f7183e72cd28f1f93e7a1a25da735e353114331787dd6e8f909c57ac0437a3"}
Nov 01 03:45:22.124804 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:22.124687    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-fdbdfefd-29891745-xpl9r"
Nov 01 03:45:22.130860 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:22.130836    4930 scope.go:122] "RemoveContainer" containerID="a41c2acd046aad5cf77c7abd6eb208a33a95edacf71b3a17a2b44d6b53894198"
Nov 01 03:45:22.140523 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:22.140477    4930 scope.go:122] "RemoveContainer" containerID="a72d0a83b9e65b15d2005dacc02fd7c3f60a4cc638a93573e83fc9ee78caf093"
Nov 01 03:45:22.149618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:22.149600    4930 scope.go:122] "RemoveContainer" containerID="483f7fda84e3b479e036a9012cd1ef473faa398a9f0634d12393e73af58912e8"
Nov 01 03:45:22.170685 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:22.170659    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5aa6bb82-d7e1-410c-9118-e7feed83bad8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aa6bb82-d7e1-410c-9118-e7feed83bad8-maas-api-service-ca\") pod \"5aa6bb82-d7e1-410c-9118-e7feed83bad8\" (UID: \"5aa6bb82-d7e1-410c-9118-e7feed83bad8\") "
Nov 01 03:45:22.170805 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:22.170717    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aa6bb82-d7e1-410c-9118-e7feed83bad8-kube-api-access-tf8xm\" (UniqueName: \"kubernetes.io/projected/5aa6bb82-d7e1-410c-9118-e7feed83bad8-kube-api-access-tf8xm\") pod \"5aa6bb82-d7e1-410c-9118-e7feed83bad8\" (UID: \"5aa6bb82-d7e1-410c-9118-e7feed83bad8\") "
Nov 01 03:45:22.171012 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:22.170988    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5aa6bb82-d7e1-410c-9118-e7feed83bad8-maas-api-service-ca" pod "5aa6bb82-d7e1-410c-9118-e7feed83bad8" (UID: "5aa6bb82-d7e1-410c-9118-e7feed83bad8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:45:22.171221 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:22.171205    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x9pt\" (UniqueName: \"kubernetes.io/projected/f8b3a0cd-6f72-4296-ae23-c03fc6e42613-kube-api-access-6x9pt\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:22.171288 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:22.171223    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aa6bb82-d7e1-410c-9118-e7feed83bad8-maas-api-service-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:22.172775 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:22.172743    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aa6bb82-d7e1-410c-9118-e7feed83bad8-kube-api-access-tf8xm" pod "5aa6bb82-d7e1-410c-9118-e7feed83bad8" (UID: "5aa6bb82-d7e1-410c-9118-e7feed83bad8"). InnerVolumeSpecName "kube-api-access-tf8xm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:45:22.259997 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:22.259918    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891745-g6s6m"]
Nov 01 03:45:22.270304 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:22.270270    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891745-g6s6m"]
Nov 01 03:45:22.272782 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:22.272748    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf8xm\" (UniqueName: \"kubernetes.io/projected/5aa6bb82-d7e1-410c-9118-e7feed83bad8-kube-api-access-tf8xm\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:22.311797 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:22.311761    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:22.461315 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:22.461286    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-fdbdfefd-29891745-xpl9r"]
Nov 01 03:45:22.474666 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:22.474634    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-fdbdfefd-29891745-xpl9r"]
Nov 01 03:45:22.597403 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:22.597363    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:22.709210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:22.709173    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2c8373-29891745-m2hl2"]
Nov 01 03:45:22.715203 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:22.715158    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-2c8373-29891745-m2hl2"]
Nov 01 03:45:22.824718 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:22.824681    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-da4f30-29891745-8trrg"]
Nov 01 03:45:22.828414 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:22.828387    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-da4f30-29891745-8trrg"]
Nov 01 03:45:23.622347 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:23.622309    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08723624-3a83-4821-8d9a-716ca8847aff" path="/var/lib/kubelet/pods/08723624-3a83-4821-8d9a-716ca8847aff/volumes"
Nov 01 03:45:23.622850 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:23.622832    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5aa6bb82-d7e1-410c-9118-e7feed83bad8" path="/var/lib/kubelet/pods/5aa6bb82-d7e1-410c-9118-e7feed83bad8/volumes"
Nov 01 03:45:23.623238 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:23.623224    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5df89ff2-793b-4eb5-9920-236b2674f026" path="/var/lib/kubelet/pods/5df89ff2-793b-4eb5-9920-236b2674f026/volumes"
Nov 01 03:45:23.623943 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:23.623928    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83bd0b30-2532-40b5-b5ab-e417f56cbe19" path="/var/lib/kubelet/pods/83bd0b30-2532-40b5-b5ab-e417f56cbe19/volumes"
Nov 01 03:45:23.624331 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:23.624317    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8b3a0cd-6f72-4296-ae23-c03fc6e42613" path="/var/lib/kubelet/pods/f8b3a0cd-6f72-4296-ae23-c03fc6e42613/volumes"
Nov 01 03:45:24.068129 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:24.068104    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"
Nov 01 03:45:24.102864 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:24.102832    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c515c2c3-7e65-4512-9209-7994209c12e5-kube-api-access-ncqld\" (UniqueName: \"kubernetes.io/projected/c515c2c3-7e65-4512-9209-7994209c12e5-kube-api-access-ncqld\") pod \"c515c2c3-7e65-4512-9209-7994209c12e5\" (UID: \"c515c2c3-7e65-4512-9209-7994209c12e5\") "
Nov 01 03:45:24.103024 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:24.102892    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c515c2c3-7e65-4512-9209-7994209c12e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c515c2c3-7e65-4512-9209-7994209c12e5-workload-socket\") pod \"c515c2c3-7e65-4512-9209-7994209c12e5\" (UID: \"c515c2c3-7e65-4512-9209-7994209c12e5\") "
Nov 01 03:45:24.103024 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:24.102920    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c515c2c3-7e65-4512-9209-7994209c12e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c515c2c3-7e65-4512-9209-7994209c12e5-istio-data\") pod \"c515c2c3-7e65-4512-9209-7994209c12e5\" (UID: \"c515c2c3-7e65-4512-9209-7994209c12e5\") "
Nov 01 03:45:24.103024 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:24.102999    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c515c2c3-7e65-4512-9209-7994209c12e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c515c2c3-7e65-4512-9209-7994209c12e5-istio-podinfo\") pod \"c515c2c3-7e65-4512-9209-7994209c12e5\" (UID: \"c515c2c3-7e65-4512-9209-7994209c12e5\") "
Nov 01 03:45:24.103207 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:24.103082    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c515c2c3-7e65-4512-9209-7994209c12e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c515c2c3-7e65-4512-9209-7994209c12e5-credential-socket\") pod \"c515c2c3-7e65-4512-9209-7994209c12e5\" (UID: \"c515c2c3-7e65-4512-9209-7994209c12e5\") "
Nov 01 03:45:24.103207 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:24.103107    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c515c2c3-7e65-4512-9209-7994209c12e5-istio-token\" (UniqueName: \"kubernetes.io/projected/c515c2c3-7e65-4512-9209-7994209c12e5-istio-token\") pod \"c515c2c3-7e65-4512-9209-7994209c12e5\" (UID: \"c515c2c3-7e65-4512-9209-7994209c12e5\") "
Nov 01 03:45:24.103335 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:24.103232    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c515c2c3-7e65-4512-9209-7994209c12e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c515c2c3-7e65-4512-9209-7994209c12e5-workload-certs\") pod \"c515c2c3-7e65-4512-9209-7994209c12e5\" (UID: \"c515c2c3-7e65-4512-9209-7994209c12e5\") "
Nov 01 03:45:24.103335 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:24.103273    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c515c2c3-7e65-4512-9209-7994209c12e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c515c2c3-7e65-4512-9209-7994209c12e5-istio-envoy\") pod \"c515c2c3-7e65-4512-9209-7994209c12e5\" (UID: \"c515c2c3-7e65-4512-9209-7994209c12e5\") "
Nov 01 03:45:24.103441 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:24.103347    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c515c2c3-7e65-4512-9209-7994209c12e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c515c2c3-7e65-4512-9209-7994209c12e5-istiod-ca-cert\") pod \"c515c2c3-7e65-4512-9209-7994209c12e5\" (UID: \"c515c2c3-7e65-4512-9209-7994209c12e5\") "
Nov 01 03:45:24.103501 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:24.103265    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c515c2c3-7e65-4512-9209-7994209c12e5-workload-socket" pod "c515c2c3-7e65-4512-9209-7994209c12e5" (UID: "c515c2c3-7e65-4512-9209-7994209c12e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:45:24.103501 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:24.103306    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c515c2c3-7e65-4512-9209-7994209c12e5-istio-data" pod "c515c2c3-7e65-4512-9209-7994209c12e5" (UID: "c515c2c3-7e65-4512-9209-7994209c12e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:45:24.103501 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:24.103465    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c515c2c3-7e65-4512-9209-7994209c12e5-workload-certs" pod "c515c2c3-7e65-4512-9209-7994209c12e5" (UID: "c515c2c3-7e65-4512-9209-7994209c12e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:45:24.104274 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:24.103881    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c515c2c3-7e65-4512-9209-7994209c12e5-istiod-ca-cert" pod "c515c2c3-7e65-4512-9209-7994209c12e5" (UID: "c515c2c3-7e65-4512-9209-7994209c12e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:45:24.104274 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:24.104113    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c515c2c3-7e65-4512-9209-7994209c12e5-credential-socket" pod "c515c2c3-7e65-4512-9209-7994209c12e5" (UID: "c515c2c3-7e65-4512-9209-7994209c12e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:45:24.104274 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:24.104186    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c515c2c3-7e65-4512-9209-7994209c12e5-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:24.104274 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:24.104212    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c515c2c3-7e65-4512-9209-7994209c12e5-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:24.104274 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:24.104228    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c515c2c3-7e65-4512-9209-7994209c12e5-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:24.104274 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:24.104242    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c515c2c3-7e65-4512-9209-7994209c12e5-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:24.105682 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:24.105652    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c515c2c3-7e65-4512-9209-7994209c12e5-kube-api-access-ncqld" pod "c515c2c3-7e65-4512-9209-7994209c12e5" (UID: "c515c2c3-7e65-4512-9209-7994209c12e5"). InnerVolumeSpecName "kube-api-access-ncqld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:45:24.105877 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:24.105846    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c515c2c3-7e65-4512-9209-7994209c12e5-istio-envoy" pod "c515c2c3-7e65-4512-9209-7994209c12e5" (UID: "c515c2c3-7e65-4512-9209-7994209c12e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:45:24.105958 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:24.105878    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c515c2c3-7e65-4512-9209-7994209c12e5-istio-podinfo" pod "c515c2c3-7e65-4512-9209-7994209c12e5" (UID: "c515c2c3-7e65-4512-9209-7994209c12e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:45:24.106089 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:24.106072    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c515c2c3-7e65-4512-9209-7994209c12e5-istio-token" pod "c515c2c3-7e65-4512-9209-7994209c12e5" (UID: "c515c2c3-7e65-4512-9209-7994209c12e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:45:24.147616 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:24.147585    4930 generic.go:356] "Generic (PLEG): container finished" podID="c515c2c3-7e65-4512-9209-7994209c12e5" containerID="e884a1d81198148e1d9e64c0b5d0028c48c9ea3898519ef56be32ab7a302dac6" exitCode=0
Nov 01 03:45:24.147743 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:24.147625    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"
Nov 01 03:45:24.147743 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:24.147671    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst" event={"ID":"c515c2c3-7e65-4512-9209-7994209c12e5","Type":"ContainerDied","Data":"e884a1d81198148e1d9e64c0b5d0028c48c9ea3898519ef56be32ab7a302dac6"}
Nov 01 03:45:24.147743 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:24.147731    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst" event={"ID":"c515c2c3-7e65-4512-9209-7994209c12e5","Type":"ContainerDied","Data":"e4fd51851cdab6903c4f54bb181348debbebc1552ae14974c384c7bf36b07230"}
Nov 01 03:45:24.147851 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:24.147758    4930 scope.go:122] "RemoveContainer" containerID="e884a1d81198148e1d9e64c0b5d0028c48c9ea3898519ef56be32ab7a302dac6"
Nov 01 03:45:24.159304 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:24.159281    4930 scope.go:122] "RemoveContainer" containerID="24e2bb6cb44527449d7d19395c72ed888c39b92171bbf9fc5f5fe1ea2ddab61c"
Nov 01 03:45:24.171815 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:24.171790    4930 scope.go:122] "RemoveContainer" containerID="e884a1d81198148e1d9e64c0b5d0028c48c9ea3898519ef56be32ab7a302dac6"
Nov 01 03:45:24.172125 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:45:24.172104    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e884a1d81198148e1d9e64c0b5d0028c48c9ea3898519ef56be32ab7a302dac6\": container with ID starting with e884a1d81198148e1d9e64c0b5d0028c48c9ea3898519ef56be32ab7a302dac6 not found: ID does not exist" containerID="e884a1d81198148e1d9e64c0b5d0028c48c9ea3898519ef56be32ab7a302dac6"
Nov 01 03:45:24.172188 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:24.172135    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e884a1d81198148e1d9e64c0b5d0028c48c9ea3898519ef56be32ab7a302dac6"} err="failed to get container status \"e884a1d81198148e1d9e64c0b5d0028c48c9ea3898519ef56be32ab7a302dac6\": rpc error: code = NotFound desc = could not find container \"e884a1d81198148e1d9e64c0b5d0028c48c9ea3898519ef56be32ab7a302dac6\": container with ID starting with e884a1d81198148e1d9e64c0b5d0028c48c9ea3898519ef56be32ab7a302dac6 not found: ID does not exist"
Nov 01 03:45:24.172188 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:24.172154    4930 scope.go:122] "RemoveContainer" containerID="24e2bb6cb44527449d7d19395c72ed888c39b92171bbf9fc5f5fe1ea2ddab61c"
Nov 01 03:45:24.172442 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:45:24.172425    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24e2bb6cb44527449d7d19395c72ed888c39b92171bbf9fc5f5fe1ea2ddab61c\": container with ID starting with 24e2bb6cb44527449d7d19395c72ed888c39b92171bbf9fc5f5fe1ea2ddab61c not found: ID does not exist" containerID="24e2bb6cb44527449d7d19395c72ed888c39b92171bbf9fc5f5fe1ea2ddab61c"
Nov 01 03:45:24.172512 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:24.172448    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24e2bb6cb44527449d7d19395c72ed888c39b92171bbf9fc5f5fe1ea2ddab61c"} err="failed to get container status \"24e2bb6cb44527449d7d19395c72ed888c39b92171bbf9fc5f5fe1ea2ddab61c\": rpc error: code = NotFound desc = could not find container \"24e2bb6cb44527449d7d19395c72ed888c39b92171bbf9fc5f5fe1ea2ddab61c\": container with ID starting with 24e2bb6cb44527449d7d19395c72ed888c39b92171bbf9fc5f5fe1ea2ddab61c not found: ID does not exist"
Nov 01 03:45:24.205134 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:24.205049    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c515c2c3-7e65-4512-9209-7994209c12e5-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:24.205134 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:24.205078    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c515c2c3-7e65-4512-9209-7994209c12e5-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:24.205134 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:24.205106    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c515c2c3-7e65-4512-9209-7994209c12e5-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:24.205134 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:24.205119    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c515c2c3-7e65-4512-9209-7994209c12e5-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:24.205134 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:24.205131    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncqld\" (UniqueName: \"kubernetes.io/projected/c515c2c3-7e65-4512-9209-7994209c12e5-kube-api-access-ncqld\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:24.396796 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:24.396760    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"]
Nov 01 03:45:24.400288 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:24.400256    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e8c789-openshift-default-774d997df-xwbst"]
Nov 01 03:45:25.623143 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:25.623110    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c515c2c3-7e65-4512-9209-7994209c12e5" path="/var/lib/kubelet/pods/c515c2c3-7e65-4512-9209-7994209c12e5/volumes"
Nov 01 03:45:25.928187 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:25.928103    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:26.666913 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:26.666878    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:26.835838 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:26.835798    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:26.981467 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:26.981369    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:29.880533 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:29.880499    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:30.063982 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.063946    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:30.309205 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.309122    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.476682    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a279c2-df99cdff7-rm7hx"]
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.477627    4930 cpu_manager.go:414] "containerMap: removing container" podUID="b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.477649    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.477669    4930 cpu_manager.go:414] "containerMap: removing container" podUID="b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.477676    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.477695    4930 cpu_manager.go:414] "containerMap: removing container" podUID="190a288b-699e-4719-949e-ec24a8dd155b" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.477704    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="190a288b-699e-4719-949e-ec24a8dd155b" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.477716    4930 cpu_manager.go:414] "containerMap: removing container" podUID="190a288b-699e-4719-949e-ec24a8dd155b" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.477725    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="190a288b-699e-4719-949e-ec24a8dd155b" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.477734    4930 cpu_manager.go:414] "containerMap: removing container" podUID="f8b3a0cd-6f72-4296-ae23-c03fc6e42613" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.477741    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8b3a0cd-6f72-4296-ae23-c03fc6e42613" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.477753    4930 cpu_manager.go:414] "containerMap: removing container" podUID="77c03a7b-ad42-424e-851f-fa2bcd3e1b5e" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.477761    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77c03a7b-ad42-424e-851f-fa2bcd3e1b5e" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.477784    4930 cpu_manager.go:414] "containerMap: removing container" podUID="5aa6bb82-d7e1-410c-9118-e7feed83bad8" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.477791    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aa6bb82-d7e1-410c-9118-e7feed83bad8" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.477807    4930 cpu_manager.go:414] "containerMap: removing container" podUID="146c7163-c4db-4c71-85a1-3536ebdeec82" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.477815    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="146c7163-c4db-4c71-85a1-3536ebdeec82" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.477826    4930 cpu_manager.go:414] "containerMap: removing container" podUID="f8b3a0cd-6f72-4296-ae23-c03fc6e42613" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.477834    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8b3a0cd-6f72-4296-ae23-c03fc6e42613" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.477846    4930 cpu_manager.go:414] "containerMap: removing container" podUID="146c7163-c4db-4c71-85a1-3536ebdeec82" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.477853    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="146c7163-c4db-4c71-85a1-3536ebdeec82" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.477869    4930 cpu_manager.go:414] "containerMap: removing container" podUID="c515c2c3-7e65-4512-9209-7994209c12e5" containerName="istio-proxy"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.477876    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c515c2c3-7e65-4512-9209-7994209c12e5" containerName="istio-proxy"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.477891    4930 cpu_manager.go:414] "containerMap: removing container" podUID="c515c2c3-7e65-4512-9209-7994209c12e5" containerName="istio-proxy"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.477898    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c515c2c3-7e65-4512-9209-7994209c12e5" containerName="istio-proxy"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.477915    4930 cpu_manager.go:414] "containerMap: removing container" podUID="83bd0b30-2532-40b5-b5ab-e417f56cbe19" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.477924    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83bd0b30-2532-40b5-b5ab-e417f56cbe19" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.477936    4930 cpu_manager.go:414] "containerMap: removing container" podUID="5df89ff2-793b-4eb5-9920-236b2674f026" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.477944    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5df89ff2-793b-4eb5-9920-236b2674f026" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.477961    4930 cpu_manager.go:414] "containerMap: removing container" podUID="08723624-3a83-4821-8d9a-716ca8847aff" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.477969    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08723624-3a83-4821-8d9a-716ca8847aff" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.477983    4930 cpu_manager.go:414] "containerMap: removing container" podUID="83bd0b30-2532-40b5-b5ab-e417f56cbe19" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.477990    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83bd0b30-2532-40b5-b5ab-e417f56cbe19" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.478004    4930 cpu_manager.go:414] "containerMap: removing container" podUID="3876a597-4092-43ab-b4a5-c89cb1f1aa46" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.478011    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3876a597-4092-43ab-b4a5-c89cb1f1aa46" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.478022    4930 cpu_manager.go:414] "containerMap: removing container" podUID="f8b3a0cd-6f72-4296-ae23-c03fc6e42613" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.478029    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8b3a0cd-6f72-4296-ae23-c03fc6e42613" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.478041    4930 cpu_manager.go:414] "containerMap: removing container" podUID="3876a597-4092-43ab-b4a5-c89cb1f1aa46" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.478048    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3876a597-4092-43ab-b4a5-c89cb1f1aa46" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.478067    4930 cpu_manager.go:414] "containerMap: removing container" podUID="08723624-3a83-4821-8d9a-716ca8847aff" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.478075    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08723624-3a83-4821-8d9a-716ca8847aff" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.478099    4930 cpu_manager.go:414] "containerMap: removing container" podUID="08723624-3a83-4821-8d9a-716ca8847aff" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.478107    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08723624-3a83-4821-8d9a-716ca8847aff" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.478122    4930 cpu_manager.go:414] "containerMap: removing container" podUID="3876a597-4092-43ab-b4a5-c89cb1f1aa46" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.478129    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3876a597-4092-43ab-b4a5-c89cb1f1aa46" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.478142    4930 cpu_manager.go:414] "containerMap: removing container" podUID="5aa6bb82-d7e1-410c-9118-e7feed83bad8" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.478149    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aa6bb82-d7e1-410c-9118-e7feed83bad8" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.478173    4930 cpu_manager.go:414] "containerMap: removing container" podUID="77c03a7b-ad42-424e-851f-fa2bcd3e1b5e" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.478180    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77c03a7b-ad42-424e-851f-fa2bcd3e1b5e" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.478192    4930 cpu_manager.go:414] "containerMap: removing container" podUID="5df89ff2-793b-4eb5-9920-236b2674f026" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.478200    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5df89ff2-793b-4eb5-9920-236b2674f026" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.478355    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="146c7163-c4db-4c71-85a1-3536ebdeec82" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.478370    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="77c03a7b-ad42-424e-851f-fa2bcd3e1b5e" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.478378    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="5df89ff2-793b-4eb5-9920-236b2674f026" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.478386    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.478396    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.478407    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="190a288b-699e-4719-949e-ec24a8dd155b" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.478415    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="190a288b-699e-4719-949e-ec24a8dd155b" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.478424    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8b3a0cd-6f72-4296-ae23-c03fc6e42613" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.478434    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aa6bb82-d7e1-410c-9118-e7feed83bad8" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.478445    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8b3a0cd-6f72-4296-ae23-c03fc6e42613" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.478456    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="146c7163-c4db-4c71-85a1-3536ebdeec82" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.478465    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="c515c2c3-7e65-4512-9209-7994209c12e5" containerName="istio-proxy"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.478474    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="c515c2c3-7e65-4512-9209-7994209c12e5" containerName="istio-proxy"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.478484    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="77c03a7b-ad42-424e-851f-fa2bcd3e1b5e" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.478492    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="83bd0b30-2532-40b5-b5ab-e417f56cbe19" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.478501    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="5df89ff2-793b-4eb5-9920-236b2674f026" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.478510    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aa6bb82-d7e1-410c-9118-e7feed83bad8" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.478520    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="08723624-3a83-4821-8d9a-716ca8847aff" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.478529    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="83bd0b30-2532-40b5-b5ab-e417f56cbe19" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.478538    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="3876a597-4092-43ab-b4a5-c89cb1f1aa46" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.478546    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8b3a0cd-6f72-4296-ae23-c03fc6e42613" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.483293    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="3876a597-4092-43ab-b4a5-c89cb1f1aa46" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.483316    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="08723624-3a83-4821-8d9a-716ca8847aff" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.483328    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="08723624-3a83-4821-8d9a-716ca8847aff" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.483339    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="83bd0b30-2532-40b5-b5ab-e417f56cbe19" containerName="cleanup"
Nov 01 03:45:30.485580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.483346    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="3876a597-4092-43ab-b4a5-c89cb1f1aa46" containerName="cleanup"
Nov 01 03:45:30.489474 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.489447    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a279c2-df99cdff7-rm7hx"
Nov 01 03:45:30.511655 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.506435    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:30.558380 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.555080    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a279c2-df99cdff7-rm7hx"]
Nov 01 03:45:30.595830 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.595799    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-a279c2\""
Nov 01 03:45:30.596035 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.595835    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-a279c2-dockercfg-5dlkz\""
Nov 01 03:45:30.613626 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.613582    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5srjn\" (UniqueName: \"kubernetes.io/projected/532b2472-d6b3-4af0-905b-98953c8b814c-kube-api-access-5srjn\") pod \"payload-processing-e2e-api-b-a279c2-df99cdff7-rm7hx\" (UID: \"532b2472-d6b3-4af0-905b-98953c8b814c\") " pod="openshift-ingress/payload-processing-e2e-api-b-a279c2-df99cdff7-rm7hx"
Nov 01 03:45:30.613796 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.613694    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/532b2472-d6b3-4af0-905b-98953c8b814c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a279c2-df99cdff7-rm7hx\" (UID: \"532b2472-d6b3-4af0-905b-98953c8b814c\") " pod="openshift-ingress/payload-processing-e2e-api-b-a279c2-df99cdff7-rm7hx"
Nov 01 03:45:30.650320 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.650288    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a279c2-7ffb8d6599-z857m"]
Nov 01 03:45:30.650919 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.650902    4930 cpu_manager.go:414] "containerMap: removing container" podUID="77c03a7b-ad42-424e-851f-fa2bcd3e1b5e" containerName="cleanup"
Nov 01 03:45:30.650974 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.650920    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77c03a7b-ad42-424e-851f-fa2bcd3e1b5e" containerName="cleanup"
Nov 01 03:45:30.650974 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.650939    4930 cpu_manager.go:414] "containerMap: removing container" podUID="146c7163-c4db-4c71-85a1-3536ebdeec82" containerName="cleanup"
Nov 01 03:45:30.650974 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.650945    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="146c7163-c4db-4c71-85a1-3536ebdeec82" containerName="cleanup"
Nov 01 03:45:30.651075 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.650987    4930 cpu_manager.go:414] "containerMap: removing container" podUID="83bd0b30-2532-40b5-b5ab-e417f56cbe19" containerName="cleanup"
Nov 01 03:45:30.651075 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.650993    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83bd0b30-2532-40b5-b5ab-e417f56cbe19" containerName="cleanup"
Nov 01 03:45:30.651075 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.651014    4930 cpu_manager.go:414] "containerMap: removing container" podUID="5aa6bb82-d7e1-410c-9118-e7feed83bad8" containerName="cleanup"
Nov 01 03:45:30.651075 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.651019    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aa6bb82-d7e1-410c-9118-e7feed83bad8" containerName="cleanup"
Nov 01 03:45:30.651075 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.651031    4930 cpu_manager.go:414] "containerMap: removing container" podUID="5df89ff2-793b-4eb5-9920-236b2674f026" containerName="cleanup"
Nov 01 03:45:30.651075 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.651037    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5df89ff2-793b-4eb5-9920-236b2674f026" containerName="cleanup"
Nov 01 03:45:30.651075 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.651044    4930 cpu_manager.go:414] "containerMap: removing container" podUID="b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950" containerName="cleanup"
Nov 01 03:45:30.651075 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.651048    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950" containerName="cleanup"
Nov 01 03:45:30.651075 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.651063    4930 cpu_manager.go:414] "containerMap: removing container" podUID="190a288b-699e-4719-949e-ec24a8dd155b" containerName="cleanup"
Nov 01 03:45:30.651075 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.651068    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="190a288b-699e-4719-949e-ec24a8dd155b" containerName="cleanup"
Nov 01 03:45:30.651384 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.651161    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="146c7163-c4db-4c71-85a1-3536ebdeec82" containerName="cleanup"
Nov 01 03:45:30.651384 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.651171    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="b31c5cbc-0fb5-4a9c-a55a-c232bbd2f950" containerName="cleanup"
Nov 01 03:45:30.651384 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.651179    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="190a288b-699e-4719-949e-ec24a8dd155b" containerName="cleanup"
Nov 01 03:45:30.651384 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.651189    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="77c03a7b-ad42-424e-851f-fa2bcd3e1b5e" containerName="cleanup"
Nov 01 03:45:30.651384 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.651196    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="5df89ff2-793b-4eb5-9920-236b2674f026" containerName="cleanup"
Nov 01 03:45:30.651384 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.651204    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aa6bb82-d7e1-410c-9118-e7feed83bad8" containerName="cleanup"
Nov 01 03:45:30.654797 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.654779    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a279c2-7ffb8d6599-z857m"
Nov 01 03:45:30.688300 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.688277    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a279c2-7ffb8d6599-z857m"]
Nov 01 03:45:30.714911 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.714886    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bzff\" (UniqueName: \"kubernetes.io/projected/bd58da84-e12c-4bf4-a64c-d3b7155f57fa-kube-api-access-8bzff\") pod \"payload-pre-processing-e2e-api-b-a279c2-7ffb8d6599-z857m\" (UID: \"bd58da84-e12c-4bf4-a64c-d3b7155f57fa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a279c2-7ffb8d6599-z857m"
Nov 01 03:45:30.715082 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.715053    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd58da84-e12c-4bf4-a64c-d3b7155f57fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a279c2-7ffb8d6599-z857m\" (UID: \"bd58da84-e12c-4bf4-a64c-d3b7155f57fa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a279c2-7ffb8d6599-z857m"
Nov 01 03:45:30.715151 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.715134    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5srjn\" (UniqueName: \"kubernetes.io/projected/532b2472-d6b3-4af0-905b-98953c8b814c-kube-api-access-5srjn\") pod \"payload-processing-e2e-api-b-a279c2-df99cdff7-rm7hx\" (UID: \"532b2472-d6b3-4af0-905b-98953c8b814c\") " pod="openshift-ingress/payload-processing-e2e-api-b-a279c2-df99cdff7-rm7hx"
Nov 01 03:45:30.715213 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.715195    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/532b2472-d6b3-4af0-905b-98953c8b814c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a279c2-df99cdff7-rm7hx\" (UID: \"532b2472-d6b3-4af0-905b-98953c8b814c\") " pod="openshift-ingress/payload-processing-e2e-api-b-a279c2-df99cdff7-rm7hx"
Nov 01 03:45:30.716110 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.716087    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/532b2472-d6b3-4af0-905b-98953c8b814c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a279c2-df99cdff7-rm7hx\" (UID: \"532b2472-d6b3-4af0-905b-98953c8b814c\") " pod="openshift-ingress/payload-processing-e2e-api-b-a279c2-df99cdff7-rm7hx"
Nov 01 03:45:30.776206 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.776180    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5srjn\" (UniqueName: \"kubernetes.io/projected/532b2472-d6b3-4af0-905b-98953c8b814c-kube-api-access-5srjn\") pod \"payload-processing-e2e-api-b-a279c2-df99cdff7-rm7hx\" (UID: \"532b2472-d6b3-4af0-905b-98953c8b814c\") " pod="openshift-ingress/payload-processing-e2e-api-b-a279c2-df99cdff7-rm7hx"
Nov 01 03:45:30.816896 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.816862    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bzff\" (UniqueName: \"kubernetes.io/projected/bd58da84-e12c-4bf4-a64c-d3b7155f57fa-kube-api-access-8bzff\") pod \"payload-pre-processing-e2e-api-b-a279c2-7ffb8d6599-z857m\" (UID: \"bd58da84-e12c-4bf4-a64c-d3b7155f57fa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a279c2-7ffb8d6599-z857m"
Nov 01 03:45:30.817038 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.816984    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd58da84-e12c-4bf4-a64c-d3b7155f57fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a279c2-7ffb8d6599-z857m\" (UID: \"bd58da84-e12c-4bf4-a64c-d3b7155f57fa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a279c2-7ffb8d6599-z857m"
Nov 01 03:45:30.817620 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.817601    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd58da84-e12c-4bf4-a64c-d3b7155f57fa-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a279c2-7ffb8d6599-z857m\" (UID: \"bd58da84-e12c-4bf4-a64c-d3b7155f57fa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a279c2-7ffb8d6599-z857m"
Nov 01 03:45:30.892650 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.892574    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a279c2-df99cdff7-rm7hx"
Nov 01 03:45:30.896382 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.896315    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bzff\" (UniqueName: \"kubernetes.io/projected/bd58da84-e12c-4bf4-a64c-d3b7155f57fa-kube-api-access-8bzff\") pod \"payload-pre-processing-e2e-api-b-a279c2-7ffb8d6599-z857m\" (UID: \"bd58da84-e12c-4bf4-a64c-d3b7155f57fa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a279c2-7ffb8d6599-z857m"
Nov 01 03:45:30.969214 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:30.968909    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a279c2-7ffb8d6599-z857m"
Nov 01 03:45:31.122309 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:31.122272    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:31.276150 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:31.276120    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a279c2-df99cdff7-rm7hx"]
Nov 01 03:45:31.276288 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:45:31.276267    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod532b2472_d6b3_4af0_905b_98953c8b814c.slice/crio-1d43fb76d6fff4535accd8b247a4f8cff6dea83276f50a35d70fb4814f93b39f WatchSource:0}: Error finding container 1d43fb76d6fff4535accd8b247a4f8cff6dea83276f50a35d70fb4814f93b39f: Status 404 returned error can't find the container with id 1d43fb76d6fff4535accd8b247a4f8cff6dea83276f50a35d70fb4814f93b39f
Nov 01 03:45:31.316161 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:31.316116    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:31.375257 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:45:31.375229    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd58da84_e12c_4bf4_a64c_d3b7155f57fa.slice/crio-3fd0f0a22a7590b666ea12a1349d2c85a67c1689cc4b402311bf5d3b970e655f WatchSource:0}: Error finding container 3fd0f0a22a7590b666ea12a1349d2c85a67c1689cc4b402311bf5d3b970e655f: Status 404 returned error can't find the container with id 3fd0f0a22a7590b666ea12a1349d2c85a67c1689cc4b402311bf5d3b970e655f
Nov 01 03:45:31.375939 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:31.375919    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a279c2-7ffb8d6599-z857m"]
Nov 01 03:45:31.484545 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:31.484515    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:32.236835 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:32.236799    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a279c2-df99cdff7-rm7hx" event={"ID":"532b2472-d6b3-4af0-905b-98953c8b814c","Type":"ContainerStarted","Data":"3d2f0b2d180c06191df3222722f56c32cf1d2927a399a0a7cd1089fd39538f3f"}
Nov 01 03:45:32.237335 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:32.236864    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a279c2-df99cdff7-rm7hx" event={"ID":"532b2472-d6b3-4af0-905b-98953c8b814c","Type":"ContainerStarted","Data":"1d43fb76d6fff4535accd8b247a4f8cff6dea83276f50a35d70fb4814f93b39f"}
Nov 01 03:45:32.237335 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:32.236894    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-a279c2-df99cdff7-rm7hx"
Nov 01 03:45:32.238637 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:32.238606    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a279c2-7ffb8d6599-z857m" event={"ID":"bd58da84-e12c-4bf4-a64c-d3b7155f57fa","Type":"ContainerStarted","Data":"5e688d8c3244479861e7209362935483f0cea38fd55e1c9e7314760d69c7906f"}
Nov 01 03:45:32.238746 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:32.238655    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a279c2-7ffb8d6599-z857m" event={"ID":"bd58da84-e12c-4bf4-a64c-d3b7155f57fa","Type":"ContainerStarted","Data":"3fd0f0a22a7590b666ea12a1349d2c85a67c1689cc4b402311bf5d3b970e655f"}
Nov 01 03:45:32.238746 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:32.238675    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a279c2-7ffb8d6599-z857m"
Nov 01 03:45:32.353849 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:32.353791    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-a279c2-df99cdff7-rm7hx" podStartSLOduration=2.353774019 podStartE2EDuration="2.353774019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:45:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:45:32.352318547 +0000 UTC m=+2336.875325136" watchObservedRunningTime="2026-11-01 03:45:32.353774019 +0000 UTC m=+2336.876780608"
Nov 01 03:45:32.474087 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:32.474009    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a279c2-7ffb8d6599-z857m" podStartSLOduration=2.473990311 podStartE2EDuration="2.473990311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:45:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:45:32.473959017 +0000 UTC m=+2336.996965605" watchObservedRunningTime="2026-11-01 03:45:32.473990311 +0000 UTC m=+2336.996996913"
Nov 01 03:45:33.893878 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:33.893843    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:35.688000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:35.687965    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:37.155455 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:37.155423    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:37.380202 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:37.380170    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:37.695410 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:37.695371    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:37.945664 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:37.945577    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:37.960619 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:37.960590    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fdbdfefd-9qs47"]
Nov 01 03:45:38.009317 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:38.009285    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fdbdfefd-9qs47"]
Nov 01 03:45:38.333844 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:38.333752    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:39.501817 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:39.501785    4930 scope.go:122] "RemoveContainer" containerID="ac37a90661a61da657110f8ee03e900f4cb61a39587b955536ef5266da847f66"
Nov 01 03:45:39.515270 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:39.515238    4930 scope.go:122] "RemoveContainer" containerID="97b71e2a40920e80db9b74b783bd2ba54980c12507d6dc99fa8e079219a23366"
Nov 01 03:45:39.621929 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:39.621897    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad" path="/var/lib/kubelet/pods/1b7551de-ee98-4dd5-ba0e-0886cfa7a2ad/volumes"
Nov 01 03:45:40.037983 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:40.037949    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:42.268607 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:42.268571    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:43.249201 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:43.249173    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a279c2-7ffb8d6599-z857m"
Nov 01 03:45:43.249396 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:43.249308    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-a279c2-df99cdff7-rm7hx"
Nov 01 03:45:43.398579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:43.398303    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:43.977797 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:43.977759    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:44.256710 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:44.256624    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:44.404130 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:44.404091    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:44.567704 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:44.567628    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4f30-lnvvf"]
Nov 01 03:45:44.615486 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:44.615455    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da4f30-lnvvf"]
Nov 01 03:45:44.999147 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:44.999114    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:45.377754 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:45.377722    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:45.623993 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:45.623960    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="695decaa-433d-4b80-92f0-761191c7c66b" path="/var/lib/kubelet/pods/695decaa-433d-4b80-92f0-761191c7c66b/volumes"
Nov 01 03:45:46.143413 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:46.143381    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:46.710586 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:46.710526    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"]
Nov 01 03:45:46.716338 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:46.716305    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"
Nov 01 03:45:46.859063 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:46.847939    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-170ffa-openshift-default-dockercfg-k4tjj\""
Nov 01 03:45:46.859063 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:46.854870    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"]
Nov 01 03:45:46.859063 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:46.855998    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df136678-b446-4fe6-8705-48d36455f70a-istio-data\") pod \"e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq\" (UID: \"df136678-b446-4fe6-8705-48d36455f70a\") " pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"
Nov 01 03:45:46.859063 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:46.856112    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dzxb\" (UniqueName: \"kubernetes.io/projected/df136678-b446-4fe6-8705-48d36455f70a-kube-api-access-2dzxb\") pod \"e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq\" (UID: \"df136678-b446-4fe6-8705-48d36455f70a\") " pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"
Nov 01 03:45:46.859063 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:46.856145    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df136678-b446-4fe6-8705-48d36455f70a-istiod-ca-cert\") pod \"e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq\" (UID: \"df136678-b446-4fe6-8705-48d36455f70a\") " pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"
Nov 01 03:45:46.859063 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:46.856206    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df136678-b446-4fe6-8705-48d36455f70a-istio-token\") pod \"e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq\" (UID: \"df136678-b446-4fe6-8705-48d36455f70a\") " pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"
Nov 01 03:45:46.859063 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:46.856244    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df136678-b446-4fe6-8705-48d36455f70a-workload-socket\") pod \"e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq\" (UID: \"df136678-b446-4fe6-8705-48d36455f70a\") " pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"
Nov 01 03:45:46.859063 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:46.856271    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df136678-b446-4fe6-8705-48d36455f70a-workload-certs\") pod \"e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq\" (UID: \"df136678-b446-4fe6-8705-48d36455f70a\") " pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"
Nov 01 03:45:46.859063 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:46.856333    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df136678-b446-4fe6-8705-48d36455f70a-credential-socket\") pod \"e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq\" (UID: \"df136678-b446-4fe6-8705-48d36455f70a\") " pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"
Nov 01 03:45:46.859063 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:46.856397    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df136678-b446-4fe6-8705-48d36455f70a-istio-podinfo\") pod \"e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq\" (UID: \"df136678-b446-4fe6-8705-48d36455f70a\") " pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"
Nov 01 03:45:46.859063 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:46.856452    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df136678-b446-4fe6-8705-48d36455f70a-istio-envoy\") pod \"e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq\" (UID: \"df136678-b446-4fe6-8705-48d36455f70a\") " pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"
Nov 01 03:45:46.926577 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:46.925189    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:46.957644 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:46.957490    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df136678-b446-4fe6-8705-48d36455f70a-istio-podinfo\") pod \"e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq\" (UID: \"df136678-b446-4fe6-8705-48d36455f70a\") " pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"
Nov 01 03:45:46.958524 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:46.958458    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df136678-b446-4fe6-8705-48d36455f70a-istio-envoy\") pod \"e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq\" (UID: \"df136678-b446-4fe6-8705-48d36455f70a\") " pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"
Nov 01 03:45:46.959206 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:46.959173    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df136678-b446-4fe6-8705-48d36455f70a-istio-data\") pod \"e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq\" (UID: \"df136678-b446-4fe6-8705-48d36455f70a\") " pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"
Nov 01 03:45:46.959348 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:46.959326    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dzxb\" (UniqueName: \"kubernetes.io/projected/df136678-b446-4fe6-8705-48d36455f70a-kube-api-access-2dzxb\") pod \"e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq\" (UID: \"df136678-b446-4fe6-8705-48d36455f70a\") " pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"
Nov 01 03:45:46.959426 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:46.959373    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df136678-b446-4fe6-8705-48d36455f70a-istiod-ca-cert\") pod \"e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq\" (UID: \"df136678-b446-4fe6-8705-48d36455f70a\") " pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"
Nov 01 03:45:46.959499 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:46.959447    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df136678-b446-4fe6-8705-48d36455f70a-istio-token\") pod \"e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq\" (UID: \"df136678-b446-4fe6-8705-48d36455f70a\") " pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"
Nov 01 03:45:46.959499 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:46.959492    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df136678-b446-4fe6-8705-48d36455f70a-workload-socket\") pod \"e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq\" (UID: \"df136678-b446-4fe6-8705-48d36455f70a\") " pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"
Nov 01 03:45:46.959808 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:46.959524    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df136678-b446-4fe6-8705-48d36455f70a-workload-certs\") pod \"e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq\" (UID: \"df136678-b446-4fe6-8705-48d36455f70a\") " pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"
Nov 01 03:45:46.959808 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:46.959638    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df136678-b446-4fe6-8705-48d36455f70a-credential-socket\") pod \"e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq\" (UID: \"df136678-b446-4fe6-8705-48d36455f70a\") " pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"
Nov 01 03:45:46.960121 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:46.960094    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df136678-b446-4fe6-8705-48d36455f70a-credential-socket\") pod \"e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq\" (UID: \"df136678-b446-4fe6-8705-48d36455f70a\") " pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"
Nov 01 03:45:46.960797 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:46.960705    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df136678-b446-4fe6-8705-48d36455f70a-workload-socket\") pod \"e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq\" (UID: \"df136678-b446-4fe6-8705-48d36455f70a\") " pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"
Nov 01 03:45:46.960996 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:46.960973    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df136678-b446-4fe6-8705-48d36455f70a-workload-certs\") pod \"e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq\" (UID: \"df136678-b446-4fe6-8705-48d36455f70a\") " pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"
Nov 01 03:45:46.961287 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:46.961245    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df136678-b446-4fe6-8705-48d36455f70a-istio-data\") pod \"e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq\" (UID: \"df136678-b446-4fe6-8705-48d36455f70a\") " pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"
Nov 01 03:45:46.961509 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:46.961490    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df136678-b446-4fe6-8705-48d36455f70a-istiod-ca-cert\") pod \"e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq\" (UID: \"df136678-b446-4fe6-8705-48d36455f70a\") " pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"
Nov 01 03:45:46.969543 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:46.969519    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df136678-b446-4fe6-8705-48d36455f70a-istio-envoy\") pod \"e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq\" (UID: \"df136678-b446-4fe6-8705-48d36455f70a\") " pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"
Nov 01 03:45:46.976604 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:46.976581    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df136678-b446-4fe6-8705-48d36455f70a-istio-podinfo\") pod \"e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq\" (UID: \"df136678-b446-4fe6-8705-48d36455f70a\") " pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"
Nov 01 03:45:47.021729 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:47.021697    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df136678-b446-4fe6-8705-48d36455f70a-istio-token\") pod \"e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq\" (UID: \"df136678-b446-4fe6-8705-48d36455f70a\") " pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"
Nov 01 03:45:47.021958 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:47.021934    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dzxb\" (UniqueName: \"kubernetes.io/projected/df136678-b446-4fe6-8705-48d36455f70a-kube-api-access-2dzxb\") pod \"e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq\" (UID: \"df136678-b446-4fe6-8705-48d36455f70a\") " pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"
Nov 01 03:45:47.037882 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:47.037860    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"
Nov 01 03:45:47.323246 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:47.323151    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:47.478116 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:47.478089    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"]
Nov 01 03:45:47.479272 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:45:47.479246    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf136678_b446_4fe6_8705_48d36455f70a.slice/crio-4392d0856b91ea02967f7cdb2da919bc2688a3c05538756e22e02f86b9424ecf WatchSource:0}: Error finding container 4392d0856b91ea02967f7cdb2da919bc2688a3c05538756e22e02f86b9424ecf: Status 404 returned error can't find the container with id 4392d0856b91ea02967f7cdb2da919bc2688a3c05538756e22e02f86b9424ecf
Nov 01 03:45:47.482207 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:47.482166    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:45:47.482305 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:47.482257    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:45:47.482347 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:47.482301    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:45:48.402130 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:48.402090    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq" event={"ID":"df136678-b446-4fe6-8705-48d36455f70a","Type":"ContainerStarted","Data":"3e880d4951c987dca9006bff08e898da20ab4f7766a79e22143fe0e1331e9341"}
Nov 01 03:45:48.402633 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:48.402201    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq" event={"ID":"df136678-b446-4fe6-8705-48d36455f70a","Type":"ContainerStarted","Data":"4392d0856b91ea02967f7cdb2da919bc2688a3c05538756e22e02f86b9424ecf"}
Nov 01 03:45:48.528367 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:48.528314    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq" podStartSLOduration=2.5283003429999997 podStartE2EDuration="2.528300343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:45:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:45:48.524788614 +0000 UTC m=+2353.047795219" watchObservedRunningTime="2026-11-01 03:45:48.528300343 +0000 UTC m=+2353.051306930"
Nov 01 03:45:49.038505 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:49.038469    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"
Nov 01 03:45:49.043805 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:49.043778    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"
Nov 01 03:45:49.122771 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:45:49.122729    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:45:49.413990 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:49.413958    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"
Nov 01 03:45:49.414925 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:49.414900    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"
Nov 01 03:45:49.471401 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:49.471359    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:49.723885 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:49.723803    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:51.197581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:51.197521    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:51.387787 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:51.387751    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:51.563901 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:51.563807    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:51.574049 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:51.574021    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"]
Nov 01 03:45:51.574258 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:51.574230    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp" podUID="cbc558d1-0eff-46fb-a780-7ebcc1939928" containerName="istio-proxy" containerID="cri-o://c5099fb040f8f6428c70efc92e151d1df3a217145c12039d9916fcd19e52f068" gracePeriod=30
Nov 01 03:45:51.868120 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:51.868080    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:52.102871 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:52.102835    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:52.582733 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:52.582694    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c8cb9a-jt4cl"]
Nov 01 03:45:52.591294 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:52.591266    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c8cb9a-jt4cl"
Nov 01 03:45:52.640973 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:52.640944    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c8cb9a-jt4cl"]
Nov 01 03:45:52.643538 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:52.643512    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:45:52.645138 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:52.645113    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b62f0069-2704-41a4-8da7-f5a5f8c77c94-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c8cb9a-jt4cl\" (UID: \"b62f0069-2704-41a4-8da7-f5a5f8c77c94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c8cb9a-jt4cl"
Nov 01 03:45:52.645277 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:52.645238    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfbbf\" (UniqueName: \"kubernetes.io/projected/b62f0069-2704-41a4-8da7-f5a5f8c77c94-kube-api-access-qfbbf\") pod \"maas-api-revoke-keys-e2e-worker-w2-c8cb9a-jt4cl\" (UID: \"b62f0069-2704-41a4-8da7-f5a5f8c77c94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c8cb9a-jt4cl"
Nov 01 03:45:52.645654 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:52.645635    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b62f0069-2704-41a4-8da7-f5a5f8c77c94-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c8cb9a-jt4cl\" (UID: \"b62f0069-2704-41a4-8da7-f5a5f8c77c94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c8cb9a-jt4cl"
Nov 01 03:45:52.685541 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:52.685506    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:45:52.685674 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:52.685511    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:45:52.685674 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:52.685536    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:45:52.746873 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:52.746843    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b62f0069-2704-41a4-8da7-f5a5f8c77c94-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c8cb9a-jt4cl\" (UID: \"b62f0069-2704-41a4-8da7-f5a5f8c77c94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c8cb9a-jt4cl"
Nov 01 03:45:52.747139 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:52.747111    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b62f0069-2704-41a4-8da7-f5a5f8c77c94-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c8cb9a-jt4cl\" (UID: \"b62f0069-2704-41a4-8da7-f5a5f8c77c94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c8cb9a-jt4cl"
Nov 01 03:45:52.747262 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:52.747229    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfbbf\" (UniqueName: \"kubernetes.io/projected/b62f0069-2704-41a4-8da7-f5a5f8c77c94-kube-api-access-qfbbf\") pod \"maas-api-revoke-keys-e2e-worker-w2-c8cb9a-jt4cl\" (UID: \"b62f0069-2704-41a4-8da7-f5a5f8c77c94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c8cb9a-jt4cl"
Nov 01 03:45:52.747687 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:52.747665    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b62f0069-2704-41a4-8da7-f5a5f8c77c94-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c8cb9a-jt4cl\" (UID: \"b62f0069-2704-41a4-8da7-f5a5f8c77c94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c8cb9a-jt4cl"
Nov 01 03:45:52.749847 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:52.749822    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b62f0069-2704-41a4-8da7-f5a5f8c77c94-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c8cb9a-jt4cl\" (UID: \"b62f0069-2704-41a4-8da7-f5a5f8c77c94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c8cb9a-jt4cl"
Nov 01 03:45:52.808608 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:52.808536    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfbbf\" (UniqueName: \"kubernetes.io/projected/b62f0069-2704-41a4-8da7-f5a5f8c77c94-kube-api-access-qfbbf\") pod \"maas-api-revoke-keys-e2e-worker-w2-c8cb9a-jt4cl\" (UID: \"b62f0069-2704-41a4-8da7-f5a5f8c77c94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c8cb9a-jt4cl"
Nov 01 03:45:52.910916 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:52.910887    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c8cb9a-jt4cl"
Nov 01 03:45:53.291153 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:53.291120    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c8cb9a-jt4cl"]
Nov 01 03:45:53.293154 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:45:53.293128    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb62f0069_2704_41a4_8da7_f5a5f8c77c94.slice/crio-a83443d8992d92a6e82efb96935e0acf94e5fa4ef364c38ae80553e801061eca WatchSource:0}: Error finding container a83443d8992d92a6e82efb96935e0acf94e5fa4ef364c38ae80553e801061eca: Status 404 returned error can't find the container with id a83443d8992d92a6e82efb96935e0acf94e5fa4ef364c38ae80553e801061eca
Nov 01 03:45:53.447900 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:53.447870    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c8cb9a-jt4cl" event={"ID":"b62f0069-2704-41a4-8da7-f5a5f8c77c94","Type":"ContainerStarted","Data":"a83443d8992d92a6e82efb96935e0acf94e5fa4ef364c38ae80553e801061eca"}
Nov 01 03:45:54.553653 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:54.543637    4930 generic.go:356] "Generic (PLEG): container finished" podID="b62f0069-2704-41a4-8da7-f5a5f8c77c94" containerID="8d960751e099b7006c7f5ed8ea67b1dfa12bc03926be84ccced86c44d92a2d3d" exitCode=0
Nov 01 03:45:54.553653 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:54.543905    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c8cb9a-jt4cl" event={"ID":"b62f0069-2704-41a4-8da7-f5a5f8c77c94","Type":"ContainerDied","Data":"8d960751e099b7006c7f5ed8ea67b1dfa12bc03926be84ccced86c44d92a2d3d"}
Nov 01 03:45:54.971616 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:54.970046    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:55.113834 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:55.113791    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:55.299663 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:55.295295    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:55.818636 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:55.817089    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-170ffa-75c44b9ddc-grjt8"]
Nov 01 03:45:55.825726 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:55.825696    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-170ffa-75c44b9ddc-grjt8"]
Nov 01 03:45:55.825848 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:55.825826    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-170ffa-75c44b9ddc-grjt8"
Nov 01 03:45:55.889200 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:55.889170    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-170ffa-dockercfg-r64nw\""
Nov 01 03:45:55.896326 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:55.896299    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c8cb9a-jt4cl"
Nov 01 03:45:55.903635 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:55.903598    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b62f0069-2704-41a4-8da7-f5a5f8c77c94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b62f0069-2704-41a4-8da7-f5a5f8c77c94-maas-api-xks-ca\") pod \"b62f0069-2704-41a4-8da7-f5a5f8c77c94\" (UID: \"b62f0069-2704-41a4-8da7-f5a5f8c77c94\") "
Nov 01 03:45:55.903753 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:55.903717    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b62f0069-2704-41a4-8da7-f5a5f8c77c94-kube-api-access-qfbbf\" (UniqueName: \"kubernetes.io/projected/b62f0069-2704-41a4-8da7-f5a5f8c77c94-kube-api-access-qfbbf\") pod \"b62f0069-2704-41a4-8da7-f5a5f8c77c94\" (UID: \"b62f0069-2704-41a4-8da7-f5a5f8c77c94\") "
Nov 01 03:45:55.903897 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:55.903882    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b62f0069-2704-41a4-8da7-f5a5f8c77c94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b62f0069-2704-41a4-8da7-f5a5f8c77c94-maas-api-openshift-ca\") pod \"b62f0069-2704-41a4-8da7-f5a5f8c77c94\" (UID: \"b62f0069-2704-41a4-8da7-f5a5f8c77c94\") "
Nov 01 03:45:55.904272 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:55.904248    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86df4787-7837-437c-a76d-2bf797b9242f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-170ffa-75c44b9ddc-grjt8\" (UID: \"86df4787-7837-437c-a76d-2bf797b9242f\") " pod="openshift-ingress/payload-processing-e2e-api-b-170ffa-75c44b9ddc-grjt8"
Nov 01 03:45:55.904385 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:55.904300    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4487s\" (UniqueName: \"kubernetes.io/projected/86df4787-7837-437c-a76d-2bf797b9242f-kube-api-access-4487s\") pod \"payload-processing-e2e-api-b-170ffa-75c44b9ddc-grjt8\" (UID: \"86df4787-7837-437c-a76d-2bf797b9242f\") " pod="openshift-ingress/payload-processing-e2e-api-b-170ffa-75c44b9ddc-grjt8"
Nov 01 03:45:55.904545 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:55.904519    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b62f0069-2704-41a4-8da7-f5a5f8c77c94-maas-api-openshift-ca" pod "b62f0069-2704-41a4-8da7-f5a5f8c77c94" (UID: "b62f0069-2704-41a4-8da7-f5a5f8c77c94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:45:55.904766 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:55.904745    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b62f0069-2704-41a4-8da7-f5a5f8c77c94-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:55.906621 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:55.906591    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b62f0069-2704-41a4-8da7-f5a5f8c77c94-kube-api-access-qfbbf" pod "b62f0069-2704-41a4-8da7-f5a5f8c77c94" (UID: "b62f0069-2704-41a4-8da7-f5a5f8c77c94"). InnerVolumeSpecName "kube-api-access-qfbbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:45:55.906738 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:55.906720    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b62f0069-2704-41a4-8da7-f5a5f8c77c94-maas-api-xks-ca" pod "b62f0069-2704-41a4-8da7-f5a5f8c77c94" (UID: "b62f0069-2704-41a4-8da7-f5a5f8c77c94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:45:55.912179 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:55.912163    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-170ffa\""
Nov 01 03:45:56.006101 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:56.006064    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4487s\" (UniqueName: \"kubernetes.io/projected/86df4787-7837-437c-a76d-2bf797b9242f-kube-api-access-4487s\") pod \"payload-processing-e2e-api-b-170ffa-75c44b9ddc-grjt8\" (UID: \"86df4787-7837-437c-a76d-2bf797b9242f\") " pod="openshift-ingress/payload-processing-e2e-api-b-170ffa-75c44b9ddc-grjt8"
Nov 01 03:45:56.006305 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:56.006293    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86df4787-7837-437c-a76d-2bf797b9242f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-170ffa-75c44b9ddc-grjt8\" (UID: \"86df4787-7837-437c-a76d-2bf797b9242f\") " pod="openshift-ingress/payload-processing-e2e-api-b-170ffa-75c44b9ddc-grjt8"
Nov 01 03:45:56.006352 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:56.006345    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b62f0069-2704-41a4-8da7-f5a5f8c77c94-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:56.006392 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:56.006357    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfbbf\" (UniqueName: \"kubernetes.io/projected/b62f0069-2704-41a4-8da7-f5a5f8c77c94-kube-api-access-qfbbf\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:56.007015 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:56.006995    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86df4787-7837-437c-a76d-2bf797b9242f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-170ffa-75c44b9ddc-grjt8\" (UID: \"86df4787-7837-437c-a76d-2bf797b9242f\") " pod="openshift-ingress/payload-processing-e2e-api-b-170ffa-75c44b9ddc-grjt8"
Nov 01 03:45:56.091630 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:56.091586    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4487s\" (UniqueName: \"kubernetes.io/projected/86df4787-7837-437c-a76d-2bf797b9242f-kube-api-access-4487s\") pod \"payload-processing-e2e-api-b-170ffa-75c44b9ddc-grjt8\" (UID: \"86df4787-7837-437c-a76d-2bf797b9242f\") " pod="openshift-ingress/payload-processing-e2e-api-b-170ffa-75c44b9ddc-grjt8"
Nov 01 03:45:56.146859 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:56.146829    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-170ffa-75c44b9ddc-grjt8"
Nov 01 03:45:56.544025 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:56.543994    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-170ffa-75c44b9ddc-grjt8"]
Nov 01 03:45:56.544198 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:45:56.544139    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86df4787_7837_437c_a76d_2bf797b9242f.slice/crio-d2c864e4ff2903ead7d1d3e9a80a129ab270c60bb10dee66671bc0cc0ba986ab WatchSource:0}: Error finding container d2c864e4ff2903ead7d1d3e9a80a129ab270c60bb10dee66671bc0cc0ba986ab: Status 404 returned error can't find the container with id d2c864e4ff2903ead7d1d3e9a80a129ab270c60bb10dee66671bc0cc0ba986ab
Nov 01 03:45:56.563486 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:56.563383    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c8cb9a-jt4cl" event={"ID":"b62f0069-2704-41a4-8da7-f5a5f8c77c94","Type":"ContainerDied","Data":"a83443d8992d92a6e82efb96935e0acf94e5fa4ef364c38ae80553e801061eca"}
Nov 01 03:45:56.563486 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:56.563423    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c8cb9a-jt4cl"
Nov 01 03:45:56.563486 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:56.563448    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a83443d8992d92a6e82efb96935e0acf94e5fa4ef364c38ae80553e801061eca"
Nov 01 03:45:56.565343 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:56.565318    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-170ffa-75c44b9ddc-grjt8" event={"ID":"86df4787-7837-437c-a76d-2bf797b9242f","Type":"ContainerStarted","Data":"d2c864e4ff2903ead7d1d3e9a80a129ab270c60bb10dee66671bc0cc0ba986ab"}
Nov 01 03:45:56.857686 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:56.857657    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"
Nov 01 03:45:56.921929 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:56.921900    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbc558d1-0eff-46fb-a780-7ebcc1939928-kube-api-access-nhsz5\" (UniqueName: \"kubernetes.io/projected/cbc558d1-0eff-46fb-a780-7ebcc1939928-kube-api-access-nhsz5\") pod \"cbc558d1-0eff-46fb-a780-7ebcc1939928\" (UID: \"cbc558d1-0eff-46fb-a780-7ebcc1939928\") "
Nov 01 03:45:56.922079 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:56.921939    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbc558d1-0eff-46fb-a780-7ebcc1939928-istio-token\" (UniqueName: \"kubernetes.io/projected/cbc558d1-0eff-46fb-a780-7ebcc1939928-istio-token\") pod \"cbc558d1-0eff-46fb-a780-7ebcc1939928\" (UID: \"cbc558d1-0eff-46fb-a780-7ebcc1939928\") "
Nov 01 03:45:56.922079 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:56.921977    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbc558d1-0eff-46fb-a780-7ebcc1939928-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc558d1-0eff-46fb-a780-7ebcc1939928-istio-envoy\") pod \"cbc558d1-0eff-46fb-a780-7ebcc1939928\" (UID: \"cbc558d1-0eff-46fb-a780-7ebcc1939928\") "
Nov 01 03:45:56.922079 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:56.922050    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cbc558d1-0eff-46fb-a780-7ebcc1939928-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc558d1-0eff-46fb-a780-7ebcc1939928-istio-podinfo\") pod \"cbc558d1-0eff-46fb-a780-7ebcc1939928\" (UID: \"cbc558d1-0eff-46fb-a780-7ebcc1939928\") "
Nov 01 03:45:56.922264 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:56.922093    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbc558d1-0eff-46fb-a780-7ebcc1939928-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc558d1-0eff-46fb-a780-7ebcc1939928-credential-socket\") pod \"cbc558d1-0eff-46fb-a780-7ebcc1939928\" (UID: \"cbc558d1-0eff-46fb-a780-7ebcc1939928\") "
Nov 01 03:45:56.922264 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:56.922158    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbc558d1-0eff-46fb-a780-7ebcc1939928-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc558d1-0eff-46fb-a780-7ebcc1939928-workload-certs\") pod \"cbc558d1-0eff-46fb-a780-7ebcc1939928\" (UID: \"cbc558d1-0eff-46fb-a780-7ebcc1939928\") "
Nov 01 03:45:56.922264 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:56.922205    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbc558d1-0eff-46fb-a780-7ebcc1939928-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc558d1-0eff-46fb-a780-7ebcc1939928-istiod-ca-cert\") pod \"cbc558d1-0eff-46fb-a780-7ebcc1939928\" (UID: \"cbc558d1-0eff-46fb-a780-7ebcc1939928\") "
Nov 01 03:45:56.922424 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:56.922330    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbc558d1-0eff-46fb-a780-7ebcc1939928-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc558d1-0eff-46fb-a780-7ebcc1939928-istio-data\") pod \"cbc558d1-0eff-46fb-a780-7ebcc1939928\" (UID: \"cbc558d1-0eff-46fb-a780-7ebcc1939928\") "
Nov 01 03:45:56.922424 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:56.922366    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc558d1-0eff-46fb-a780-7ebcc1939928-credential-socket" pod "cbc558d1-0eff-46fb-a780-7ebcc1939928" (UID: "cbc558d1-0eff-46fb-a780-7ebcc1939928"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:45:56.922424 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:56.922376    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbc558d1-0eff-46fb-a780-7ebcc1939928-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc558d1-0eff-46fb-a780-7ebcc1939928-workload-socket\") pod \"cbc558d1-0eff-46fb-a780-7ebcc1939928\" (UID: \"cbc558d1-0eff-46fb-a780-7ebcc1939928\") "
Nov 01 03:45:56.922597 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:56.922568    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc558d1-0eff-46fb-a780-7ebcc1939928-workload-socket" pod "cbc558d1-0eff-46fb-a780-7ebcc1939928" (UID: "cbc558d1-0eff-46fb-a780-7ebcc1939928"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:45:56.922597 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:56.922580    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc558d1-0eff-46fb-a780-7ebcc1939928-workload-certs" pod "cbc558d1-0eff-46fb-a780-7ebcc1939928" (UID: "cbc558d1-0eff-46fb-a780-7ebcc1939928"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:45:56.922800 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:56.922777    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc558d1-0eff-46fb-a780-7ebcc1939928-istio-data" pod "cbc558d1-0eff-46fb-a780-7ebcc1939928" (UID: "cbc558d1-0eff-46fb-a780-7ebcc1939928"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:45:56.922977 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:56.922876    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbc558d1-0eff-46fb-a780-7ebcc1939928-istiod-ca-cert" pod "cbc558d1-0eff-46fb-a780-7ebcc1939928" (UID: "cbc558d1-0eff-46fb-a780-7ebcc1939928"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:45:56.923161 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:56.923140    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc558d1-0eff-46fb-a780-7ebcc1939928-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:56.923239 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:56.923167    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc558d1-0eff-46fb-a780-7ebcc1939928-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:56.923239 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:56.923181    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc558d1-0eff-46fb-a780-7ebcc1939928-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:56.923239 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:56.923196    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc558d1-0eff-46fb-a780-7ebcc1939928-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:56.923239 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:56.923211    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc558d1-0eff-46fb-a780-7ebcc1939928-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:56.924471 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:56.924451    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbc558d1-0eff-46fb-a780-7ebcc1939928-kube-api-access-nhsz5" pod "cbc558d1-0eff-46fb-a780-7ebcc1939928" (UID: "cbc558d1-0eff-46fb-a780-7ebcc1939928"). InnerVolumeSpecName "kube-api-access-nhsz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:45:56.924547 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:56.924523    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbc558d1-0eff-46fb-a780-7ebcc1939928-istio-token" pod "cbc558d1-0eff-46fb-a780-7ebcc1939928" (UID: "cbc558d1-0eff-46fb-a780-7ebcc1939928"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:45:56.924741 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:56.924723    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbc558d1-0eff-46fb-a780-7ebcc1939928-istio-podinfo" pod "cbc558d1-0eff-46fb-a780-7ebcc1939928" (UID: "cbc558d1-0eff-46fb-a780-7ebcc1939928"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:45:56.924818 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:56.924741    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc558d1-0eff-46fb-a780-7ebcc1939928-istio-envoy" pod "cbc558d1-0eff-46fb-a780-7ebcc1939928" (UID: "cbc558d1-0eff-46fb-a780-7ebcc1939928"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:45:57.024902 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:57.024867    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhsz5\" (UniqueName: \"kubernetes.io/projected/cbc558d1-0eff-46fb-a780-7ebcc1939928-kube-api-access-nhsz5\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:57.024902 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:57.024905    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbc558d1-0eff-46fb-a780-7ebcc1939928-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:57.025132 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:57.024921    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc558d1-0eff-46fb-a780-7ebcc1939928-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:57.025132 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:57.024935    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc558d1-0eff-46fb-a780-7ebcc1939928-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:45:57.070526 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:57.070494    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"]
Nov 01 03:45:57.097992 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:57.097928    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"]
Nov 01 03:45:57.263025 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:57.262987    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:45:57.574524 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:57.574433    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-170ffa-75c44b9ddc-grjt8" event={"ID":"86df4787-7837-437c-a76d-2bf797b9242f","Type":"ContainerStarted","Data":"eba404c37d24cd2b883c784b477b212eea119cb2b2ef6cc004328e927d6ae774"}
Nov 01 03:45:57.574524 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:57.574506    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-170ffa-75c44b9ddc-grjt8"
Nov 01 03:45:57.576296 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:57.576266    4930 generic.go:356] "Generic (PLEG): container finished" podID="cbc558d1-0eff-46fb-a780-7ebcc1939928" containerID="c5099fb040f8f6428c70efc92e151d1df3a217145c12039d9916fcd19e52f068" exitCode=0
Nov 01 03:45:57.576424 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:57.576318    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"
Nov 01 03:45:57.576424 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:57.576346    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp" event={"ID":"cbc558d1-0eff-46fb-a780-7ebcc1939928","Type":"ContainerDied","Data":"c5099fb040f8f6428c70efc92e151d1df3a217145c12039d9916fcd19e52f068"}
Nov 01 03:45:57.576424 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:57.576396    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp" event={"ID":"cbc558d1-0eff-46fb-a780-7ebcc1939928","Type":"ContainerDied","Data":"731f50264c3ac9dfe2dccae7e1687eb149d36f0e501ce78fba96a0df6e202aef"}
Nov 01 03:45:57.576424 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:57.576418    4930 scope.go:122] "RemoveContainer" containerID="c5099fb040f8f6428c70efc92e151d1df3a217145c12039d9916fcd19e52f068"
Nov 01 03:45:57.587990 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:57.587973    4930 scope.go:122] "RemoveContainer" containerID="9081f00c6971d46de11917b10aa3ec83e72de04f3eea18d8d3d871a37934e8af"
Nov 01 03:45:57.597690 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:57.597676    4930 scope.go:122] "RemoveContainer" containerID="c5099fb040f8f6428c70efc92e151d1df3a217145c12039d9916fcd19e52f068"
Nov 01 03:45:57.597956 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:45:57.597939    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5099fb040f8f6428c70efc92e151d1df3a217145c12039d9916fcd19e52f068\": container with ID starting with c5099fb040f8f6428c70efc92e151d1df3a217145c12039d9916fcd19e52f068 not found: ID does not exist" containerID="c5099fb040f8f6428c70efc92e151d1df3a217145c12039d9916fcd19e52f068"
Nov 01 03:45:57.598035 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:57.597962    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5099fb040f8f6428c70efc92e151d1df3a217145c12039d9916fcd19e52f068"} err="failed to get container status \"c5099fb040f8f6428c70efc92e151d1df3a217145c12039d9916fcd19e52f068\": rpc error: code = NotFound desc = could not find container \"c5099fb040f8f6428c70efc92e151d1df3a217145c12039d9916fcd19e52f068\": container with ID starting with c5099fb040f8f6428c70efc92e151d1df3a217145c12039d9916fcd19e52f068 not found: ID does not exist"
Nov 01 03:45:57.598035 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:57.597979    4930 scope.go:122] "RemoveContainer" containerID="9081f00c6971d46de11917b10aa3ec83e72de04f3eea18d8d3d871a37934e8af"
Nov 01 03:45:57.598222 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:45:57.598205    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9081f00c6971d46de11917b10aa3ec83e72de04f3eea18d8d3d871a37934e8af\": container with ID starting with 9081f00c6971d46de11917b10aa3ec83e72de04f3eea18d8d3d871a37934e8af not found: ID does not exist" containerID="9081f00c6971d46de11917b10aa3ec83e72de04f3eea18d8d3d871a37934e8af"
Nov 01 03:45:57.598265 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:57.598225    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9081f00c6971d46de11917b10aa3ec83e72de04f3eea18d8d3d871a37934e8af"} err="failed to get container status \"9081f00c6971d46de11917b10aa3ec83e72de04f3eea18d8d3d871a37934e8af\": rpc error: code = NotFound desc = could not find container \"9081f00c6971d46de11917b10aa3ec83e72de04f3eea18d8d3d871a37934e8af\": container with ID starting with 9081f00c6971d46de11917b10aa3ec83e72de04f3eea18d8d3d871a37934e8af not found: ID does not exist"
Nov 01 03:45:57.696312 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:57.696248    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-170ffa-75c44b9ddc-grjt8" podStartSLOduration=2.6962283989999998 podStartE2EDuration="2.696228399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:45:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:45:57.694111309 +0000 UTC m=+2362.217117900" watchObservedRunningTime="2026-11-01 03:45:57.696228399 +0000 UTC m=+2362.219234990"
Nov 01 03:45:57.826874 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:57.824629    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"]
Nov 01 03:45:57.862016 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:57.861981    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-da4f30-openshift-default-5fb64b479-9htvp"]
Nov 01 03:45:59.630819 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:45:59.630784    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbc558d1-0eff-46fb-a780-7ebcc1939928" path="/var/lib/kubelet/pods/cbc558d1-0eff-46fb-a780-7ebcc1939928/volumes"
Nov 01 03:46:00.529581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:00.529514    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:46:04.530244 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:04.530206    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:46:04.812087 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:04.811999    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:46:05.449221 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:05.449189    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:46:08.588570 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:08.588522    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-170ffa-75c44b9ddc-grjt8"
Nov 01 03:46:10.352058 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:10.352018    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:46:12.091529 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.091470    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm" podUID="75d142b3-7172-4b57-bc07-a2f404c4656e" containerName="main" containerID="cri-o://4f89b4379fe0a3729d8ba889940d964e9964876362cdfa6d6a25741d85491dd3" gracePeriod=45
Nov 01 03:46:12.119067 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.119035    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf" podUID="40881383-61bb-4fc8-9c75-f8ffce4834c9" containerName="main" containerID="cri-o://c17ddece108cfda9bd3fd88e744de128173b6f42f4b663ef7ffe3d972c58daef" gracePeriod=45
Nov 01 03:46:12.399122 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.399099    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:46:12.416695 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.416672    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:46:12.529694 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.529663    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75d142b3-7172-4b57-bc07-a2f404c4656e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75d142b3-7172-4b57-bc07-a2f404c4656e-tmp-dir\") pod \"75d142b3-7172-4b57-bc07-a2f404c4656e\" (UID: \"75d142b3-7172-4b57-bc07-a2f404c4656e\") "
Nov 01 03:46:12.529859 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.529741    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75d142b3-7172-4b57-bc07-a2f404c4656e-kube-api-access-ctrm7\" (UniqueName: \"kubernetes.io/projected/75d142b3-7172-4b57-bc07-a2f404c4656e-kube-api-access-ctrm7\") pod \"75d142b3-7172-4b57-bc07-a2f404c4656e\" (UID: \"75d142b3-7172-4b57-bc07-a2f404c4656e\") "
Nov 01 03:46:12.529859 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.529789    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40881383-61bb-4fc8-9c75-f8ffce4834c9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40881383-61bb-4fc8-9c75-f8ffce4834c9-model-cache\") pod \"40881383-61bb-4fc8-9c75-f8ffce4834c9\" (UID: \"40881383-61bb-4fc8-9c75-f8ffce4834c9\") "
Nov 01 03:46:12.529859 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.529843    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40881383-61bb-4fc8-9c75-f8ffce4834c9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/40881383-61bb-4fc8-9c75-f8ffce4834c9-dshm\") pod \"40881383-61bb-4fc8-9c75-f8ffce4834c9\" (UID: \"40881383-61bb-4fc8-9c75-f8ffce4834c9\") "
Nov 01 03:46:12.530048 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.529881    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40881383-61bb-4fc8-9c75-f8ffce4834c9-tls-certs\" (UniqueName: \"kubernetes.io/secret/40881383-61bb-4fc8-9c75-f8ffce4834c9-tls-certs\") pod \"40881383-61bb-4fc8-9c75-f8ffce4834c9\" (UID: \"40881383-61bb-4fc8-9c75-f8ffce4834c9\") "
Nov 01 03:46:12.530048 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.529913    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75d142b3-7172-4b57-bc07-a2f404c4656e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75d142b3-7172-4b57-bc07-a2f404c4656e-model-cache\") pod \"75d142b3-7172-4b57-bc07-a2f404c4656e\" (UID: \"75d142b3-7172-4b57-bc07-a2f404c4656e\") "
Nov 01 03:46:12.530048 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.529924    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75d142b3-7172-4b57-bc07-a2f404c4656e-tmp-dir" pod "75d142b3-7172-4b57-bc07-a2f404c4656e" (UID: "75d142b3-7172-4b57-bc07-a2f404c4656e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:46:12.530048 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.530005    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75d142b3-7172-4b57-bc07-a2f404c4656e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/75d142b3-7172-4b57-bc07-a2f404c4656e-dshm\") pod \"75d142b3-7172-4b57-bc07-a2f404c4656e\" (UID: \"75d142b3-7172-4b57-bc07-a2f404c4656e\") "
Nov 01 03:46:12.530869 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.530084    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40881383-61bb-4fc8-9c75-f8ffce4834c9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40881383-61bb-4fc8-9c75-f8ffce4834c9-tmp-dir\") pod \"40881383-61bb-4fc8-9c75-f8ffce4834c9\" (UID: \"40881383-61bb-4fc8-9c75-f8ffce4834c9\") "
Nov 01 03:46:12.530869 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.530132    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40881383-61bb-4fc8-9c75-f8ffce4834c9-model-cache" pod "40881383-61bb-4fc8-9c75-f8ffce4834c9" (UID: "40881383-61bb-4fc8-9c75-f8ffce4834c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:46:12.530869 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.530156    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75d142b3-7172-4b57-bc07-a2f404c4656e-home\" (UniqueName: \"kubernetes.io/empty-dir/75d142b3-7172-4b57-bc07-a2f404c4656e-home\") pod \"75d142b3-7172-4b57-bc07-a2f404c4656e\" (UID: \"75d142b3-7172-4b57-bc07-a2f404c4656e\") "
Nov 01 03:46:12.530869 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.530207    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40881383-61bb-4fc8-9c75-f8ffce4834c9-home\" (UniqueName: \"kubernetes.io/empty-dir/40881383-61bb-4fc8-9c75-f8ffce4834c9-home\") pod \"40881383-61bb-4fc8-9c75-f8ffce4834c9\" (UID: \"40881383-61bb-4fc8-9c75-f8ffce4834c9\") "
Nov 01 03:46:12.530869 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.530244    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40881383-61bb-4fc8-9c75-f8ffce4834c9-kube-api-access-jc7gk\" (UniqueName: \"kubernetes.io/projected/40881383-61bb-4fc8-9c75-f8ffce4834c9-kube-api-access-jc7gk\") pod \"40881383-61bb-4fc8-9c75-f8ffce4834c9\" (UID: \"40881383-61bb-4fc8-9c75-f8ffce4834c9\") "
Nov 01 03:46:12.530869 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.530280    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75d142b3-7172-4b57-bc07-a2f404c4656e-tls-certs\" (UniqueName: \"kubernetes.io/secret/75d142b3-7172-4b57-bc07-a2f404c4656e-tls-certs\") pod \"75d142b3-7172-4b57-bc07-a2f404c4656e\" (UID: \"75d142b3-7172-4b57-bc07-a2f404c4656e\") "
Nov 01 03:46:12.530869 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.530392    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75d142b3-7172-4b57-bc07-a2f404c4656e-model-cache" pod "75d142b3-7172-4b57-bc07-a2f404c4656e" (UID: "75d142b3-7172-4b57-bc07-a2f404c4656e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:46:12.530869 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.530475    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40881383-61bb-4fc8-9c75-f8ffce4834c9-tmp-dir" pod "40881383-61bb-4fc8-9c75-f8ffce4834c9" (UID: "40881383-61bb-4fc8-9c75-f8ffce4834c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:46:12.530869 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.530482    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75d142b3-7172-4b57-bc07-a2f404c4656e-home" pod "75d142b3-7172-4b57-bc07-a2f404c4656e" (UID: "75d142b3-7172-4b57-bc07-a2f404c4656e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:46:12.530869 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.530648    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40881383-61bb-4fc8-9c75-f8ffce4834c9-home" pod "40881383-61bb-4fc8-9c75-f8ffce4834c9" (UID: "40881383-61bb-4fc8-9c75-f8ffce4834c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:46:12.531488 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.531065    4930 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75d142b3-7172-4b57-bc07-a2f404c4656e-home\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:46:12.531488 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.531087    4930 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40881383-61bb-4fc8-9c75-f8ffce4834c9-home\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:46:12.531488 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.531100    4930 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75d142b3-7172-4b57-bc07-a2f404c4656e-tmp-dir\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:46:12.531488 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.531114    4930 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40881383-61bb-4fc8-9c75-f8ffce4834c9-model-cache\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:46:12.531488 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.531130    4930 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75d142b3-7172-4b57-bc07-a2f404c4656e-model-cache\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:46:12.531488 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.531143    4930 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40881383-61bb-4fc8-9c75-f8ffce4834c9-tmp-dir\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:46:12.535927 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.535887    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75d142b3-7172-4b57-bc07-a2f404c4656e-dshm" pod "75d142b3-7172-4b57-bc07-a2f404c4656e" (UID: "75d142b3-7172-4b57-bc07-a2f404c4656e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:46:12.535927 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.535901    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40881383-61bb-4fc8-9c75-f8ffce4834c9-dshm" pod "40881383-61bb-4fc8-9c75-f8ffce4834c9" (UID: "40881383-61bb-4fc8-9c75-f8ffce4834c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:46:12.536141 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.535931    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40881383-61bb-4fc8-9c75-f8ffce4834c9-kube-api-access-jc7gk" pod "40881383-61bb-4fc8-9c75-f8ffce4834c9" (UID: "40881383-61bb-4fc8-9c75-f8ffce4834c9"). InnerVolumeSpecName "kube-api-access-jc7gk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:46:12.536141 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.535938    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40881383-61bb-4fc8-9c75-f8ffce4834c9-tls-certs" pod "40881383-61bb-4fc8-9c75-f8ffce4834c9" (UID: "40881383-61bb-4fc8-9c75-f8ffce4834c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:46:12.536141 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.535972    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75d142b3-7172-4b57-bc07-a2f404c4656e-kube-api-access-ctrm7" pod "75d142b3-7172-4b57-bc07-a2f404c4656e" (UID: "75d142b3-7172-4b57-bc07-a2f404c4656e"). InnerVolumeSpecName "kube-api-access-ctrm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:46:12.537428 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.537404    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75d142b3-7172-4b57-bc07-a2f404c4656e-tls-certs" pod "75d142b3-7172-4b57-bc07-a2f404c4656e" (UID: "75d142b3-7172-4b57-bc07-a2f404c4656e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:46:12.633331 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.633241    4930 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40881383-61bb-4fc8-9c75-f8ffce4834c9-dshm\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:46:12.633331 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.633275    4930 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40881383-61bb-4fc8-9c75-f8ffce4834c9-tls-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:46:12.633331 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.633287    4930 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75d142b3-7172-4b57-bc07-a2f404c4656e-dshm\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:46:12.633331 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.633303    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc7gk\" (UniqueName: \"kubernetes.io/projected/40881383-61bb-4fc8-9c75-f8ffce4834c9-kube-api-access-jc7gk\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:46:12.633331 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.633315    4930 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75d142b3-7172-4b57-bc07-a2f404c4656e-tls-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:46:12.633331 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.633324    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctrm7\" (UniqueName: \"kubernetes.io/projected/75d142b3-7172-4b57-bc07-a2f404c4656e-kube-api-access-ctrm7\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:46:12.722282 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.722244    4930 generic.go:356] "Generic (PLEG): container finished" podID="40881383-61bb-4fc8-9c75-f8ffce4834c9" containerID="c17ddece108cfda9bd3fd88e744de128173b6f42f4b663ef7ffe3d972c58daef" exitCode=0
Nov 01 03:46:12.722455 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.722290    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"
Nov 01 03:46:12.722455 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.722324    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf" event={"ID":"40881383-61bb-4fc8-9c75-f8ffce4834c9","Type":"ContainerDied","Data":"c17ddece108cfda9bd3fd88e744de128173b6f42f4b663ef7ffe3d972c58daef"}
Nov 01 03:46:12.722455 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.722377    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf" event={"ID":"40881383-61bb-4fc8-9c75-f8ffce4834c9","Type":"ContainerDied","Data":"f6c5bb10d0c66dd0631c674a2b89f53b2a32c5c0e2e6eb5ef90a90d2189d51bc"}
Nov 01 03:46:12.722455 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.722399    4930 scope.go:122] "RemoveContainer" containerID="c17ddece108cfda9bd3fd88e744de128173b6f42f4b663ef7ffe3d972c58daef"
Nov 01 03:46:12.736821 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.736791    4930 generic.go:356] "Generic (PLEG): container finished" podID="75d142b3-7172-4b57-bc07-a2f404c4656e" containerID="4f89b4379fe0a3729d8ba889940d964e9964876362cdfa6d6a25741d85491dd3" exitCode=0
Nov 01 03:46:12.737020 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.736997    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm" event={"ID":"75d142b3-7172-4b57-bc07-a2f404c4656e","Type":"ContainerDied","Data":"4f89b4379fe0a3729d8ba889940d964e9964876362cdfa6d6a25741d85491dd3"}
Nov 01 03:46:12.737112 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.737068    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm" event={"ID":"75d142b3-7172-4b57-bc07-a2f404c4656e","Type":"ContainerDied","Data":"7c26a1d58e68e1187ca042c85fb853f07b13328c7cc01cbcca3188291f6b471b"}
Nov 01 03:46:12.737186 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.737174    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"
Nov 01 03:46:12.749630 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.749602    4930 scope.go:122] "RemoveContainer" containerID="c17ddece108cfda9bd3fd88e744de128173b6f42f4b663ef7ffe3d972c58daef"
Nov 01 03:46:12.750025 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:46:12.749997    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c17ddece108cfda9bd3fd88e744de128173b6f42f4b663ef7ffe3d972c58daef\": container with ID starting with c17ddece108cfda9bd3fd88e744de128173b6f42f4b663ef7ffe3d972c58daef not found: ID does not exist" containerID="c17ddece108cfda9bd3fd88e744de128173b6f42f4b663ef7ffe3d972c58daef"
Nov 01 03:46:12.750105 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.750040    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c17ddece108cfda9bd3fd88e744de128173b6f42f4b663ef7ffe3d972c58daef"} err="failed to get container status \"c17ddece108cfda9bd3fd88e744de128173b6f42f4b663ef7ffe3d972c58daef\": rpc error: code = NotFound desc = could not find container \"c17ddece108cfda9bd3fd88e744de128173b6f42f4b663ef7ffe3d972c58daef\": container with ID starting with c17ddece108cfda9bd3fd88e744de128173b6f42f4b663ef7ffe3d972c58daef not found: ID does not exist"
Nov 01 03:46:12.750105 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.750068    4930 scope.go:122] "RemoveContainer" containerID="4f89b4379fe0a3729d8ba889940d964e9964876362cdfa6d6a25741d85491dd3"
Nov 01 03:46:12.759775 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.759754    4930 scope.go:122] "RemoveContainer" containerID="4f89b4379fe0a3729d8ba889940d964e9964876362cdfa6d6a25741d85491dd3"
Nov 01 03:46:12.760038 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:46:12.760020    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f89b4379fe0a3729d8ba889940d964e9964876362cdfa6d6a25741d85491dd3\": container with ID starting with 4f89b4379fe0a3729d8ba889940d964e9964876362cdfa6d6a25741d85491dd3 not found: ID does not exist" containerID="4f89b4379fe0a3729d8ba889940d964e9964876362cdfa6d6a25741d85491dd3"
Nov 01 03:46:12.760100 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.760043    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f89b4379fe0a3729d8ba889940d964e9964876362cdfa6d6a25741d85491dd3"} err="failed to get container status \"4f89b4379fe0a3729d8ba889940d964e9964876362cdfa6d6a25741d85491dd3\": rpc error: code = NotFound desc = could not find container \"4f89b4379fe0a3729d8ba889940d964e9964876362cdfa6d6a25741d85491dd3\": container with ID starting with 4f89b4379fe0a3729d8ba889940d964e9964876362cdfa6d6a25741d85491dd3 not found: ID does not exist"
Nov 01 03:46:12.886416 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.886351    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"]
Nov 01 03:46:12.889658 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.889638    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-c8cb9a/distinct-w2-c8cb9a-kserve-68fbc8bd95-vndxm"]
Nov 01 03:46:12.996266 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.996230    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"]
Nov 01 03:46:12.999638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:12.999615    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-c8cb9a/simulator-w2-c8cb9a-kserve-75d44fd4d9-95blf"]
Nov 01 03:46:13.621783 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:13.621753    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40881383-61bb-4fc8-9c75-f8ffce4834c9" path="/var/lib/kubelet/pods/40881383-61bb-4fc8-9c75-f8ffce4834c9/volumes"
Nov 01 03:46:13.622222 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:13.622208    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75d142b3-7172-4b57-bc07-a2f404c4656e" path="/var/lib/kubelet/pods/75d142b3-7172-4b57-bc07-a2f404c4656e/volumes"
Nov 01 03:46:15.113851 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:15.113815    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:46:19.122706 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:46:19.122618    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:46:19.923047 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:19.923016    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:46:20.068334 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:20.068298    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:46:20.412028 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:20.411994    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:46:21.079343 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:21.079307    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:46:24.873846 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:24.873812    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:46:27.028302 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:27.028261    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:46:27.290343 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:27.290258    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:46:29.664465 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:29.664426    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:46:33.865442 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:33.865403    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3a679a8-45wbr"]
Nov 01 03:46:33.866050 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:33.866034    4930 cpu_manager.go:414] "containerMap: removing container" podUID="cbc558d1-0eff-46fb-a780-7ebcc1939928" containerName="istio-proxy"
Nov 01 03:46:33.866116 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:33.866051    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbc558d1-0eff-46fb-a780-7ebcc1939928" containerName="istio-proxy"
Nov 01 03:46:33.866116 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:33.866079    4930 cpu_manager.go:414] "containerMap: removing container" podUID="75d142b3-7172-4b57-bc07-a2f404c4656e" containerName="main"
Nov 01 03:46:33.866116 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:33.866087    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75d142b3-7172-4b57-bc07-a2f404c4656e" containerName="main"
Nov 01 03:46:33.866116 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:33.866099    4930 cpu_manager.go:414] "containerMap: removing container" podUID="40881383-61bb-4fc8-9c75-f8ffce4834c9" containerName="main"
Nov 01 03:46:33.866116 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:33.866104    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40881383-61bb-4fc8-9c75-f8ffce4834c9" containerName="main"
Nov 01 03:46:33.866116 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:33.866115    4930 cpu_manager.go:414] "containerMap: removing container" podUID="cbc558d1-0eff-46fb-a780-7ebcc1939928" containerName="istio-proxy"
Nov 01 03:46:33.866445 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:33.866120    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbc558d1-0eff-46fb-a780-7ebcc1939928" containerName="istio-proxy"
Nov 01 03:46:33.866445 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:33.866145    4930 cpu_manager.go:414] "containerMap: removing container" podUID="b62f0069-2704-41a4-8da7-f5a5f8c77c94" containerName="revoke-keys"
Nov 01 03:46:33.866445 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:33.866150    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b62f0069-2704-41a4-8da7-f5a5f8c77c94" containerName="revoke-keys"
Nov 01 03:46:33.866445 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:33.866261    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="b62f0069-2704-41a4-8da7-f5a5f8c77c94" containerName="revoke-keys"
Nov 01 03:46:33.866445 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:33.866272    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbc558d1-0eff-46fb-a780-7ebcc1939928" containerName="istio-proxy"
Nov 01 03:46:33.866445 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:33.866280    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="75d142b3-7172-4b57-bc07-a2f404c4656e" containerName="main"
Nov 01 03:46:33.866445 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:33.866286    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="40881383-61bb-4fc8-9c75-f8ffce4834c9" containerName="main"
Nov 01 03:46:33.870230 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:33.870206    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a679a8-45wbr"
Nov 01 03:46:33.928956 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:33.928924    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa699d2c-hht2n"]
Nov 01 03:46:33.929930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:33.929905    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbc558d1-0eff-46fb-a780-7ebcc1939928" containerName="istio-proxy"
Nov 01 03:46:33.933814 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:33.933793    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa699d2c-hht2n"
Nov 01 03:46:33.976723 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:33.976672    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:46:33.976863 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:33.976726    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:46:33.976863 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:33.976765    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:46:33.976863 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:33.976826    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:46:33.977797 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:33.977769    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3a679a8-45wbr"]
Nov 01 03:46:33.992001 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:33.991976    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa699d2c-hht2n"]
Nov 01 03:46:34.007900 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:34.007871    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35100fb8-078b-40cf-a01a-f92b559d6a54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3a679a8-45wbr\" (UID: \"35100fb8-078b-40cf-a01a-f92b559d6a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a679a8-45wbr"
Nov 01 03:46:34.008031 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:34.007952    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fe8f9eb-df64-4911-a80d-d0416d9b48c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa699d2c-hht2n\" (UID: \"2fe8f9eb-df64-4911-a80d-d0416d9b48c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa699d2c-hht2n"
Nov 01 03:46:34.008101 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:34.008050    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fe8f9eb-df64-4911-a80d-d0416d9b48c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa699d2c-hht2n\" (UID: \"2fe8f9eb-df64-4911-a80d-d0416d9b48c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa699d2c-hht2n"
Nov 01 03:46:34.008230 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:34.008200    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35100fb8-078b-40cf-a01a-f92b559d6a54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3a679a8-45wbr\" (UID: \"35100fb8-078b-40cf-a01a-f92b559d6a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a679a8-45wbr"
Nov 01 03:46:34.008365 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:34.008334    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwzr4\" (UniqueName: \"kubernetes.io/projected/35100fb8-078b-40cf-a01a-f92b559d6a54-kube-api-access-fwzr4\") pod \"maas-api-revoke-sub-d3a679a8-45wbr\" (UID: \"35100fb8-078b-40cf-a01a-f92b559d6a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a679a8-45wbr"
Nov 01 03:46:34.008431 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:34.008374    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f66d4\" (UniqueName: \"kubernetes.io/projected/2fe8f9eb-df64-4911-a80d-d0416d9b48c4-kube-api-access-f66d4\") pod \"maas-api-revoke-sub-aa699d2c-hht2n\" (UID: \"2fe8f9eb-df64-4911-a80d-d0416d9b48c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa699d2c-hht2n"
Nov 01 03:46:34.109442 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:34.109402    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwzr4\" (UniqueName: \"kubernetes.io/projected/35100fb8-078b-40cf-a01a-f92b559d6a54-kube-api-access-fwzr4\") pod \"maas-api-revoke-sub-d3a679a8-45wbr\" (UID: \"35100fb8-078b-40cf-a01a-f92b559d6a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a679a8-45wbr"
Nov 01 03:46:34.109628 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:34.109458    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f66d4\" (UniqueName: \"kubernetes.io/projected/2fe8f9eb-df64-4911-a80d-d0416d9b48c4-kube-api-access-f66d4\") pod \"maas-api-revoke-sub-aa699d2c-hht2n\" (UID: \"2fe8f9eb-df64-4911-a80d-d0416d9b48c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa699d2c-hht2n"
Nov 01 03:46:34.109628 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:34.109519    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35100fb8-078b-40cf-a01a-f92b559d6a54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3a679a8-45wbr\" (UID: \"35100fb8-078b-40cf-a01a-f92b559d6a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a679a8-45wbr"
Nov 01 03:46:34.109628 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:34.109543    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fe8f9eb-df64-4911-a80d-d0416d9b48c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa699d2c-hht2n\" (UID: \"2fe8f9eb-df64-4911-a80d-d0416d9b48c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa699d2c-hht2n"
Nov 01 03:46:34.109820 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:34.109654    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fe8f9eb-df64-4911-a80d-d0416d9b48c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa699d2c-hht2n\" (UID: \"2fe8f9eb-df64-4911-a80d-d0416d9b48c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa699d2c-hht2n"
Nov 01 03:46:34.109820 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:34.109732    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35100fb8-078b-40cf-a01a-f92b559d6a54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3a679a8-45wbr\" (UID: \"35100fb8-078b-40cf-a01a-f92b559d6a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a679a8-45wbr"
Nov 01 03:46:34.110311 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:34.110280    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fe8f9eb-df64-4911-a80d-d0416d9b48c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa699d2c-hht2n\" (UID: \"2fe8f9eb-df64-4911-a80d-d0416d9b48c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa699d2c-hht2n"
Nov 01 03:46:34.110450 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:34.110363    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35100fb8-078b-40cf-a01a-f92b559d6a54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3a679a8-45wbr\" (UID: \"35100fb8-078b-40cf-a01a-f92b559d6a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a679a8-45wbr"
Nov 01 03:46:34.112019 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:34.111999    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35100fb8-078b-40cf-a01a-f92b559d6a54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3a679a8-45wbr\" (UID: \"35100fb8-078b-40cf-a01a-f92b559d6a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a679a8-45wbr"
Nov 01 03:46:34.112113 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:34.112061    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fe8f9eb-df64-4911-a80d-d0416d9b48c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa699d2c-hht2n\" (UID: \"2fe8f9eb-df64-4911-a80d-d0416d9b48c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa699d2c-hht2n"
Nov 01 03:46:34.170301 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:34.170224    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f66d4\" (UniqueName: \"kubernetes.io/projected/2fe8f9eb-df64-4911-a80d-d0416d9b48c4-kube-api-access-f66d4\") pod \"maas-api-revoke-sub-aa699d2c-hht2n\" (UID: \"2fe8f9eb-df64-4911-a80d-d0416d9b48c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa699d2c-hht2n"
Nov 01 03:46:34.170301 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:34.170292    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwzr4\" (UniqueName: \"kubernetes.io/projected/35100fb8-078b-40cf-a01a-f92b559d6a54-kube-api-access-fwzr4\") pod \"maas-api-revoke-sub-d3a679a8-45wbr\" (UID: \"35100fb8-078b-40cf-a01a-f92b559d6a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a679a8-45wbr"
Nov 01 03:46:34.185111 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:34.185081    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a679a8-45wbr"
Nov 01 03:46:34.248464 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:34.248403    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa699d2c-hht2n"
Nov 01 03:46:34.600945 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:34.600912    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3a679a8-45wbr"]
Nov 01 03:46:34.601761 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:46:34.601734    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35100fb8_078b_40cf_a01a_f92b559d6a54.slice/crio-0130feec98a4cfc53b47c734934136dcf7f624ecf417afae5a3838f745402f1e WatchSource:0}: Error finding container 0130feec98a4cfc53b47c734934136dcf7f624ecf417afae5a3838f745402f1e: Status 404 returned error can't find the container with id 0130feec98a4cfc53b47c734934136dcf7f624ecf417afae5a3838f745402f1e
Nov 01 03:46:34.672962 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:34.672935    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa699d2c-hht2n"]
Nov 01 03:46:34.675781 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:46:34.675754    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fe8f9eb_df64_4911_a80d_d0416d9b48c4.slice/crio-760ea514fe0f511c1bba6490358dacba2cd7ae75c698cb7fd384f0619d4c8c6a WatchSource:0}: Error finding container 760ea514fe0f511c1bba6490358dacba2cd7ae75c698cb7fd384f0619d4c8c6a: Status 404 returned error can't find the container with id 760ea514fe0f511c1bba6490358dacba2cd7ae75c698cb7fd384f0619d4c8c6a
Nov 01 03:46:34.745259 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:34.745216    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:46:34.918300 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:34.918257    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa699d2c-hht2n" event={"ID":"2fe8f9eb-df64-4911-a80d-d0416d9b48c4","Type":"ContainerStarted","Data":"760ea514fe0f511c1bba6490358dacba2cd7ae75c698cb7fd384f0619d4c8c6a"}
Nov 01 03:46:34.919906 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:34.919876    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a679a8-45wbr" event={"ID":"35100fb8-078b-40cf-a01a-f92b559d6a54","Type":"ContainerStarted","Data":"0130feec98a4cfc53b47c734934136dcf7f624ecf417afae5a3838f745402f1e"}
Nov 01 03:46:35.534581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:35.530670    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:46:35.696856 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:35.696819    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:46:35.930951 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:35.930909    4930 generic.go:356] "Generic (PLEG): container finished" podID="35100fb8-078b-40cf-a01a-f92b559d6a54" containerID="178af8eef4981adaa2a84b281e9506e29c21e0701cb3c96cffa502f942e81468" exitCode=0
Nov 01 03:46:35.931389 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:35.930944    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a679a8-45wbr" event={"ID":"35100fb8-078b-40cf-a01a-f92b559d6a54","Type":"ContainerDied","Data":"178af8eef4981adaa2a84b281e9506e29c21e0701cb3c96cffa502f942e81468"}
Nov 01 03:46:35.932661 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:35.932638    4930 generic.go:356] "Generic (PLEG): container finished" podID="2fe8f9eb-df64-4911-a80d-d0416d9b48c4" containerID="48d7a189b781566bb304c358915d7d08ad7615eff2e76d5506e02eeb97c6f800" exitCode=0
Nov 01 03:46:35.932785 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:35.932698    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa699d2c-hht2n" event={"ID":"2fe8f9eb-df64-4911-a80d-d0416d9b48c4","Type":"ContainerDied","Data":"48d7a189b781566bb304c358915d7d08ad7615eff2e76d5506e02eeb97c6f800"}
Nov 01 03:46:37.131359 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:37.131335    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa699d2c-hht2n"
Nov 01 03:46:37.135435 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:37.135414    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a679a8-45wbr"
Nov 01 03:46:37.261324 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:37.261247    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fe8f9eb-df64-4911-a80d-d0416d9b48c4-kube-api-access-f66d4\" (UniqueName: \"kubernetes.io/projected/2fe8f9eb-df64-4911-a80d-d0416d9b48c4-kube-api-access-f66d4\") pod \"2fe8f9eb-df64-4911-a80d-d0416d9b48c4\" (UID: \"2fe8f9eb-df64-4911-a80d-d0416d9b48c4\") "
Nov 01 03:46:37.261503 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:37.261372    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35100fb8-078b-40cf-a01a-f92b559d6a54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35100fb8-078b-40cf-a01a-f92b559d6a54-maas-api-openshift-ca\") pod \"35100fb8-078b-40cf-a01a-f92b559d6a54\" (UID: \"35100fb8-078b-40cf-a01a-f92b559d6a54\") "
Nov 01 03:46:37.261503 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:37.261439    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fe8f9eb-df64-4911-a80d-d0416d9b48c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fe8f9eb-df64-4911-a80d-d0416d9b48c4-maas-api-openshift-ca\") pod \"2fe8f9eb-df64-4911-a80d-d0416d9b48c4\" (UID: \"2fe8f9eb-df64-4911-a80d-d0416d9b48c4\") "
Nov 01 03:46:37.261503 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:37.261482    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fe8f9eb-df64-4911-a80d-d0416d9b48c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fe8f9eb-df64-4911-a80d-d0416d9b48c4-maas-api-xks-ca\") pod \"2fe8f9eb-df64-4911-a80d-d0416d9b48c4\" (UID: \"2fe8f9eb-df64-4911-a80d-d0416d9b48c4\") "
Nov 01 03:46:37.261723 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:37.261525    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35100fb8-078b-40cf-a01a-f92b559d6a54-kube-api-access-fwzr4\" (UniqueName: \"kubernetes.io/projected/35100fb8-078b-40cf-a01a-f92b559d6a54-kube-api-access-fwzr4\") pod \"35100fb8-078b-40cf-a01a-f92b559d6a54\" (UID: \"35100fb8-078b-40cf-a01a-f92b559d6a54\") "
Nov 01 03:46:37.261839 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:37.261789    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35100fb8-078b-40cf-a01a-f92b559d6a54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35100fb8-078b-40cf-a01a-f92b559d6a54-maas-api-xks-ca\") pod \"35100fb8-078b-40cf-a01a-f92b559d6a54\" (UID: \"35100fb8-078b-40cf-a01a-f92b559d6a54\") "
Nov 01 03:46:37.261960 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:37.261932    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fe8f9eb-df64-4911-a80d-d0416d9b48c4-maas-api-openshift-ca" pod "2fe8f9eb-df64-4911-a80d-d0416d9b48c4" (UID: "2fe8f9eb-df64-4911-a80d-d0416d9b48c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:46:37.262533 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:37.262420    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fe8f9eb-df64-4911-a80d-d0416d9b48c4-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:46:37.262533 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:37.262512    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35100fb8-078b-40cf-a01a-f92b559d6a54-maas-api-openshift-ca" pod "35100fb8-078b-40cf-a01a-f92b559d6a54" (UID: "35100fb8-078b-40cf-a01a-f92b559d6a54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:46:37.264065 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:37.264031    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fe8f9eb-df64-4911-a80d-d0416d9b48c4-kube-api-access-f66d4" pod "2fe8f9eb-df64-4911-a80d-d0416d9b48c4" (UID: "2fe8f9eb-df64-4911-a80d-d0416d9b48c4"). InnerVolumeSpecName "kube-api-access-f66d4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:46:37.264328 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:37.264307    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35100fb8-078b-40cf-a01a-f92b559d6a54-maas-api-xks-ca" pod "35100fb8-078b-40cf-a01a-f92b559d6a54" (UID: "35100fb8-078b-40cf-a01a-f92b559d6a54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:46:37.265032 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:37.265006    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35100fb8-078b-40cf-a01a-f92b559d6a54-kube-api-access-fwzr4" pod "35100fb8-078b-40cf-a01a-f92b559d6a54" (UID: "35100fb8-078b-40cf-a01a-f92b559d6a54"). InnerVolumeSpecName "kube-api-access-fwzr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:46:37.265143 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:37.265088    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fe8f9eb-df64-4911-a80d-d0416d9b48c4-maas-api-xks-ca" pod "2fe8f9eb-df64-4911-a80d-d0416d9b48c4" (UID: "2fe8f9eb-df64-4911-a80d-d0416d9b48c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:46:37.364611 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:37.363530    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35100fb8-078b-40cf-a01a-f92b559d6a54-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:46:37.364611 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:37.363599    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f66d4\" (UniqueName: \"kubernetes.io/projected/2fe8f9eb-df64-4911-a80d-d0416d9b48c4-kube-api-access-f66d4\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:46:37.364611 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:37.363619    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35100fb8-078b-40cf-a01a-f92b559d6a54-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:46:37.364611 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:37.363634    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fe8f9eb-df64-4911-a80d-d0416d9b48c4-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:46:37.364611 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:37.363649    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwzr4\" (UniqueName: \"kubernetes.io/projected/35100fb8-078b-40cf-a01a-f92b559d6a54-kube-api-access-fwzr4\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:46:37.950487 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:37.950400    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa699d2c-hht2n"
Nov 01 03:46:37.950487 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:37.950431    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa699d2c-hht2n" event={"ID":"2fe8f9eb-df64-4911-a80d-d0416d9b48c4","Type":"ContainerDied","Data":"760ea514fe0f511c1bba6490358dacba2cd7ae75c698cb7fd384f0619d4c8c6a"}
Nov 01 03:46:37.950733 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:37.950489    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="760ea514fe0f511c1bba6490358dacba2cd7ae75c698cb7fd384f0619d4c8c6a"
Nov 01 03:46:37.951946 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:37.951923    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a679a8-45wbr" event={"ID":"35100fb8-078b-40cf-a01a-f92b559d6a54","Type":"ContainerDied","Data":"0130feec98a4cfc53b47c734934136dcf7f624ecf417afae5a3838f745402f1e"}
Nov 01 03:46:37.951946 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:37.951941    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3a679a8-45wbr"
Nov 01 03:46:37.952098 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:37.951966    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0130feec98a4cfc53b47c734934136dcf7f624ecf417afae5a3838f745402f1e"
Nov 01 03:46:38.256960 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:38.256874    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jh955_f29ea61e-2a78-4c11-9566-f52497e135ff/console-operator/2.log"
Nov 01 03:46:38.294308 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:38.294284    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jh955_f29ea61e-2a78-4c11-9566-f52497e135ff/console-operator/2.log"
Nov 01 03:46:39.488842 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:39.488806    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:46:39.614926 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:39.614883    4930 scope.go:122] "RemoveContainer" containerID="b3e97344120d9e39da6dccc6c71483f63018238994907b311bb8d4918341c1d3"
Nov 01 03:46:39.626246 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:39.626222    4930 scope.go:122] "RemoveContainer" containerID="de34d01ca23a5c16a185f85a556db8581c5feb369faf7dcd4ff8b30f19db2a37"
Nov 01 03:46:39.636631 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:39.636610    4930 scope.go:122] "RemoveContainer" containerID="5de91ad8e4e388fbe1e0613ed53765ff30a26f655ceb2390666f0d4c6c8de3da"
Nov 01 03:46:39.645981 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:39.645957    4930 scope.go:122] "RemoveContainer" containerID="e1590c7dd6bf82b6c05759094e8165f92986e62abda9d6590863f6f311be92ad"
Nov 01 03:46:39.737589 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:39.732416    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:46:39.876713 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:39.876663    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:46:40.384477 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:40.384440    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:46:41.887902 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:41.887868    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:46:42.806419 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:42.806383    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:46:44.585728 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:44.585690    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:46:48.145900 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:48.145863    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:46:49.122660 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:46:49.122621    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:46:49.350883 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:49.350846    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:46:49.833222 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:49.833183    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:46:51.533114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:51.533074    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:46:51.737397 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:51.737366    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:46:51.872573 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:51.872523    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"]
Nov 01 03:46:51.872818 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:51.872791    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9" podUID="cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20" containerName="istio-proxy" containerID="cri-o://71f46d6bf0f05d45d4c2965a70f7474e8cb6cd51eb63322d5531ca89d4ce2907" gracePeriod=30
Nov 01 03:46:51.884172 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:51.884134    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:46:54.016291 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:54.016257    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:46:54.161308 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:54.161273    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:46:54.307299 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:54.307214    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:46:55.264648 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:55.264619    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:46:56.439074 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:56.439034    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c8373-lgpzq"]
Nov 01 03:46:56.439991 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:56.439969    4930 cpu_manager.go:414] "containerMap: removing container" podUID="2fe8f9eb-df64-4911-a80d-d0416d9b48c4" containerName="revoke-keys"
Nov 01 03:46:56.440047 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:56.439994    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fe8f9eb-df64-4911-a80d-d0416d9b48c4" containerName="revoke-keys"
Nov 01 03:46:56.440047 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:56.440025    4930 cpu_manager.go:414] "containerMap: removing container" podUID="35100fb8-078b-40cf-a01a-f92b559d6a54" containerName="revoke-keys"
Nov 01 03:46:56.440047 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:56.440033    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35100fb8-078b-40cf-a01a-f92b559d6a54" containerName="revoke-keys"
Nov 01 03:46:56.440205 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:56.440192    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fe8f9eb-df64-4911-a80d-d0416d9b48c4" containerName="revoke-keys"
Nov 01 03:46:56.440242 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:56.440211    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="35100fb8-078b-40cf-a01a-f92b559d6a54" containerName="revoke-keys"
Nov 01 03:46:56.446106 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:56.446088    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c8373-lgpzq"
Nov 01 03:46:56.502291 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:56.502257    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c8373-lgpzq"]
Nov 01 03:46:56.543482 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:56.543442    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/830483ce-0cd5-45a5-a562-201d1ba81feb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2c8373-lgpzq\" (UID: \"830483ce-0cd5-45a5-a562-201d1ba81feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c8373-lgpzq"
Nov 01 03:46:56.543665 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:56.543625    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndbcb\" (UniqueName: \"kubernetes.io/projected/830483ce-0cd5-45a5-a562-201d1ba81feb-kube-api-access-ndbcb\") pod \"maas-api-revoke-keys-e2e-api-a-2c8373-lgpzq\" (UID: \"830483ce-0cd5-45a5-a562-201d1ba81feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c8373-lgpzq"
Nov 01 03:46:56.543756 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:56.543736    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/830483ce-0cd5-45a5-a562-201d1ba81feb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2c8373-lgpzq\" (UID: \"830483ce-0cd5-45a5-a562-201d1ba81feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c8373-lgpzq"
Nov 01 03:46:56.547285 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:56.547263    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:46:56.547409 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:56.547319    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:46:56.551736 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:56.551716    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:46:56.551847 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:56.551748    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:46:56.644616 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:56.644549    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/830483ce-0cd5-45a5-a562-201d1ba81feb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2c8373-lgpzq\" (UID: \"830483ce-0cd5-45a5-a562-201d1ba81feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c8373-lgpzq"
Nov 01 03:46:56.644783 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:56.644708    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndbcb\" (UniqueName: \"kubernetes.io/projected/830483ce-0cd5-45a5-a562-201d1ba81feb-kube-api-access-ndbcb\") pod \"maas-api-revoke-keys-e2e-api-a-2c8373-lgpzq\" (UID: \"830483ce-0cd5-45a5-a562-201d1ba81feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c8373-lgpzq"
Nov 01 03:46:56.644783 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:56.644759    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/830483ce-0cd5-45a5-a562-201d1ba81feb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2c8373-lgpzq\" (UID: \"830483ce-0cd5-45a5-a562-201d1ba81feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c8373-lgpzq"
Nov 01 03:46:56.645207 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:56.645184    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/830483ce-0cd5-45a5-a562-201d1ba81feb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2c8373-lgpzq\" (UID: \"830483ce-0cd5-45a5-a562-201d1ba81feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c8373-lgpzq"
Nov 01 03:46:56.647148 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:56.647118    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/830483ce-0cd5-45a5-a562-201d1ba81feb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2c8373-lgpzq\" (UID: \"830483ce-0cd5-45a5-a562-201d1ba81feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c8373-lgpzq"
Nov 01 03:46:56.705197 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:56.705131    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndbcb\" (UniqueName: \"kubernetes.io/projected/830483ce-0cd5-45a5-a562-201d1ba81feb-kube-api-access-ndbcb\") pod \"maas-api-revoke-keys-e2e-api-a-2c8373-lgpzq\" (UID: \"830483ce-0cd5-45a5-a562-201d1ba81feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c8373-lgpzq"
Nov 01 03:46:56.759766 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:56.759737    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c8373-lgpzq"
Nov 01 03:46:57.116035 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:57.115997    4930 generic.go:356] "Generic (PLEG): container finished" podID="cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20" containerID="71f46d6bf0f05d45d4c2965a70f7474e8cb6cd51eb63322d5531ca89d4ce2907" exitCode=0
Nov 01 03:46:57.116224 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:57.116069    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9" event={"ID":"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20","Type":"ContainerDied","Data":"71f46d6bf0f05d45d4c2965a70f7474e8cb6cd51eb63322d5531ca89d4ce2907"}
Nov 01 03:46:57.116224 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:57.116143    4930 scope.go:122] "RemoveContainer" containerID="c2d5829d14a12ad87679cd55ed38ee14bcc992364fd70ad5c1245017a0bd6315"
Nov 01 03:46:57.141198 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:57.141174    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c8373-lgpzq"]
Nov 01 03:46:57.141318 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:46:57.141299    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod830483ce_0cd5_45a5_a562_201d1ba81feb.slice/crio-badb5e33bcd19b958295f1cf3fe8edc5507dc34885aa3f85b1f323c2875dcb78 WatchSource:0}: Error finding container badb5e33bcd19b958295f1cf3fe8edc5507dc34885aa3f85b1f323c2875dcb78: Status 404 returned error can't find the container with id badb5e33bcd19b958295f1cf3fe8edc5507dc34885aa3f85b1f323c2875dcb78
Nov 01 03:46:57.142365 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:57.142350    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"
Nov 01 03:46:57.251666 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:57.251607    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-istiod-ca-cert\") pod \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\" (UID: \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\") "
Nov 01 03:46:57.251666 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:57.251659    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-workload-socket\") pod \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\" (UID: \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\") "
Nov 01 03:46:57.251842 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:57.251776    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-workload-certs\") pod \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\" (UID: \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\") "
Nov 01 03:46:57.251934 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:57.251920    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-istio-data\") pod \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\" (UID: \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\") "
Nov 01 03:46:57.251991 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:57.251953    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-istio-envoy\") pod \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\" (UID: \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\") "
Nov 01 03:46:57.252053 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:57.252011    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-istio-podinfo\") pod \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\" (UID: \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\") "
Nov 01 03:46:57.252053 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:57.252010    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-workload-socket" pod "cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20" (UID: "cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:46:57.252053 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:57.252033    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-workload-certs" pod "cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20" (UID: "cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:46:57.252211 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:57.252056    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-istio-token\" (UniqueName: \"kubernetes.io/projected/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-istio-token\") pod \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\" (UID: \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\") "
Nov 01 03:46:57.252211 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:57.252060    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-istiod-ca-cert" pod "cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20" (UID: "cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:46:57.252211 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:57.252090    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-kube-api-access-whqx5\" (UniqueName: \"kubernetes.io/projected/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-kube-api-access-whqx5\") pod \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\" (UID: \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\") "
Nov 01 03:46:57.252211 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:57.252121    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-credential-socket\") pod \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\" (UID: \"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20\") "
Nov 01 03:46:57.252211 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:57.252188    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-istio-data" pod "cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20" (UID: "cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:46:57.252643 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:57.252617    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-credential-socket" pod "cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20" (UID: "cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:46:57.252739 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:57.252673    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:46:57.252739 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:57.252694    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:46:57.252739 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:57.252708    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:46:57.252739 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:57.252721    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:46:57.254218 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:57.254194    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-istio-envoy" pod "cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20" (UID: "cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:46:57.254625 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:57.254590    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-istio-token" pod "cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20" (UID: "cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:46:57.254724 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:57.254672    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-kube-api-access-whqx5" pod "cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20" (UID: "cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20"). InnerVolumeSpecName "kube-api-access-whqx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:46:57.254724 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:57.254678    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-istio-podinfo" pod "cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20" (UID: "cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:46:57.353913 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:57.353885    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:46:57.353913 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:57.353912    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:46:57.353913 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:57.353921    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:46:57.354191 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:57.353931    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whqx5\" (UniqueName: \"kubernetes.io/projected/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-kube-api-access-whqx5\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:46:57.354191 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:57.353940    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:46:58.124723 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:58.124693    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"
Nov 01 03:46:58.124723 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:58.124708    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9" event={"ID":"cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20","Type":"ContainerDied","Data":"f34512be1c8ad3a24bd9138caa0316ea624938b45beb62852c863811efe1873e"}
Nov 01 03:46:58.125203 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:58.124768    4930 scope.go:122] "RemoveContainer" containerID="71f46d6bf0f05d45d4c2965a70f7474e8cb6cd51eb63322d5531ca89d4ce2907"
Nov 01 03:46:58.126720 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:58.126688    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c8373-lgpzq" event={"ID":"830483ce-0cd5-45a5-a562-201d1ba81feb","Type":"ContainerStarted","Data":"b86f9c9de01c897ef85827caa032d4fe67075e95b3b925d97388fc199d645f9b"}
Nov 01 03:46:58.126869 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:58.126727    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c8373-lgpzq" event={"ID":"830483ce-0cd5-45a5-a562-201d1ba81feb","Type":"ContainerStarted","Data":"badb5e33bcd19b958295f1cf3fe8edc5507dc34885aa3f85b1f323c2875dcb78"}
Nov 01 03:46:58.340806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:58.340774    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"]
Nov 01 03:46:58.390807 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:58.390736    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-fdbdfefd-openshift-default-5974544877-4zkd9"]
Nov 01 03:46:58.506487 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:58.506439    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c8373-lgpzq" podStartSLOduration=2.506425901 podStartE2EDuration="2.506425901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:46:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:46:58.502145196 +0000 UTC m=+2423.025151796" watchObservedRunningTime="2026-11-01 03:46:58.506425901 +0000 UTC m=+2423.029432488"
Nov 01 03:46:58.512463 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:58.512432    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55"]
Nov 01 03:46:58.512671 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:58.512649    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55" podUID="ab72eb99-1ab5-4e1d-8057-82feb8b67a66" containerName="istio-proxy" containerID="cri-o://2870bd8b8a4485aa9c8a05375b8c7c290694e3d562c326993d227d1436ee5cd2" gracePeriod=30
Nov 01 03:46:59.137446 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:59.137412    4930 generic.go:356] "Generic (PLEG): container finished" podID="830483ce-0cd5-45a5-a562-201d1ba81feb" containerID="b86f9c9de01c897ef85827caa032d4fe67075e95b3b925d97388fc199d645f9b" exitCode=0
Nov 01 03:46:59.137920 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:59.137501    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c8373-lgpzq" event={"ID":"830483ce-0cd5-45a5-a562-201d1ba81feb","Type":"ContainerDied","Data":"b86f9c9de01c897ef85827caa032d4fe67075e95b3b925d97388fc199d645f9b"}
Nov 01 03:46:59.433095 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:59.433009    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:46:59.621873 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:46:59.621838    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20" path="/var/lib/kubelet/pods/cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20/volumes"
Nov 01 03:47:00.308253 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:00.308221    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c8373-lgpzq"
Nov 01 03:47:00.498140 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:00.498062    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/830483ce-0cd5-45a5-a562-201d1ba81feb-kube-api-access-ndbcb\" (UniqueName: \"kubernetes.io/projected/830483ce-0cd5-45a5-a562-201d1ba81feb-kube-api-access-ndbcb\") pod \"830483ce-0cd5-45a5-a562-201d1ba81feb\" (UID: \"830483ce-0cd5-45a5-a562-201d1ba81feb\") "
Nov 01 03:47:00.498140 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:00.498101    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/830483ce-0cd5-45a5-a562-201d1ba81feb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/830483ce-0cd5-45a5-a562-201d1ba81feb-maas-api-xks-ca\") pod \"830483ce-0cd5-45a5-a562-201d1ba81feb\" (UID: \"830483ce-0cd5-45a5-a562-201d1ba81feb\") "
Nov 01 03:47:00.498333 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:00.498148    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/830483ce-0cd5-45a5-a562-201d1ba81feb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/830483ce-0cd5-45a5-a562-201d1ba81feb-maas-api-openshift-ca\") pod \"830483ce-0cd5-45a5-a562-201d1ba81feb\" (UID: \"830483ce-0cd5-45a5-a562-201d1ba81feb\") "
Nov 01 03:47:00.498615 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:00.498576    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/830483ce-0cd5-45a5-a562-201d1ba81feb-maas-api-openshift-ca" pod "830483ce-0cd5-45a5-a562-201d1ba81feb" (UID: "830483ce-0cd5-45a5-a562-201d1ba81feb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:47:00.498768 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:00.498690    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/830483ce-0cd5-45a5-a562-201d1ba81feb-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:47:00.500339 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:00.500311    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/830483ce-0cd5-45a5-a562-201d1ba81feb-kube-api-access-ndbcb" pod "830483ce-0cd5-45a5-a562-201d1ba81feb" (UID: "830483ce-0cd5-45a5-a562-201d1ba81feb"). InnerVolumeSpecName "kube-api-access-ndbcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:47:00.500431 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:00.500372    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/830483ce-0cd5-45a5-a562-201d1ba81feb-maas-api-xks-ca" pod "830483ce-0cd5-45a5-a562-201d1ba81feb" (UID: "830483ce-0cd5-45a5-a562-201d1ba81feb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:47:00.599971 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:00.599940    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndbcb\" (UniqueName: \"kubernetes.io/projected/830483ce-0cd5-45a5-a562-201d1ba81feb-kube-api-access-ndbcb\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:47:00.599971 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:00.599970    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/830483ce-0cd5-45a5-a562-201d1ba81feb-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:47:00.642412 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:00.642387    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-swk55 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:47:00.642579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:00.642435    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55" podUID="ab72eb99-1ab5-4e1d-8057-82feb8b67a66" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:47:01.154671 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:01.154639    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c8373-lgpzq" event={"ID":"830483ce-0cd5-45a5-a562-201d1ba81feb","Type":"ContainerDied","Data":"badb5e33bcd19b958295f1cf3fe8edc5507dc34885aa3f85b1f323c2875dcb78"}
Nov 01 03:47:01.154671 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:01.154674    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c8373-lgpzq"
Nov 01 03:47:01.154887 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:01.154686    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="badb5e33bcd19b958295f1cf3fe8edc5507dc34885aa3f85b1f323c2875dcb78"
Nov 01 03:47:03.797293 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:03.797221    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55"
Nov 01 03:47:03.830415 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:03.830388    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-istio-envoy\") pod \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\" (UID: \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\") "
Nov 01 03:47:03.830588 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:03.830446    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-istiod-ca-cert\") pod \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\" (UID: \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\") "
Nov 01 03:47:03.830588 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:03.830475    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-istio-token\" (UniqueName: \"kubernetes.io/projected/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-istio-token\") pod \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\" (UID: \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\") "
Nov 01 03:47:03.830588 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:03.830510    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-credential-socket\") pod \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\" (UID: \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\") "
Nov 01 03:47:03.830791 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:03.830586    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-workload-socket\") pod \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\" (UID: \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\") "
Nov 01 03:47:03.830791 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:03.830663    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-kube-api-access-l7x48\" (UniqueName: \"kubernetes.io/projected/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-kube-api-access-l7x48\") pod \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\" (UID: \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\") "
Nov 01 03:47:03.830791 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:03.830688    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-workload-certs\") pod \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\" (UID: \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\") "
Nov 01 03:47:03.830791 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:03.830717    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-istio-podinfo\") pod \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\" (UID: \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\") "
Nov 01 03:47:03.830791 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:03.830753    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-istio-data\") pod \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\" (UID: \"ab72eb99-1ab5-4e1d-8057-82feb8b67a66\") "
Nov 01 03:47:03.831096 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:03.830814    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-credential-socket" pod "ab72eb99-1ab5-4e1d-8057-82feb8b67a66" (UID: "ab72eb99-1ab5-4e1d-8057-82feb8b67a66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:47:03.831096 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:03.830860    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-workload-socket" pod "ab72eb99-1ab5-4e1d-8057-82feb8b67a66" (UID: "ab72eb99-1ab5-4e1d-8057-82feb8b67a66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:47:03.831096 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:03.831041    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-istiod-ca-cert" pod "ab72eb99-1ab5-4e1d-8057-82feb8b67a66" (UID: "ab72eb99-1ab5-4e1d-8057-82feb8b67a66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:47:03.831096 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:03.831061    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-workload-certs" pod "ab72eb99-1ab5-4e1d-8057-82feb8b67a66" (UID: "ab72eb99-1ab5-4e1d-8057-82feb8b67a66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:47:03.831325 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:03.831258    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-istio-data" pod "ab72eb99-1ab5-4e1d-8057-82feb8b67a66" (UID: "ab72eb99-1ab5-4e1d-8057-82feb8b67a66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:47:03.831516 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:03.831437    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:47:03.831516 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:03.831459    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:47:03.831516 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:03.831473    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:47:03.831516 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:03.831492    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:47:03.831516 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:03.831506    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:47:03.832927 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:03.832899    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-istio-podinfo" pod "ab72eb99-1ab5-4e1d-8057-82feb8b67a66" (UID: "ab72eb99-1ab5-4e1d-8057-82feb8b67a66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:47:03.832927 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:03.832899    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-istio-envoy" pod "ab72eb99-1ab5-4e1d-8057-82feb8b67a66" (UID: "ab72eb99-1ab5-4e1d-8057-82feb8b67a66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:47:03.833104 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:03.832965    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-istio-token" pod "ab72eb99-1ab5-4e1d-8057-82feb8b67a66" (UID: "ab72eb99-1ab5-4e1d-8057-82feb8b67a66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:47:03.833221 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:03.833204    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-kube-api-access-l7x48" pod "ab72eb99-1ab5-4e1d-8057-82feb8b67a66" (UID: "ab72eb99-1ab5-4e1d-8057-82feb8b67a66"). InnerVolumeSpecName "kube-api-access-l7x48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:47:03.932992 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:03.932959    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:47:03.932992 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:03.932987    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7x48\" (UniqueName: \"kubernetes.io/projected/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-kube-api-access-l7x48\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:47:03.932992 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:03.932997    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:47:03.933205 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:03.933006    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab72eb99-1ab5-4e1d-8057-82feb8b67a66-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:47:04.183174 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:04.183136    4930 generic.go:356] "Generic (PLEG): container finished" podID="ab72eb99-1ab5-4e1d-8057-82feb8b67a66" containerID="2870bd8b8a4485aa9c8a05375b8c7c290694e3d562c326993d227d1436ee5cd2" exitCode=0
Nov 01 03:47:04.183366 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:04.183207    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55"
Nov 01 03:47:04.183366 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:04.183216    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55" event={"ID":"ab72eb99-1ab5-4e1d-8057-82feb8b67a66","Type":"ContainerDied","Data":"2870bd8b8a4485aa9c8a05375b8c7c290694e3d562c326993d227d1436ee5cd2"}
Nov 01 03:47:04.183366 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:04.183284    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55" event={"ID":"ab72eb99-1ab5-4e1d-8057-82feb8b67a66","Type":"ContainerDied","Data":"4c7ed8bf62de2685f0b0fe65efe9fcdd44949fd626454a927a123f79cc6640e3"}
Nov 01 03:47:04.183366 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:04.183314    4930 scope.go:122] "RemoveContainer" containerID="2870bd8b8a4485aa9c8a05375b8c7c290694e3d562c326993d227d1436ee5cd2"
Nov 01 03:47:04.197634 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:04.197610    4930 scope.go:122] "RemoveContainer" containerID="2870bd8b8a4485aa9c8a05375b8c7c290694e3d562c326993d227d1436ee5cd2"
Nov 01 03:47:04.198109 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:47:04.198077    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2870bd8b8a4485aa9c8a05375b8c7c290694e3d562c326993d227d1436ee5cd2\": container with ID starting with 2870bd8b8a4485aa9c8a05375b8c7c290694e3d562c326993d227d1436ee5cd2 not found: ID does not exist" containerID="2870bd8b8a4485aa9c8a05375b8c7c290694e3d562c326993d227d1436ee5cd2"
Nov 01 03:47:04.198219 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:04.198121    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2870bd8b8a4485aa9c8a05375b8c7c290694e3d562c326993d227d1436ee5cd2"} err="failed to get container status \"2870bd8b8a4485aa9c8a05375b8c7c290694e3d562c326993d227d1436ee5cd2\": rpc error: code = NotFound desc = could not find container \"2870bd8b8a4485aa9c8a05375b8c7c290694e3d562c326993d227d1436ee5cd2\": container with ID starting with 2870bd8b8a4485aa9c8a05375b8c7c290694e3d562c326993d227d1436ee5cd2 not found: ID does not exist"
Nov 01 03:47:04.350009 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:04.349975    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55"]
Nov 01 03:47:04.400394 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:04.400354    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-swk55"]
Nov 01 03:47:04.669738 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:04.669703    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:47:04.823351 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:04.823293    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:47:04.969474 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:04.969378    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:47:05.224373 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:05.224256    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf"]
Nov 01 03:47:05.224924 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:05.224887    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf" podUID="fd5ef13a-5e5e-45c5-89e9-fdf4d6899067" containerName="istio-proxy" containerID="cri-o://fbbd4cfcaf19fa12907a174dec4a95dbe4faa4dffba643a7c85423579ae31143" gracePeriod=30
Nov 01 03:47:05.621685 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:05.621648    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab72eb99-1ab5-4e1d-8057-82feb8b67a66" path="/var/lib/kubelet/pods/ab72eb99-1ab5-4e1d-8057-82feb8b67a66/volumes"
Nov 01 03:47:06.520970 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:06.520932    4930 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:47:06.521341 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:06.520992    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf" podUID="fd5ef13a-5e5e-45c5-89e9-fdf4d6899067" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:47:07.967047 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:07.967010    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2c8373-966cf6b8c-8n27c"]
Nov 01 03:47:08.073178 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:08.073145    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2c8373-755cc448b7-8hkdb"]
Nov 01 03:47:08.339921 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:08.339881    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:47:08.486154 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:08.486106    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:47:09.743159 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:09.743118    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:47:10.524405 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:10.524371    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf"
Nov 01 03:47:10.554476 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:10.554395    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-kube-api-access-f8h2j\" (UniqueName: \"kubernetes.io/projected/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-kube-api-access-f8h2j\") pod \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\" (UID: \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\") "
Nov 01 03:47:10.554476 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:10.554454    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-istio-podinfo\") pod \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\" (UID: \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\") "
Nov 01 03:47:10.554476 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:10.554475    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-workload-certs\") pod \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\" (UID: \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\") "
Nov 01 03:47:10.554819 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:10.554527    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-istio-envoy\") pod \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\" (UID: \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\") "
Nov 01 03:47:10.554819 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:10.554544    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-istio-data\") pod \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\" (UID: \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\") "
Nov 01 03:47:10.554819 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:10.554599    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-istio-token\" (UniqueName: \"kubernetes.io/projected/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-istio-token\") pod \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\" (UID: \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\") "
Nov 01 03:47:10.554819 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:10.554693    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-credential-socket\") pod \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\" (UID: \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\") "
Nov 01 03:47:10.554819 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:10.554719    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-workload-socket\") pod \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\" (UID: \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\") "
Nov 01 03:47:10.554819 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:10.554811    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-istiod-ca-cert\") pod \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\" (UID: \"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067\") "
Nov 01 03:47:10.555138 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:10.554847    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-workload-certs" pod "fd5ef13a-5e5e-45c5-89e9-fdf4d6899067" (UID: "fd5ef13a-5e5e-45c5-89e9-fdf4d6899067"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:47:10.555138 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:10.555040    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-credential-socket" pod "fd5ef13a-5e5e-45c5-89e9-fdf4d6899067" (UID: "fd5ef13a-5e5e-45c5-89e9-fdf4d6899067"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:47:10.555262 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:10.555194    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-istio-data" pod "fd5ef13a-5e5e-45c5-89e9-fdf4d6899067" (UID: "fd5ef13a-5e5e-45c5-89e9-fdf4d6899067"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:47:10.555478 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:10.555453    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:47:10.555601 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:10.555484    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:47:10.555601 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:10.555501    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:47:10.555769 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:10.555740    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-workload-socket" pod "fd5ef13a-5e5e-45c5-89e9-fdf4d6899067" (UID: "fd5ef13a-5e5e-45c5-89e9-fdf4d6899067"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:47:10.555856 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:10.555834    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-istiod-ca-cert" pod "fd5ef13a-5e5e-45c5-89e9-fdf4d6899067" (UID: "fd5ef13a-5e5e-45c5-89e9-fdf4d6899067"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:47:10.556918 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:10.556888    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-istio-envoy" pod "fd5ef13a-5e5e-45c5-89e9-fdf4d6899067" (UID: "fd5ef13a-5e5e-45c5-89e9-fdf4d6899067"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:47:10.557020 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:10.556976    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-kube-api-access-f8h2j" pod "fd5ef13a-5e5e-45c5-89e9-fdf4d6899067" (UID: "fd5ef13a-5e5e-45c5-89e9-fdf4d6899067"). InnerVolumeSpecName "kube-api-access-f8h2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:47:10.557226 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:10.557205    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-istio-podinfo" pod "fd5ef13a-5e5e-45c5-89e9-fdf4d6899067" (UID: "fd5ef13a-5e5e-45c5-89e9-fdf4d6899067"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:47:10.557293 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:10.557226    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-istio-token" pod "fd5ef13a-5e5e-45c5-89e9-fdf4d6899067" (UID: "fd5ef13a-5e5e-45c5-89e9-fdf4d6899067"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:47:10.657183 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:10.657137    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8h2j\" (UniqueName: \"kubernetes.io/projected/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-kube-api-access-f8h2j\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:47:10.657183 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:10.657171    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:47:10.657183 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:10.657185    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:47:10.657444 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:10.657197    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:47:10.657444 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:10.657210    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:47:10.657444 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:10.657305    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:47:11.340354 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:11.340301    4930 generic.go:356] "Generic (PLEG): container finished" podID="fd5ef13a-5e5e-45c5-89e9-fdf4d6899067" containerID="fbbd4cfcaf19fa12907a174dec4a95dbe4faa4dffba643a7c85423579ae31143" exitCode=0
Nov 01 03:47:11.341280 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:11.340790    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf"
Nov 01 03:47:11.344601 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:11.340816    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf" event={"ID":"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067","Type":"ContainerDied","Data":"fbbd4cfcaf19fa12907a174dec4a95dbe4faa4dffba643a7c85423579ae31143"}
Nov 01 03:47:11.344771 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:11.344649    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf" event={"ID":"fd5ef13a-5e5e-45c5-89e9-fdf4d6899067","Type":"ContainerDied","Data":"828028ca9708d7473713337fee159a7e93856ce8d5bbcc87a70a71c054e91cac"}
Nov 01 03:47:11.344771 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:11.344695    4930 scope.go:122] "RemoveContainer" containerID="fbbd4cfcaf19fa12907a174dec4a95dbe4faa4dffba643a7c85423579ae31143"
Nov 01 03:47:11.380827 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:11.378464    4930 scope.go:122] "RemoveContainer" containerID="fbbd4cfcaf19fa12907a174dec4a95dbe4faa4dffba643a7c85423579ae31143"
Nov 01 03:47:11.380827 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:47:11.378949    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbbd4cfcaf19fa12907a174dec4a95dbe4faa4dffba643a7c85423579ae31143\": container with ID starting with fbbd4cfcaf19fa12907a174dec4a95dbe4faa4dffba643a7c85423579ae31143 not found: ID does not exist" containerID="fbbd4cfcaf19fa12907a174dec4a95dbe4faa4dffba643a7c85423579ae31143"
Nov 01 03:47:11.380827 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:11.378992    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbbd4cfcaf19fa12907a174dec4a95dbe4faa4dffba643a7c85423579ae31143"} err="failed to get container status \"fbbd4cfcaf19fa12907a174dec4a95dbe4faa4dffba643a7c85423579ae31143\": rpc error: code = NotFound desc = could not find container \"fbbd4cfcaf19fa12907a174dec4a95dbe4faa4dffba643a7c85423579ae31143\": container with ID starting with fbbd4cfcaf19fa12907a174dec4a95dbe4faa4dffba643a7c85423579ae31143 not found: ID does not exist"
Nov 01 03:47:11.571329 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:11.571293    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf"]
Nov 01 03:47:11.623978 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:11.623898    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-2w8gf"]
Nov 01 03:47:12.987678 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:12.987633    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-2c8373-966cf6b8c-8n27c" podUID="c031a873-378d-4081-845e-891f20c1354f" containerName="payload-processing" containerID="cri-o://d599992fc0cc58dfa484cc279f60a4bdf20647d98b723e5e9a7312f81ee2271d" gracePeriod=25
Nov 01 03:47:13.074668 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:13.074619    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c8373-755cc448b7-8hkdb" podUID="f5dcc946-5a58-43bd-88c5-520f7ecb6104" containerName="payload-pre-processing" containerID="cri-o://fef784a71930a0146fb4b7c723a394e75d36b6b8ad3b0c0098bfb7b7852460d5" gracePeriod=25
Nov 01 03:47:13.402644 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:13.395820    4930 generic.go:356] "Generic (PLEG): container finished" podID="c031a873-378d-4081-845e-891f20c1354f" containerID="d599992fc0cc58dfa484cc279f60a4bdf20647d98b723e5e9a7312f81ee2271d" exitCode=0
Nov 01 03:47:13.402644 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:13.396049    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2c8373-966cf6b8c-8n27c" event={"ID":"c031a873-378d-4081-845e-891f20c1354f","Type":"ContainerDied","Data":"d599992fc0cc58dfa484cc279f60a4bdf20647d98b723e5e9a7312f81ee2271d"}
Nov 01 03:47:13.402644 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:13.396101    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2c8373-966cf6b8c-8n27c" event={"ID":"c031a873-378d-4081-845e-891f20c1354f","Type":"ContainerDied","Data":"1b227f221c952cb0830ab5181f6c2a0e8e6cda81495bd0c11d6d2d6783a2ff40"}
Nov 01 03:47:13.402644 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:13.396128    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b227f221c952cb0830ab5181f6c2a0e8e6cda81495bd0c11d6d2d6783a2ff40"
Nov 01 03:47:13.402644 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:13.399920    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c8373-755cc448b7-8hkdb" event={"ID":"f5dcc946-5a58-43bd-88c5-520f7ecb6104","Type":"ContainerDied","Data":"fef784a71930a0146fb4b7c723a394e75d36b6b8ad3b0c0098bfb7b7852460d5"}
Nov 01 03:47:13.402644 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:13.399882    4930 generic.go:356] "Generic (PLEG): container finished" podID="f5dcc946-5a58-43bd-88c5-520f7ecb6104" containerID="fef784a71930a0146fb4b7c723a394e75d36b6b8ad3b0c0098bfb7b7852460d5" exitCode=0
Nov 01 03:47:13.418579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:13.403443    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2c8373-966cf6b8c-8n27c"
Nov 01 03:47:13.535337 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:13.535245    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c031a873-378d-4081-845e-891f20c1354f-kube-api-access-jt4w2\" (UniqueName: \"kubernetes.io/projected/c031a873-378d-4081-845e-891f20c1354f-kube-api-access-jt4w2\") pod \"c031a873-378d-4081-845e-891f20c1354f\" (UID: \"c031a873-378d-4081-845e-891f20c1354f\") "
Nov 01 03:47:13.535519 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:13.535485    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c031a873-378d-4081-845e-891f20c1354f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c031a873-378d-4081-845e-891f20c1354f-plugins-config-volume\") pod \"c031a873-378d-4081-845e-891f20c1354f\" (UID: \"c031a873-378d-4081-845e-891f20c1354f\") "
Nov 01 03:47:13.543123 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:13.540903    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c031a873-378d-4081-845e-891f20c1354f-kube-api-access-jt4w2" pod "c031a873-378d-4081-845e-891f20c1354f" (UID: "c031a873-378d-4081-845e-891f20c1354f"). InnerVolumeSpecName "kube-api-access-jt4w2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:47:13.543123 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:13.542801    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c031a873-378d-4081-845e-891f20c1354f-plugins-config-volume" pod "c031a873-378d-4081-845e-891f20c1354f" (UID: "c031a873-378d-4081-845e-891f20c1354f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:47:13.639068 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:13.639033    4930 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c031a873-378d-4081-845e-891f20c1354f-plugins-config-volume\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:47:13.639068 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:13.639068    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt4w2\" (UniqueName: \"kubernetes.io/projected/c031a873-378d-4081-845e-891f20c1354f-kube-api-access-jt4w2\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:47:13.645670 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:13.645639    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c8373-755cc448b7-8hkdb"
Nov 01 03:47:13.646691 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:13.646659    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd5ef13a-5e5e-45c5-89e9-fdf4d6899067" path="/var/lib/kubelet/pods/fd5ef13a-5e5e-45c5-89e9-fdf4d6899067/volumes"
Nov 01 03:47:13.740347 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:13.740310    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5dcc946-5a58-43bd-88c5-520f7ecb6104-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5dcc946-5a58-43bd-88c5-520f7ecb6104-plugins-config-volume\") pod \"f5dcc946-5a58-43bd-88c5-520f7ecb6104\" (UID: \"f5dcc946-5a58-43bd-88c5-520f7ecb6104\") "
Nov 01 03:47:13.740491 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:13.740424    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5dcc946-5a58-43bd-88c5-520f7ecb6104-kube-api-access-qlv4f\" (UniqueName: \"kubernetes.io/projected/f5dcc946-5a58-43bd-88c5-520f7ecb6104-kube-api-access-qlv4f\") pod \"f5dcc946-5a58-43bd-88c5-520f7ecb6104\" (UID: \"f5dcc946-5a58-43bd-88c5-520f7ecb6104\") "
Nov 01 03:47:13.741620 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:13.740747    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5dcc946-5a58-43bd-88c5-520f7ecb6104-plugins-config-volume" pod "f5dcc946-5a58-43bd-88c5-520f7ecb6104" (UID: "f5dcc946-5a58-43bd-88c5-520f7ecb6104"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:47:13.741620 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:13.741179    4930 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5dcc946-5a58-43bd-88c5-520f7ecb6104-plugins-config-volume\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:47:13.744443 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:13.744417    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5dcc946-5a58-43bd-88c5-520f7ecb6104-kube-api-access-qlv4f" pod "f5dcc946-5a58-43bd-88c5-520f7ecb6104" (UID: "f5dcc946-5a58-43bd-88c5-520f7ecb6104"). InnerVolumeSpecName "kube-api-access-qlv4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:47:13.842785 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:13.842747    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlv4f\" (UniqueName: \"kubernetes.io/projected/f5dcc946-5a58-43bd-88c5-520f7ecb6104-kube-api-access-qlv4f\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:47:14.409274 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:14.409247    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2c8373-966cf6b8c-8n27c"
Nov 01 03:47:14.409751 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:14.409269    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c8373-755cc448b7-8hkdb" event={"ID":"f5dcc946-5a58-43bd-88c5-520f7ecb6104","Type":"ContainerDied","Data":"ea69cc77e0037d2568bfeba1dc3807420c1e11ad0e428d450749e2894d797009"}
Nov 01 03:47:14.409751 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:14.409334    4930 scope.go:122] "RemoveContainer" containerID="fef784a71930a0146fb4b7c723a394e75d36b6b8ad3b0c0098bfb7b7852460d5"
Nov 01 03:47:14.409751 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:14.409245    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c8373-755cc448b7-8hkdb"
Nov 01 03:47:14.630444 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:14.630409    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2c8373-966cf6b8c-8n27c"]
Nov 01 03:47:14.680931 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:14.680855    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2c8373-966cf6b8c-8n27c"]
Nov 01 03:47:14.789137 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:14.789084    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2c8373-755cc448b7-8hkdb"]
Nov 01 03:47:14.793266 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:14.793236    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2c8373-755cc448b7-8hkdb"]
Nov 01 03:47:14.853265 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:14.853226    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:47:15.621649 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:15.621612    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c031a873-378d-4081-845e-891f20c1354f" path="/var/lib/kubelet/pods/c031a873-378d-4081-845e-891f20c1354f/volumes"
Nov 01 03:47:15.622088 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:15.622073    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5dcc946-5a58-43bd-88c5-520f7ecb6104" path="/var/lib/kubelet/pods/f5dcc946-5a58-43bd-88c5-520f7ecb6104/volumes"
Nov 01 03:47:19.167577 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:47:19.157235    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:47:19.780540 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:19.780506    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:47:24.782692 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:24.782650    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:47:28.810520 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.810419    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8"]
Nov 01 03:47:28.811231 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.811206    4930 cpu_manager.go:414] "containerMap: removing container" podUID="830483ce-0cd5-45a5-a562-201d1ba81feb" containerName="revoke-keys"
Nov 01 03:47:28.811357 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.811232    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="830483ce-0cd5-45a5-a562-201d1ba81feb" containerName="revoke-keys"
Nov 01 03:47:28.811357 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.811258    4930 cpu_manager.go:414] "containerMap: removing container" podUID="c031a873-378d-4081-845e-891f20c1354f" containerName="payload-processing"
Nov 01 03:47:28.811357 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.811267    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c031a873-378d-4081-845e-891f20c1354f" containerName="payload-processing"
Nov 01 03:47:28.811357 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.811295    4930 cpu_manager.go:414] "containerMap: removing container" podUID="fd5ef13a-5e5e-45c5-89e9-fdf4d6899067" containerName="istio-proxy"
Nov 01 03:47:28.811357 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.811303    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd5ef13a-5e5e-45c5-89e9-fdf4d6899067" containerName="istio-proxy"
Nov 01 03:47:28.811357 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.811326    4930 cpu_manager.go:414] "containerMap: removing container" podUID="cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20" containerName="istio-proxy"
Nov 01 03:47:28.811357 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.811333    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20" containerName="istio-proxy"
Nov 01 03:47:28.811763 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.811361    4930 cpu_manager.go:414] "containerMap: removing container" podUID="ab72eb99-1ab5-4e1d-8057-82feb8b67a66" containerName="istio-proxy"
Nov 01 03:47:28.811763 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.811368    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab72eb99-1ab5-4e1d-8057-82feb8b67a66" containerName="istio-proxy"
Nov 01 03:47:28.811763 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.811383    4930 cpu_manager.go:414] "containerMap: removing container" podUID="f5dcc946-5a58-43bd-88c5-520f7ecb6104" containerName="payload-pre-processing"
Nov 01 03:47:28.811763 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.811391    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5dcc946-5a58-43bd-88c5-520f7ecb6104" containerName="payload-pre-processing"
Nov 01 03:47:28.811763 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.811570    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20" containerName="istio-proxy"
Nov 01 03:47:28.811763 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.811585    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab72eb99-1ab5-4e1d-8057-82feb8b67a66" containerName="istio-proxy"
Nov 01 03:47:28.811763 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.811595    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5dcc946-5a58-43bd-88c5-520f7ecb6104" containerName="payload-pre-processing"
Nov 01 03:47:28.811763 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.811610    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20" containerName="istio-proxy"
Nov 01 03:47:28.811763 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.811622    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="830483ce-0cd5-45a5-a562-201d1ba81feb" containerName="revoke-keys"
Nov 01 03:47:28.811763 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.811634    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="c031a873-378d-4081-845e-891f20c1354f" containerName="payload-processing"
Nov 01 03:47:28.811763 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.811647    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd5ef13a-5e5e-45c5-89e9-fdf4d6899067" containerName="istio-proxy"
Nov 01 03:47:28.816994 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.816972    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8"
Nov 01 03:47:28.861655 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.861622    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3626111a-ef96-41b1-9185-849ab00524cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-b97j8\" (UID: \"3626111a-ef96-41b1-9185-849ab00524cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8"
Nov 01 03:47:28.861815 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.861679    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3626111a-ef96-41b1-9185-849ab00524cc-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-b97j8\" (UID: \"3626111a-ef96-41b1-9185-849ab00524cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8"
Nov 01 03:47:28.861815 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.861741    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3626111a-ef96-41b1-9185-849ab00524cc-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-b97j8\" (UID: \"3626111a-ef96-41b1-9185-849ab00524cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8"
Nov 01 03:47:28.861815 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.861808    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3626111a-ef96-41b1-9185-849ab00524cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-b97j8\" (UID: \"3626111a-ef96-41b1-9185-849ab00524cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8"
Nov 01 03:47:28.861961 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.861857    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3626111a-ef96-41b1-9185-849ab00524cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-b97j8\" (UID: \"3626111a-ef96-41b1-9185-849ab00524cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8"
Nov 01 03:47:28.861961 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.861919    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t6lt\" (UniqueName: \"kubernetes.io/projected/3626111a-ef96-41b1-9185-849ab00524cc-kube-api-access-4t6lt\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-b97j8\" (UID: \"3626111a-ef96-41b1-9185-849ab00524cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8"
Nov 01 03:47:28.862077 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.861960    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3626111a-ef96-41b1-9185-849ab00524cc-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-b97j8\" (UID: \"3626111a-ef96-41b1-9185-849ab00524cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8"
Nov 01 03:47:28.862077 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.861982    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3626111a-ef96-41b1-9185-849ab00524cc-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-b97j8\" (UID: \"3626111a-ef96-41b1-9185-849ab00524cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8"
Nov 01 03:47:28.862184 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.862089    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3626111a-ef96-41b1-9185-849ab00524cc-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-b97j8\" (UID: \"3626111a-ef96-41b1-9185-849ab00524cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8"
Nov 01 03:47:28.962983 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.962910    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t6lt\" (UniqueName: \"kubernetes.io/projected/3626111a-ef96-41b1-9185-849ab00524cc-kube-api-access-4t6lt\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-b97j8\" (UID: \"3626111a-ef96-41b1-9185-849ab00524cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8"
Nov 01 03:47:28.963227 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.963018    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3626111a-ef96-41b1-9185-849ab00524cc-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-b97j8\" (UID: \"3626111a-ef96-41b1-9185-849ab00524cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8"
Nov 01 03:47:28.963227 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.963170    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3626111a-ef96-41b1-9185-849ab00524cc-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-b97j8\" (UID: \"3626111a-ef96-41b1-9185-849ab00524cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8"
Nov 01 03:47:28.963428 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.963248    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3626111a-ef96-41b1-9185-849ab00524cc-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-b97j8\" (UID: \"3626111a-ef96-41b1-9185-849ab00524cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8"
Nov 01 03:47:28.963428 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.963340    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3626111a-ef96-41b1-9185-849ab00524cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-b97j8\" (UID: \"3626111a-ef96-41b1-9185-849ab00524cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8"
Nov 01 03:47:28.963428 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.963406    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3626111a-ef96-41b1-9185-849ab00524cc-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-b97j8\" (UID: \"3626111a-ef96-41b1-9185-849ab00524cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8"
Nov 01 03:47:28.963658 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.963426    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3626111a-ef96-41b1-9185-849ab00524cc-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-b97j8\" (UID: \"3626111a-ef96-41b1-9185-849ab00524cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8"
Nov 01 03:47:28.963658 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.963435    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3626111a-ef96-41b1-9185-849ab00524cc-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-b97j8\" (UID: \"3626111a-ef96-41b1-9185-849ab00524cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8"
Nov 01 03:47:28.963658 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.963498    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3626111a-ef96-41b1-9185-849ab00524cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-b97j8\" (UID: \"3626111a-ef96-41b1-9185-849ab00524cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8"
Nov 01 03:47:28.963658 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.963583    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3626111a-ef96-41b1-9185-849ab00524cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-b97j8\" (UID: \"3626111a-ef96-41b1-9185-849ab00524cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8"
Nov 01 03:47:28.964324 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.964297    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3626111a-ef96-41b1-9185-849ab00524cc-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-b97j8\" (UID: \"3626111a-ef96-41b1-9185-849ab00524cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8"
Nov 01 03:47:28.964454 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.964417    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3626111a-ef96-41b1-9185-849ab00524cc-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-b97j8\" (UID: \"3626111a-ef96-41b1-9185-849ab00524cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8"
Nov 01 03:47:28.964575 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.964539    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3626111a-ef96-41b1-9185-849ab00524cc-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-b97j8\" (UID: \"3626111a-ef96-41b1-9185-849ab00524cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8"
Nov 01 03:47:28.965147 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.965121    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3626111a-ef96-41b1-9185-849ab00524cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-b97j8\" (UID: \"3626111a-ef96-41b1-9185-849ab00524cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8"
Nov 01 03:47:28.965995 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.965973    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3626111a-ef96-41b1-9185-849ab00524cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-b97j8\" (UID: \"3626111a-ef96-41b1-9185-849ab00524cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8"
Nov 01 03:47:28.967811 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:28.967784    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3626111a-ef96-41b1-9185-849ab00524cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-b97j8\" (UID: \"3626111a-ef96-41b1-9185-849ab00524cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8"
Nov 01 03:47:29.024646 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:29.024610    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3626111a-ef96-41b1-9185-849ab00524cc-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-b97j8\" (UID: \"3626111a-ef96-41b1-9185-849ab00524cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8"
Nov 01 03:47:29.026023 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:29.025998    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t6lt\" (UniqueName: \"kubernetes.io/projected/3626111a-ef96-41b1-9185-849ab00524cc-kube-api-access-4t6lt\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-b97j8\" (UID: \"3626111a-ef96-41b1-9185-849ab00524cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8"
Nov 01 03:47:29.032921 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:29.032893    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8"]
Nov 01 03:47:29.131971 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:29.131940    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8"
Nov 01 03:47:29.576759 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:29.576595    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8"]
Nov 01 03:47:29.576884 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:47:29.576803    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3626111a_ef96_41b1_9185_849ab00524cc.slice/crio-b09c3e127a1806514a797447fa130ebb2adeaf6de20ff85bbc33b1953f0fe611 WatchSource:0}: Error finding container b09c3e127a1806514a797447fa130ebb2adeaf6de20ff85bbc33b1953f0fe611: Status 404 returned error can't find the container with id b09c3e127a1806514a797447fa130ebb2adeaf6de20ff85bbc33b1953f0fe611
Nov 01 03:47:29.579685 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:29.579652    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:47:29.579803 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:29.579736    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:47:29.579872 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:29.579811    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:47:29.743828 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:29.743785    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:47:30.598847 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:30.598805    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8" event={"ID":"3626111a-ef96-41b1-9185-849ab00524cc","Type":"ContainerStarted","Data":"33560a8f32dd5c660dd0e3db16c1155d51b530e45403c649d4cfb246c8f17b70"}
Nov 01 03:47:30.599359 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:30.598880    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8" event={"ID":"3626111a-ef96-41b1-9185-849ab00524cc","Type":"ContainerStarted","Data":"b09c3e127a1806514a797447fa130ebb2adeaf6de20ff85bbc33b1953f0fe611"}
Nov 01 03:47:30.726435 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:30.726380    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8" podStartSLOduration=2.726366472 podStartE2EDuration="2.726366472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:47:30.725091484 +0000 UTC m=+2455.248098072" watchObservedRunningTime="2026-11-01 03:47:30.726366472 +0000 UTC m=+2455.249373059"
Nov 01 03:47:31.132518 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:31.132481    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8"
Nov 01 03:47:31.134499 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:31.134468    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-b97j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Nov 01 03:47:31.134683 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:31.134529    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8" podUID="3626111a-ef96-41b1-9185-849ab00524cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Nov 01 03:47:31.684226 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:31.681136    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:47:32.132465 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:32.132429    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-b97j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Nov 01 03:47:32.132714 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:32.132491    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8" podUID="3626111a-ef96-41b1-9185-849ab00524cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Nov 01 03:47:33.132910 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:33.132877    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-b97j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Nov 01 03:47:33.133323 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:33.132934    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8" podUID="3626111a-ef96-41b1-9185-849ab00524cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Nov 01 03:47:34.132657 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:34.132615    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-b97j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Nov 01 03:47:34.132859 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:34.132694    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8" podUID="3626111a-ef96-41b1-9185-849ab00524cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Nov 01 03:47:34.885131 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:34.885087    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:47:35.132298 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:35.132266    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-b97j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Nov 01 03:47:35.132485 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:35.132327    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8" podUID="3626111a-ef96-41b1-9185-849ab00524cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Nov 01 03:47:36.133044 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:36.133007    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-b97j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Nov 01 03:47:36.133452 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:36.133067    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8" podUID="3626111a-ef96-41b1-9185-849ab00524cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Nov 01 03:47:37.133125 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:37.133090    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-b97j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Nov 01 03:47:37.133546 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:37.133143    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8" podUID="3626111a-ef96-41b1-9185-849ab00524cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Nov 01 03:47:38.351674 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:38.351639    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8"
Nov 01 03:47:38.352219 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:38.352194    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8"
Nov 01 03:47:38.353269 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:38.353245    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8"
Nov 01 03:47:39.719890 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:39.719860    4930 scope.go:122] "RemoveContainer" containerID="d599992fc0cc58dfa484cc279f60a4bdf20647d98b723e5e9a7312f81ee2271d"
Nov 01 03:47:39.731513 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:39.731490    4930 scope.go:122] "RemoveContainer" containerID="a8c532ad7602ce7ca62182dcd377955be598b8bcc319f1dcf2184ecea74dd76c"
Nov 01 03:47:39.740833 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:39.740814    4930 scope.go:122] "RemoveContainer" containerID="de58bc85f6daad73d7be66a4261d110f37e0ad8df5ffa8e1e0ef5d26ec707018"
Nov 01 03:47:39.767203 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:39.767178    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:47:40.777337 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:40.777303    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:47:43.923000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:43.922967    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg"]
Nov 01 03:47:43.923551 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:43.923536    4930 cpu_manager.go:414] "containerMap: removing container" podUID="cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20" containerName="istio-proxy"
Nov 01 03:47:43.923551 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:43.923564    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb1aaf35-8c75-4c36-a2cd-193d7cf4fa20" containerName="istio-proxy"
Nov 01 03:47:43.928651 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:43.928633    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg"
Nov 01 03:47:44.034095 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:44.034062    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg"]
Nov 01 03:47:44.078295 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:44.078268    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cxhqg\" (UID: \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg"
Nov 01 03:47:44.078432 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:44.078315    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cxhqg\" (UID: \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg"
Nov 01 03:47:44.078432 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:44.078343    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cxhqg\" (UID: \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg"
Nov 01 03:47:44.078524 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:44.078460    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cxhqg\" (UID: \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg"
Nov 01 03:47:44.078524 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:44.078505    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cxhqg\" (UID: \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg"
Nov 01 03:47:44.078524 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:44.078521    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cxhqg\" (UID: \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg"
Nov 01 03:47:44.078685 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:44.078669    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlgg9\" (UniqueName: \"kubernetes.io/projected/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-kube-api-access-nlgg9\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cxhqg\" (UID: \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg"
Nov 01 03:47:44.078734 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:44.078700    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cxhqg\" (UID: \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg"
Nov 01 03:47:44.078816 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:44.078804    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cxhqg\" (UID: \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg"
Nov 01 03:47:44.180108 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:44.180040    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cxhqg\" (UID: \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg"
Nov 01 03:47:44.180108 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:44.180090    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cxhqg\" (UID: \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg"
Nov 01 03:47:44.180320 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:44.180118    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cxhqg\" (UID: \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg"
Nov 01 03:47:44.180320 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:44.180165    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlgg9\" (UniqueName: \"kubernetes.io/projected/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-kube-api-access-nlgg9\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cxhqg\" (UID: \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg"
Nov 01 03:47:44.180320 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:44.180200    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cxhqg\" (UID: \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg"
Nov 01 03:47:44.180320 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:44.180280    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cxhqg\" (UID: \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg"
Nov 01 03:47:44.180548 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:44.180369    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cxhqg\" (UID: \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg"
Nov 01 03:47:44.180548 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:44.180429    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cxhqg\" (UID: \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg"
Nov 01 03:47:44.180548 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:44.180468    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cxhqg\" (UID: \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg"
Nov 01 03:47:44.180740 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:44.180656    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cxhqg\" (UID: \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg"
Nov 01 03:47:44.180740 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:44.180731    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cxhqg\" (UID: \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg"
Nov 01 03:47:44.180887 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:44.180862    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cxhqg\" (UID: \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg"
Nov 01 03:47:44.181149 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:44.181090    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cxhqg\" (UID: \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg"
Nov 01 03:47:44.181278 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:44.181226    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cxhqg\" (UID: \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg"
Nov 01 03:47:44.182817 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:44.182799    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cxhqg\" (UID: \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg"
Nov 01 03:47:44.182932 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:44.182916    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cxhqg\" (UID: \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg"
Nov 01 03:47:44.240414 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:44.240384    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cxhqg\" (UID: \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg"
Nov 01 03:47:44.240598 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:44.240577    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlgg9\" (UniqueName: \"kubernetes.io/projected/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-kube-api-access-nlgg9\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cxhqg\" (UID: \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg"
Nov 01 03:47:44.246368 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:44.246351    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg"
Nov 01 03:47:44.641202 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:44.641166    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:47:44.676899 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:44.676872    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg"]
Nov 01 03:47:44.677406 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:47:44.677382    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a4480eb_b9d2_4f5c_8dbc_dbcbfe88671f.slice/crio-125558f2e1542a144031b7dd605c0d0d74d6c6c385d83037c9b8cc4e379efe9d WatchSource:0}: Error finding container 125558f2e1542a144031b7dd605c0d0d74d6c6c385d83037c9b8cc4e379efe9d: Status 404 returned error can't find the container with id 125558f2e1542a144031b7dd605c0d0d74d6c6c385d83037c9b8cc4e379efe9d
Nov 01 03:47:44.679955 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:44.679922    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:47:44.680050 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:44.679991    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:47:44.680050 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:44.680023    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:47:44.727990 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:44.727955    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg" event={"ID":"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f","Type":"ContainerStarted","Data":"125558f2e1542a144031b7dd605c0d0d74d6c6c385d83037c9b8cc4e379efe9d"}
Nov 01 03:47:45.736620 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:45.736581    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg" event={"ID":"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f","Type":"ContainerStarted","Data":"119476fa35f6c3aafce8b0fb2c85e01ae6bcf2fa23d5366588d75967cec3b506"}
Nov 01 03:47:45.906273 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:45.906215    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg" podStartSLOduration=2.906200373 podStartE2EDuration="2.906200373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:47:45.904535013 +0000 UTC m=+2470.427541601" watchObservedRunningTime="2026-11-01 03:47:45.906200373 +0000 UTC m=+2470.429206960"
Nov 01 03:47:46.246991 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:46.246956    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg"
Nov 01 03:47:46.248848 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:46.248820    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Nov 01 03:47:46.248982 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:46.248872    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg" podUID="1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Nov 01 03:47:47.247228 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:47.247146    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Nov 01 03:47:47.247228 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:47.247200    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg" podUID="1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Nov 01 03:47:48.247291 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:48.247258    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Nov 01 03:47:48.247728 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:48.247308    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg" podUID="1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Nov 01 03:47:49.123406 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:47:49.123368    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:47:49.247067 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:49.247040    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Nov 01 03:47:49.247265 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:49.247246    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg" podUID="1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Nov 01 03:47:49.409596 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:49.409500    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:47:50.217480 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:50.217451    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:47:50.247498 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:50.247468    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Nov 01 03:47:50.247709 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:50.247676    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg" podUID="1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Nov 01 03:47:51.247358 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:51.247318    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Nov 01 03:47:51.247890 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:51.247379    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg" podUID="1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Nov 01 03:47:52.247152 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:52.247112    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused" start-of-body=
Nov 01 03:47:52.247340 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:52.247179    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg" podUID="1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.235:15021/healthz/ready\": dial tcp 10.133.0.235:15021: connect: connection refused"
Nov 01 03:47:53.252149 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:53.252119    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg"
Nov 01 03:47:53.252619 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:53.252596    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg"
Nov 01 03:47:53.253209 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:53.253189    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg"
Nov 01 03:47:53.669999 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:53.669968    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:47:55.211976 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:55.211936    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:47:55.409475 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:55.409436    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:47:55.566620 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:55.566483    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:47:55.837130 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:55.837095    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c8cb9a-jt4cl"]
Nov 01 03:47:55.884966 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:55.884927    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c8cb9a-jt4cl"]
Nov 01 03:47:56.243602 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:56.243442    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:47:57.620674 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:47:57.620645    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b62f0069-2704-41a4-8da7-f5a5f8c77c94" path="/var/lib/kubelet/pods/b62f0069-2704-41a4-8da7-f5a5f8c77c94/volumes"
Nov 01 03:48:00.544887 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:00.544844    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:48:02.520750 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:02.520716    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:48:02.680365 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:02.680325    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:48:02.886278 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:02.886234    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:48:05.280234 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:05.280173    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:48:05.714440 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:05.714391    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:48:07.609669 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:07.609633    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fd6cf-fct5g"]
Nov 01 03:48:07.613767 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:07.613740    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fd6cf-fct5g"
Nov 01 03:48:07.629478 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:07.629443    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brbt5\" (UniqueName: \"kubernetes.io/projected/8cfeed06-fada-48eb-956b-317138fa24fa-kube-api-access-brbt5\") pod \"maas-api-revoke-keys-e2e-api-a-5fd6cf-fct5g\" (UID: \"8cfeed06-fada-48eb-956b-317138fa24fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fd6cf-fct5g"
Nov 01 03:48:07.629648 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:07.629595    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cfeed06-fada-48eb-956b-317138fa24fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5fd6cf-fct5g\" (UID: \"8cfeed06-fada-48eb-956b-317138fa24fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fd6cf-fct5g"
Nov 01 03:48:07.629726 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:07.629696    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cfeed06-fada-48eb-956b-317138fa24fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5fd6cf-fct5g\" (UID: \"8cfeed06-fada-48eb-956b-317138fa24fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fd6cf-fct5g"
Nov 01 03:48:07.715778 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:07.715747    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:48:07.715778 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:07.715757    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:48:07.716002 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:07.715799    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:48:07.716002 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:07.715888    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:48:07.721775 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:07.721751    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fd6cf-fct5g"]
Nov 01 03:48:07.730803 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:07.730774    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cfeed06-fada-48eb-956b-317138fa24fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5fd6cf-fct5g\" (UID: \"8cfeed06-fada-48eb-956b-317138fa24fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fd6cf-fct5g"
Nov 01 03:48:07.730938 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:07.730916    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brbt5\" (UniqueName: \"kubernetes.io/projected/8cfeed06-fada-48eb-956b-317138fa24fa-kube-api-access-brbt5\") pod \"maas-api-revoke-keys-e2e-api-a-5fd6cf-fct5g\" (UID: \"8cfeed06-fada-48eb-956b-317138fa24fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fd6cf-fct5g"
Nov 01 03:48:07.731030 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:07.731014    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cfeed06-fada-48eb-956b-317138fa24fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5fd6cf-fct5g\" (UID: \"8cfeed06-fada-48eb-956b-317138fa24fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fd6cf-fct5g"
Nov 01 03:48:07.731705 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:07.731686    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cfeed06-fada-48eb-956b-317138fa24fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5fd6cf-fct5g\" (UID: \"8cfeed06-fada-48eb-956b-317138fa24fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fd6cf-fct5g"
Nov 01 03:48:07.733175 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:07.733154    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cfeed06-fada-48eb-956b-317138fa24fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5fd6cf-fct5g\" (UID: \"8cfeed06-fada-48eb-956b-317138fa24fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fd6cf-fct5g"
Nov 01 03:48:07.791851 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:07.791827    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brbt5\" (UniqueName: \"kubernetes.io/projected/8cfeed06-fada-48eb-956b-317138fa24fa-kube-api-access-brbt5\") pod \"maas-api-revoke-keys-e2e-api-a-5fd6cf-fct5g\" (UID: \"8cfeed06-fada-48eb-956b-317138fa24fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fd6cf-fct5g"
Nov 01 03:48:07.936315 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:07.936243    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fd6cf-fct5g"
Nov 01 03:48:08.326950 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:08.326448    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fd6cf-fct5g"]
Nov 01 03:48:09.070312 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:09.070225    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fd6cf-fct5g" event={"ID":"8cfeed06-fada-48eb-956b-317138fa24fa","Type":"ContainerStarted","Data":"099dcc73f336a57c40cc71a67c8580274f1490365daa79079118cba1c375caea"}
Nov 01 03:48:09.070312 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:09.070290    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fd6cf-fct5g" event={"ID":"8cfeed06-fada-48eb-956b-317138fa24fa","Type":"ContainerStarted","Data":"972078e73952856cd34927639d1a618622dceddefac990274ef98f00475cdc59"}
Nov 01 03:48:10.080114 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:10.080073    4930 generic.go:356] "Generic (PLEG): container finished" podID="8cfeed06-fada-48eb-956b-317138fa24fa" containerID="099dcc73f336a57c40cc71a67c8580274f1490365daa79079118cba1c375caea" exitCode=0
Nov 01 03:48:10.080529 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:10.080147    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fd6cf-fct5g" event={"ID":"8cfeed06-fada-48eb-956b-317138fa24fa","Type":"ContainerDied","Data":"099dcc73f336a57c40cc71a67c8580274f1490365daa79079118cba1c375caea"}
Nov 01 03:48:10.856260 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:10.856212    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:48:11.285895 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:11.285863    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fd6cf-fct5g"
Nov 01 03:48:11.295945 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:11.295917    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cfeed06-fada-48eb-956b-317138fa24fa-kube-api-access-brbt5\" (UniqueName: \"kubernetes.io/projected/8cfeed06-fada-48eb-956b-317138fa24fa-kube-api-access-brbt5\") pod \"8cfeed06-fada-48eb-956b-317138fa24fa\" (UID: \"8cfeed06-fada-48eb-956b-317138fa24fa\") "
Nov 01 03:48:11.296095 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:11.295955    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cfeed06-fada-48eb-956b-317138fa24fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cfeed06-fada-48eb-956b-317138fa24fa-maas-api-openshift-ca\") pod \"8cfeed06-fada-48eb-956b-317138fa24fa\" (UID: \"8cfeed06-fada-48eb-956b-317138fa24fa\") "
Nov 01 03:48:11.296095 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:11.296047    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cfeed06-fada-48eb-956b-317138fa24fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cfeed06-fada-48eb-956b-317138fa24fa-maas-api-xks-ca\") pod \"8cfeed06-fada-48eb-956b-317138fa24fa\" (UID: \"8cfeed06-fada-48eb-956b-317138fa24fa\") "
Nov 01 03:48:11.296404 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:11.296377    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cfeed06-fada-48eb-956b-317138fa24fa-maas-api-openshift-ca" pod "8cfeed06-fada-48eb-956b-317138fa24fa" (UID: "8cfeed06-fada-48eb-956b-317138fa24fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:48:11.296551 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:11.296536    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cfeed06-fada-48eb-956b-317138fa24fa-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:48:11.298097 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:11.298044    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cfeed06-fada-48eb-956b-317138fa24fa-maas-api-xks-ca" pod "8cfeed06-fada-48eb-956b-317138fa24fa" (UID: "8cfeed06-fada-48eb-956b-317138fa24fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:48:11.298188 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:11.298090    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cfeed06-fada-48eb-956b-317138fa24fa-kube-api-access-brbt5" pod "8cfeed06-fada-48eb-956b-317138fa24fa" (UID: "8cfeed06-fada-48eb-956b-317138fa24fa"). InnerVolumeSpecName "kube-api-access-brbt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:48:11.398113 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:11.398018    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brbt5\" (UniqueName: \"kubernetes.io/projected/8cfeed06-fada-48eb-956b-317138fa24fa-kube-api-access-brbt5\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:48:11.398113 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:11.398052    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cfeed06-fada-48eb-956b-317138fa24fa-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:48:12.098661 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:12.098630    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fd6cf-fct5g"
Nov 01 03:48:12.098842 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:12.098664    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fd6cf-fct5g" event={"ID":"8cfeed06-fada-48eb-956b-317138fa24fa","Type":"ContainerDied","Data":"972078e73952856cd34927639d1a618622dceddefac990274ef98f00475cdc59"}
Nov 01 03:48:12.098842 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:12.098737    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="972078e73952856cd34927639d1a618622dceddefac990274ef98f00475cdc59"
Nov 01 03:48:16.090820 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:16.090744    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:48:19.122698 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:48:19.122661    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:48:19.457117 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:19.457029    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:48:19.608305 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:19.608228    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:48:19.695859 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:19.695823    4930 patch_prober.go:29] interesting pod/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": read tcp 10.133.0.2:48602->10.133.0.228:15021: read: connection reset by peer" start-of-body=
Nov 01 03:48:19.696013 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:19.695976    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq" podUID="df136678-b446-4fe6-8705-48d36455f70a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": read tcp 10.133.0.2:48602->10.133.0.228:15021: read: connection reset by peer"
Nov 01 03:48:20.193450 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:20.193411    4930 generic.go:356] "Generic (PLEG): container finished" podID="df136678-b446-4fe6-8705-48d36455f70a" containerID="3e880d4951c987dca9006bff08e898da20ab4f7766a79e22143fe0e1331e9341" exitCode=0
Nov 01 03:48:20.193856 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:20.193488    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq" event={"ID":"df136678-b446-4fe6-8705-48d36455f70a","Type":"ContainerDied","Data":"3e880d4951c987dca9006bff08e898da20ab4f7766a79e22143fe0e1331e9341"}
Nov 01 03:48:20.194038 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:20.194018    4930 scope.go:122] "RemoveContainer" containerID="3e880d4951c987dca9006bff08e898da20ab4f7766a79e22143fe0e1331e9341"
Nov 01 03:48:20.195830 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:20.195801    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:48:20.195931 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:20.195871    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:48:20.195931 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:20.195912    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:48:21.198230 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:21.198171    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfd0f261-4s25s"]
Nov 01 03:48:21.199133 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:21.199111    4930 cpu_manager.go:414] "containerMap: removing container" podUID="8cfeed06-fada-48eb-956b-317138fa24fa" containerName="revoke-keys"
Nov 01 03:48:21.199207 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:21.199136    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cfeed06-fada-48eb-956b-317138fa24fa" containerName="revoke-keys"
Nov 01 03:48:21.199330 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:21.199318    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cfeed06-fada-48eb-956b-317138fa24fa" containerName="revoke-keys"
Nov 01 03:48:21.203654 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:21.203630    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd0f261-4s25s"
Nov 01 03:48:21.209362 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:21.209333    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq" event={"ID":"df136678-b446-4fe6-8705-48d36455f70a","Type":"ContainerStarted","Data":"60d8d43549422f0cf244a102fd44e6d5b3f36f78d8dde4d60b828c9761abc4d2"}
Nov 01 03:48:21.222757 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:21.222730    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:48:21.256138 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:21.256117    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:48:21.347323 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:21.304749    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:48:21.347323 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:21.304855    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:48:21.347323 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:21.306686    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:48:21.347323 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:21.337623    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfd0f261-4s25s"]
Nov 01 03:48:21.395587 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:21.394925    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96f9h\" (UniqueName: \"kubernetes.io/projected/28d34d07-bfbd-4231-869b-e7ebf0519d02-kube-api-access-96f9h\") pod \"maas-api-revoke-sub-bfd0f261-4s25s\" (UID: \"28d34d07-bfbd-4231-869b-e7ebf0519d02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd0f261-4s25s"
Nov 01 03:48:21.395587 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:21.395124    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28d34d07-bfbd-4231-869b-e7ebf0519d02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfd0f261-4s25s\" (UID: \"28d34d07-bfbd-4231-869b-e7ebf0519d02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd0f261-4s25s"
Nov 01 03:48:21.395587 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:21.395257    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28d34d07-bfbd-4231-869b-e7ebf0519d02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfd0f261-4s25s\" (UID: \"28d34d07-bfbd-4231-869b-e7ebf0519d02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd0f261-4s25s"
Nov 01 03:48:21.496700 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:21.496589    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96f9h\" (UniqueName: \"kubernetes.io/projected/28d34d07-bfbd-4231-869b-e7ebf0519d02-kube-api-access-96f9h\") pod \"maas-api-revoke-sub-bfd0f261-4s25s\" (UID: \"28d34d07-bfbd-4231-869b-e7ebf0519d02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd0f261-4s25s"
Nov 01 03:48:21.497065 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:21.497036    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28d34d07-bfbd-4231-869b-e7ebf0519d02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfd0f261-4s25s\" (UID: \"28d34d07-bfbd-4231-869b-e7ebf0519d02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd0f261-4s25s"
Nov 01 03:48:21.497184 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:21.497162    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28d34d07-bfbd-4231-869b-e7ebf0519d02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfd0f261-4s25s\" (UID: \"28d34d07-bfbd-4231-869b-e7ebf0519d02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd0f261-4s25s"
Nov 01 03:48:21.497975 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:21.497945    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28d34d07-bfbd-4231-869b-e7ebf0519d02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfd0f261-4s25s\" (UID: \"28d34d07-bfbd-4231-869b-e7ebf0519d02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd0f261-4s25s"
Nov 01 03:48:21.499876 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:21.499848    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28d34d07-bfbd-4231-869b-e7ebf0519d02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfd0f261-4s25s\" (UID: \"28d34d07-bfbd-4231-869b-e7ebf0519d02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd0f261-4s25s"
Nov 01 03:48:21.557787 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:21.557746    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96f9h\" (UniqueName: \"kubernetes.io/projected/28d34d07-bfbd-4231-869b-e7ebf0519d02-kube-api-access-96f9h\") pod \"maas-api-revoke-sub-bfd0f261-4s25s\" (UID: \"28d34d07-bfbd-4231-869b-e7ebf0519d02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd0f261-4s25s"
Nov 01 03:48:21.819938 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:21.819850    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd0f261-4s25s"
Nov 01 03:48:22.038872 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:22.038838    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"
Nov 01 03:48:22.039056 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:22.039034    4930 patch_prober.go:29] interesting pod/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Nov 01 03:48:22.039103 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:22.039081    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq" podUID="df136678-b446-4fe6-8705-48d36455f70a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Nov 01 03:48:22.239656 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:22.239629    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfd0f261-4s25s"]
Nov 01 03:48:22.240403 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:48:22.240379    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28d34d07_bfbd_4231_869b_e7ebf0519d02.slice/crio-181758c3c26f17805fe53e565607fcb8afc04ad51cd02f89f0a2ed0bab460343 WatchSource:0}: Error finding container 181758c3c26f17805fe53e565607fcb8afc04ad51cd02f89f0a2ed0bab460343: Status 404 returned error can't find the container with id 181758c3c26f17805fe53e565607fcb8afc04ad51cd02f89f0a2ed0bab460343
Nov 01 03:48:23.038684 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:23.038643    4930 patch_prober.go:29] interesting pod/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Nov 01 03:48:23.038876 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:23.038709    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq" podUID="df136678-b446-4fe6-8705-48d36455f70a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Nov 01 03:48:23.227806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:23.227769    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd0f261-4s25s" event={"ID":"28d34d07-bfbd-4231-869b-e7ebf0519d02","Type":"ContainerStarted","Data":"bd921bea64c3d77a46fa8973e88ace2ccb55e7588411d36ac93e9c4749c841fa"}
Nov 01 03:48:23.227966 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:23.227827    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd0f261-4s25s" event={"ID":"28d34d07-bfbd-4231-869b-e7ebf0519d02","Type":"ContainerStarted","Data":"181758c3c26f17805fe53e565607fcb8afc04ad51cd02f89f0a2ed0bab460343"}
Nov 01 03:48:23.350047 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:23.349991    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd0f261-4s25s" podStartSLOduration=2.349975835 podStartE2EDuration="2.349975835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:48:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:48:23.347301992 +0000 UTC m=+2507.870308580" watchObservedRunningTime="2026-11-01 03:48:23.349975835 +0000 UTC m=+2507.872982469"
Nov 01 03:48:24.038235 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:24.038199    4930 patch_prober.go:29] interesting pod/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Nov 01 03:48:24.038428 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:24.038253    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq" podUID="df136678-b446-4fe6-8705-48d36455f70a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Nov 01 03:48:24.237432 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:24.237396    4930 generic.go:356] "Generic (PLEG): container finished" podID="28d34d07-bfbd-4231-869b-e7ebf0519d02" containerID="bd921bea64c3d77a46fa8973e88ace2ccb55e7588411d36ac93e9c4749c841fa" exitCode=0
Nov 01 03:48:24.237637 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:24.237493    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd0f261-4s25s" event={"ID":"28d34d07-bfbd-4231-869b-e7ebf0519d02","Type":"ContainerDied","Data":"bd921bea64c3d77a46fa8973e88ace2ccb55e7588411d36ac93e9c4749c841fa"}
Nov 01 03:48:25.038891 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:25.038857    4930 patch_prober.go:29] interesting pod/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Nov 01 03:48:25.039303 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:25.038921    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq" podUID="df136678-b446-4fe6-8705-48d36455f70a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Nov 01 03:48:25.553822 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:25.553794    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd0f261-4s25s"
Nov 01 03:48:25.593051 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:25.593007    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28d34d07-bfbd-4231-869b-e7ebf0519d02-kube-api-access-96f9h\" (UniqueName: \"kubernetes.io/projected/28d34d07-bfbd-4231-869b-e7ebf0519d02-kube-api-access-96f9h\") pod \"28d34d07-bfbd-4231-869b-e7ebf0519d02\" (UID: \"28d34d07-bfbd-4231-869b-e7ebf0519d02\") "
Nov 01 03:48:25.593233 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:25.593063    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28d34d07-bfbd-4231-869b-e7ebf0519d02-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28d34d07-bfbd-4231-869b-e7ebf0519d02-maas-api-xks-ca\") pod \"28d34d07-bfbd-4231-869b-e7ebf0519d02\" (UID: \"28d34d07-bfbd-4231-869b-e7ebf0519d02\") "
Nov 01 03:48:25.593331 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:25.593315    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28d34d07-bfbd-4231-869b-e7ebf0519d02-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28d34d07-bfbd-4231-869b-e7ebf0519d02-maas-api-openshift-ca\") pod \"28d34d07-bfbd-4231-869b-e7ebf0519d02\" (UID: \"28d34d07-bfbd-4231-869b-e7ebf0519d02\") "
Nov 01 03:48:25.593839 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:25.593806    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28d34d07-bfbd-4231-869b-e7ebf0519d02-maas-api-openshift-ca" pod "28d34d07-bfbd-4231-869b-e7ebf0519d02" (UID: "28d34d07-bfbd-4231-869b-e7ebf0519d02"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:48:25.595274 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:25.595245    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28d34d07-bfbd-4231-869b-e7ebf0519d02-kube-api-access-96f9h" pod "28d34d07-bfbd-4231-869b-e7ebf0519d02" (UID: "28d34d07-bfbd-4231-869b-e7ebf0519d02"). InnerVolumeSpecName "kube-api-access-96f9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:48:25.595379 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:25.595303    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28d34d07-bfbd-4231-869b-e7ebf0519d02-maas-api-xks-ca" pod "28d34d07-bfbd-4231-869b-e7ebf0519d02" (UID: "28d34d07-bfbd-4231-869b-e7ebf0519d02"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:48:25.694766 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:25.694720    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28d34d07-bfbd-4231-869b-e7ebf0519d02-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:48:25.694766 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:25.694761    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96f9h\" (UniqueName: \"kubernetes.io/projected/28d34d07-bfbd-4231-869b-e7ebf0519d02-kube-api-access-96f9h\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:48:25.694766 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:25.694771    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28d34d07-bfbd-4231-869b-e7ebf0519d02-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:48:26.039349 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:26.039255    4930 patch_prober.go:29] interesting pod/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Nov 01 03:48:26.039349 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:26.039323    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq" podUID="df136678-b446-4fe6-8705-48d36455f70a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Nov 01 03:48:26.055623 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:26.055586    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:48:26.253708 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:26.253681    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd0f261-4s25s"
Nov 01 03:48:26.253884 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:26.253717    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd0f261-4s25s" event={"ID":"28d34d07-bfbd-4231-869b-e7ebf0519d02","Type":"ContainerDied","Data":"181758c3c26f17805fe53e565607fcb8afc04ad51cd02f89f0a2ed0bab460343"}
Nov 01 03:48:26.253884 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:26.253764    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="181758c3c26f17805fe53e565607fcb8afc04ad51cd02f89f0a2ed0bab460343"
Nov 01 03:48:27.042520 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:27.042492    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"
Nov 01 03:48:27.043118 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:27.043035    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"
Nov 01 03:48:27.043672 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:27.043655    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"
Nov 01 03:48:29.870569 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:29.870504    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:48:31.159130 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:31.159098    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:48:31.611214 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:31.611180    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2906cb-c788n"]
Nov 01 03:48:31.612145 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:31.612125    4930 cpu_manager.go:414] "containerMap: removing container" podUID="28d34d07-bfbd-4231-869b-e7ebf0519d02" containerName="revoke-keys"
Nov 01 03:48:31.612222 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:31.612148    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28d34d07-bfbd-4231-869b-e7ebf0519d02" containerName="revoke-keys"
Nov 01 03:48:31.612318 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:31.612306    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="28d34d07-bfbd-4231-869b-e7ebf0519d02" containerName="revoke-keys"
Nov 01 03:48:31.616627 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:31.616607    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2906cb-c788n"
Nov 01 03:48:31.673366 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:31.673341    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2906cb-c788n"]
Nov 01 03:48:31.678123 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:31.678095    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drcr5\" (UniqueName: \"kubernetes.io/projected/e552bc78-982d-4ce6-97c1-693fbdfa46e5-kube-api-access-drcr5\") pod \"maas-api-revoke-keys-e2e-api-a-2906cb-c788n\" (UID: \"e552bc78-982d-4ce6-97c1-693fbdfa46e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2906cb-c788n"
Nov 01 03:48:31.678315 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:31.678287    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e552bc78-982d-4ce6-97c1-693fbdfa46e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2906cb-c788n\" (UID: \"e552bc78-982d-4ce6-97c1-693fbdfa46e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2906cb-c788n"
Nov 01 03:48:31.678451 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:31.678321    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e552bc78-982d-4ce6-97c1-693fbdfa46e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2906cb-c788n\" (UID: \"e552bc78-982d-4ce6-97c1-693fbdfa46e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2906cb-c788n"
Nov 01 03:48:31.717804 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:31.717778    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:48:31.717954 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:31.717805    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:48:31.718003 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:31.717957    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:48:31.718513 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:31.718492    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:48:31.779785 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:31.779757    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e552bc78-982d-4ce6-97c1-693fbdfa46e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2906cb-c788n\" (UID: \"e552bc78-982d-4ce6-97c1-693fbdfa46e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2906cb-c788n"
Nov 01 03:48:31.779919 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:31.779791    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e552bc78-982d-4ce6-97c1-693fbdfa46e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2906cb-c788n\" (UID: \"e552bc78-982d-4ce6-97c1-693fbdfa46e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2906cb-c788n"
Nov 01 03:48:31.779919 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:31.779875    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drcr5\" (UniqueName: \"kubernetes.io/projected/e552bc78-982d-4ce6-97c1-693fbdfa46e5-kube-api-access-drcr5\") pod \"maas-api-revoke-keys-e2e-api-a-2906cb-c788n\" (UID: \"e552bc78-982d-4ce6-97c1-693fbdfa46e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2906cb-c788n"
Nov 01 03:48:31.780510 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:31.780489    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e552bc78-982d-4ce6-97c1-693fbdfa46e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2906cb-c788n\" (UID: \"e552bc78-982d-4ce6-97c1-693fbdfa46e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2906cb-c788n"
Nov 01 03:48:31.782630 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:31.782603    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e552bc78-982d-4ce6-97c1-693fbdfa46e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2906cb-c788n\" (UID: \"e552bc78-982d-4ce6-97c1-693fbdfa46e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2906cb-c788n"
Nov 01 03:48:31.840879 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:31.840855    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drcr5\" (UniqueName: \"kubernetes.io/projected/e552bc78-982d-4ce6-97c1-693fbdfa46e5-kube-api-access-drcr5\") pod \"maas-api-revoke-keys-e2e-api-a-2906cb-c788n\" (UID: \"e552bc78-982d-4ce6-97c1-693fbdfa46e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2906cb-c788n"
Nov 01 03:48:31.931057 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:31.930996    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2906cb-c788n"
Nov 01 03:48:32.006092 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:32.006057    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:48:32.312145 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:48:32.312114    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode552bc78_982d_4ce6_97c1_693fbdfa46e5.slice/crio-ca230f7209daf3acfd7237e9fda196f15d132670a5975925bdfb08e99dae3d06 WatchSource:0}: Error finding container ca230f7209daf3acfd7237e9fda196f15d132670a5975925bdfb08e99dae3d06: Status 404 returned error can't find the container with id ca230f7209daf3acfd7237e9fda196f15d132670a5975925bdfb08e99dae3d06
Nov 01 03:48:32.313848 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:32.313819    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2906cb-c788n"]
Nov 01 03:48:32.582285 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:32.582257    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d7f535-g7b2w"]
Nov 01 03:48:32.586143 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:32.586126    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d7f535-g7b2w"
Nov 01 03:48:32.647789 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:32.647762    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d7f535-g7b2w"]
Nov 01 03:48:32.692529 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:32.692502    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae793f09-0221-462d-b200-c02136887efe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-d7f535-g7b2w\" (UID: \"ae793f09-0221-462d-b200-c02136887efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d7f535-g7b2w"
Nov 01 03:48:32.692947 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:32.692927    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae793f09-0221-462d-b200-c02136887efe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-d7f535-g7b2w\" (UID: \"ae793f09-0221-462d-b200-c02136887efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d7f535-g7b2w"
Nov 01 03:48:32.693042 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:32.692984    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmgfv\" (UniqueName: \"kubernetes.io/projected/ae793f09-0221-462d-b200-c02136887efe-kube-api-access-fmgfv\") pod \"maas-api-revoke-keys-e2e-worker-w3-d7f535-g7b2w\" (UID: \"ae793f09-0221-462d-b200-c02136887efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d7f535-g7b2w"
Nov 01 03:48:32.793949 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:32.793914    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae793f09-0221-462d-b200-c02136887efe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-d7f535-g7b2w\" (UID: \"ae793f09-0221-462d-b200-c02136887efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d7f535-g7b2w"
Nov 01 03:48:32.794110 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:32.793988    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmgfv\" (UniqueName: \"kubernetes.io/projected/ae793f09-0221-462d-b200-c02136887efe-kube-api-access-fmgfv\") pod \"maas-api-revoke-keys-e2e-worker-w3-d7f535-g7b2w\" (UID: \"ae793f09-0221-462d-b200-c02136887efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d7f535-g7b2w"
Nov 01 03:48:32.794110 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:32.794034    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae793f09-0221-462d-b200-c02136887efe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-d7f535-g7b2w\" (UID: \"ae793f09-0221-462d-b200-c02136887efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d7f535-g7b2w"
Nov 01 03:48:32.794685 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:32.794660    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae793f09-0221-462d-b200-c02136887efe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-d7f535-g7b2w\" (UID: \"ae793f09-0221-462d-b200-c02136887efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d7f535-g7b2w"
Nov 01 03:48:32.796471 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:32.796453    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae793f09-0221-462d-b200-c02136887efe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-d7f535-g7b2w\" (UID: \"ae793f09-0221-462d-b200-c02136887efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d7f535-g7b2w"
Nov 01 03:48:32.855690 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:32.855664    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmgfv\" (UniqueName: \"kubernetes.io/projected/ae793f09-0221-462d-b200-c02136887efe-kube-api-access-fmgfv\") pod \"maas-api-revoke-keys-e2e-worker-w3-d7f535-g7b2w\" (UID: \"ae793f09-0221-462d-b200-c02136887efe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d7f535-g7b2w"
Nov 01 03:48:32.896476 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:32.896441    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:48:32.900319 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:32.900297    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d7f535-g7b2w"
Nov 01 03:48:33.046037 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:33.046002    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:48:33.279410 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:48:33.279383    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae793f09_0221_462d_b200_c02136887efe.slice/crio-096b66cb1f63fe2ff2f9eaac89420afb2bbc015d510c0dcb0d3626377e0e6f53 WatchSource:0}: Error finding container 096b66cb1f63fe2ff2f9eaac89420afb2bbc015d510c0dcb0d3626377e0e6f53: Status 404 returned error can't find the container with id 096b66cb1f63fe2ff2f9eaac89420afb2bbc015d510c0dcb0d3626377e0e6f53
Nov 01 03:48:33.281143 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:33.281114    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d7f535-g7b2w"]
Nov 01 03:48:33.313011 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:33.312982    4930 generic.go:356] "Generic (PLEG): container finished" podID="e552bc78-982d-4ce6-97c1-693fbdfa46e5" containerID="9ab10151b42588d9660052bfb3f5db2315b7b1e5963dfce614fa2434e58caad9" exitCode=0
Nov 01 03:48:33.313430 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:33.313065    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2906cb-c788n" event={"ID":"e552bc78-982d-4ce6-97c1-693fbdfa46e5","Type":"ContainerDied","Data":"9ab10151b42588d9660052bfb3f5db2315b7b1e5963dfce614fa2434e58caad9"}
Nov 01 03:48:33.313430 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:33.313116    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2906cb-c788n" event={"ID":"e552bc78-982d-4ce6-97c1-693fbdfa46e5","Type":"ContainerStarted","Data":"ca230f7209daf3acfd7237e9fda196f15d132670a5975925bdfb08e99dae3d06"}
Nov 01 03:48:33.314624 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:33.314537    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d7f535-g7b2w" event={"ID":"ae793f09-0221-462d-b200-c02136887efe","Type":"ContainerStarted","Data":"096b66cb1f63fe2ff2f9eaac89420afb2bbc015d510c0dcb0d3626377e0e6f53"}
Nov 01 03:48:34.323397 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:34.323363    4930 generic.go:356] "Generic (PLEG): container finished" podID="ae793f09-0221-462d-b200-c02136887efe" containerID="60668cc1f4f36b471432208b7e237f66602c76edb4c74ff4ad14a81189f405b0" exitCode=0
Nov 01 03:48:34.323824 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:34.323442    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d7f535-g7b2w" event={"ID":"ae793f09-0221-462d-b200-c02136887efe","Type":"ContainerDied","Data":"60668cc1f4f36b471432208b7e237f66602c76edb4c74ff4ad14a81189f405b0"}
Nov 01 03:48:34.493880 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:34.493852    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2906cb-c788n"
Nov 01 03:48:34.617683 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:34.617649    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e552bc78-982d-4ce6-97c1-693fbdfa46e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e552bc78-982d-4ce6-97c1-693fbdfa46e5-maas-api-xks-ca\") pod \"e552bc78-982d-4ce6-97c1-693fbdfa46e5\" (UID: \"e552bc78-982d-4ce6-97c1-693fbdfa46e5\") "
Nov 01 03:48:34.617878 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:34.617703    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e552bc78-982d-4ce6-97c1-693fbdfa46e5-kube-api-access-drcr5\" (UniqueName: \"kubernetes.io/projected/e552bc78-982d-4ce6-97c1-693fbdfa46e5-kube-api-access-drcr5\") pod \"e552bc78-982d-4ce6-97c1-693fbdfa46e5\" (UID: \"e552bc78-982d-4ce6-97c1-693fbdfa46e5\") "
Nov 01 03:48:34.617878 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:34.617787    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e552bc78-982d-4ce6-97c1-693fbdfa46e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e552bc78-982d-4ce6-97c1-693fbdfa46e5-maas-api-openshift-ca\") pod \"e552bc78-982d-4ce6-97c1-693fbdfa46e5\" (UID: \"e552bc78-982d-4ce6-97c1-693fbdfa46e5\") "
Nov 01 03:48:34.618812 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:34.618780    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e552bc78-982d-4ce6-97c1-693fbdfa46e5-maas-api-openshift-ca" pod "e552bc78-982d-4ce6-97c1-693fbdfa46e5" (UID: "e552bc78-982d-4ce6-97c1-693fbdfa46e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:48:34.620703 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:34.620674    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e552bc78-982d-4ce6-97c1-693fbdfa46e5-kube-api-access-drcr5" pod "e552bc78-982d-4ce6-97c1-693fbdfa46e5" (UID: "e552bc78-982d-4ce6-97c1-693fbdfa46e5"). InnerVolumeSpecName "kube-api-access-drcr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:48:34.621225 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:34.621199    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e552bc78-982d-4ce6-97c1-693fbdfa46e5-maas-api-xks-ca" pod "e552bc78-982d-4ce6-97c1-693fbdfa46e5" (UID: "e552bc78-982d-4ce6-97c1-693fbdfa46e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:48:34.719577 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:34.719480    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e552bc78-982d-4ce6-97c1-693fbdfa46e5-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:48:34.719577 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:34.719512    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drcr5\" (UniqueName: \"kubernetes.io/projected/e552bc78-982d-4ce6-97c1-693fbdfa46e5-kube-api-access-drcr5\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:48:34.719577 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:34.719523    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e552bc78-982d-4ce6-97c1-693fbdfa46e5-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:48:35.332316 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:35.332289    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2906cb-c788n"
Nov 01 03:48:35.332766 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:35.332329    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2906cb-c788n" event={"ID":"e552bc78-982d-4ce6-97c1-693fbdfa46e5","Type":"ContainerDied","Data":"ca230f7209daf3acfd7237e9fda196f15d132670a5975925bdfb08e99dae3d06"}
Nov 01 03:48:35.332766 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:35.332379    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca230f7209daf3acfd7237e9fda196f15d132670a5975925bdfb08e99dae3d06"
Nov 01 03:48:35.502538 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:35.502514    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d7f535-g7b2w"
Nov 01 03:48:35.630739 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:35.630702    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae793f09-0221-462d-b200-c02136887efe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae793f09-0221-462d-b200-c02136887efe-maas-api-xks-ca\") pod \"ae793f09-0221-462d-b200-c02136887efe\" (UID: \"ae793f09-0221-462d-b200-c02136887efe\") "
Nov 01 03:48:35.630892 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:35.630802    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae793f09-0221-462d-b200-c02136887efe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae793f09-0221-462d-b200-c02136887efe-maas-api-openshift-ca\") pod \"ae793f09-0221-462d-b200-c02136887efe\" (UID: \"ae793f09-0221-462d-b200-c02136887efe\") "
Nov 01 03:48:35.631004 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:35.630986    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae793f09-0221-462d-b200-c02136887efe-kube-api-access-fmgfv\" (UniqueName: \"kubernetes.io/projected/ae793f09-0221-462d-b200-c02136887efe-kube-api-access-fmgfv\") pod \"ae793f09-0221-462d-b200-c02136887efe\" (UID: \"ae793f09-0221-462d-b200-c02136887efe\") "
Nov 01 03:48:35.631206 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:35.631179    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae793f09-0221-462d-b200-c02136887efe-maas-api-openshift-ca" pod "ae793f09-0221-462d-b200-c02136887efe" (UID: "ae793f09-0221-462d-b200-c02136887efe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:48:35.631497 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:35.631475    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae793f09-0221-462d-b200-c02136887efe-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:48:35.633358 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:35.633319    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae793f09-0221-462d-b200-c02136887efe-kube-api-access-fmgfv" pod "ae793f09-0221-462d-b200-c02136887efe" (UID: "ae793f09-0221-462d-b200-c02136887efe"). InnerVolumeSpecName "kube-api-access-fmgfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:48:35.634216 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:35.634191    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae793f09-0221-462d-b200-c02136887efe-maas-api-xks-ca" pod "ae793f09-0221-462d-b200-c02136887efe" (UID: "ae793f09-0221-462d-b200-c02136887efe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:48:35.732439 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:35.732410    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmgfv\" (UniqueName: \"kubernetes.io/projected/ae793f09-0221-462d-b200-c02136887efe-kube-api-access-fmgfv\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:48:35.732439 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:35.732438    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae793f09-0221-462d-b200-c02136887efe-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:48:36.340528 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:36.340495    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d7f535-g7b2w"
Nov 01 03:48:36.340528 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:36.340520    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d7f535-g7b2w" event={"ID":"ae793f09-0221-462d-b200-c02136887efe","Type":"ContainerDied","Data":"096b66cb1f63fe2ff2f9eaac89420afb2bbc015d510c0dcb0d3626377e0e6f53"}
Nov 01 03:48:36.341054 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:36.340602    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="096b66cb1f63fe2ff2f9eaac89420afb2bbc015d510c0dcb0d3626377e0e6f53"
Nov 01 03:48:36.352085 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:36.352057    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:48:37.157486 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:37.157447    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:48:39.794796 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:39.794759    4930 scope.go:122] "RemoveContainer" containerID="079e52026a3008e76c0eb15672a6ff2ab50eef50da4def67a41b611ef249b18a"
Nov 01 03:48:39.806218 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:39.806197    4930 scope.go:122] "RemoveContainer" containerID="0c497e2b6a9baa51c7324bd3b76f45d6aaa806cd0fe0f52ad420b05723337047"
Nov 01 03:48:39.816031 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:39.816004    4930 scope.go:122] "RemoveContainer" containerID="8e376d0c566316a35d7dc4cb79578e5a4c0ce5065de2784974117ebbc4f3084f"
Nov 01 03:48:41.072823 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:41.072790    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:48:41.460373 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:41.460334    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:48:43.449338 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:43.449298    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:48:43.601196 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:43.601155    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:48:44.058482 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:44.058444    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43005547-kh5lj"]
Nov 01 03:48:44.107014 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:44.106982    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f855682-z69dn"]
Nov 01 03:48:44.110468 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:44.110443    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43005547-kh5lj"]
Nov 01 03:48:44.113482 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:44.113459    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f855682-z69dn"]
Nov 01 03:48:45.623413 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:45.623375    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="002037cc-a7d9-4eb3-af38-f3c45f47583b" path="/var/lib/kubelet/pods/002037cc-a7d9-4eb3-af38-f3c45f47583b/volumes"
Nov 01 03:48:45.624044 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:45.624025    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28a14d04-9448-4306-9af5-3a3259ce2beb" path="/var/lib/kubelet/pods/28a14d04-9448-4306-9af5-3a3259ce2beb/volumes"
Nov 01 03:48:45.657161 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:45.657131    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-620c59b5-2pbz9"]
Nov 01 03:48:45.657716 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:45.657698    4930 cpu_manager.go:414] "containerMap: removing container" podUID="e552bc78-982d-4ce6-97c1-693fbdfa46e5" containerName="revoke-keys"
Nov 01 03:48:45.657716 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:45.657715    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e552bc78-982d-4ce6-97c1-693fbdfa46e5" containerName="revoke-keys"
Nov 01 03:48:45.657863 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:45.657744    4930 cpu_manager.go:414] "containerMap: removing container" podUID="ae793f09-0221-462d-b200-c02136887efe" containerName="revoke-keys"
Nov 01 03:48:45.657863 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:45.657750    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae793f09-0221-462d-b200-c02136887efe" containerName="revoke-keys"
Nov 01 03:48:45.657863 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:45.657854    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae793f09-0221-462d-b200-c02136887efe" containerName="revoke-keys"
Nov 01 03:48:45.658039 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:45.657870    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="e552bc78-982d-4ce6-97c1-693fbdfa46e5" containerName="revoke-keys"
Nov 01 03:48:45.665378 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:45.664384    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-620c59b5-2pbz9"
Nov 01 03:48:45.713016 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:45.712983    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ca62595-4t74g"]
Nov 01 03:48:45.717949 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:45.717920    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca62595-4t74g"
Nov 01 03:48:45.720757 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:45.720734    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-620c59b5-2pbz9"]
Nov 01 03:48:45.758184 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:45.758157    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:48:45.765776 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:45.765746    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:48:45.765950 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:45.765815    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:48:45.765950 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:45.765825    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:48:45.790536 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:45.790488    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ca62595-4t74g"]
Nov 01 03:48:45.793582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:45.793508    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baa79f53-4c43-4b71-9e2a-70033aa07c73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-620c59b5-2pbz9\" (UID: \"baa79f53-4c43-4b71-9e2a-70033aa07c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-620c59b5-2pbz9"
Nov 01 03:48:45.793727 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:45.793702    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7p6s\" (UniqueName: \"kubernetes.io/projected/baa79f53-4c43-4b71-9e2a-70033aa07c73-kube-api-access-k7p6s\") pod \"maas-api-revoke-sub-620c59b5-2pbz9\" (UID: \"baa79f53-4c43-4b71-9e2a-70033aa07c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-620c59b5-2pbz9"
Nov 01 03:48:45.794755 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:45.794725    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baa79f53-4c43-4b71-9e2a-70033aa07c73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-620c59b5-2pbz9\" (UID: \"baa79f53-4c43-4b71-9e2a-70033aa07c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-620c59b5-2pbz9"
Nov 01 03:48:45.896356 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:45.896270    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baa79f53-4c43-4b71-9e2a-70033aa07c73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-620c59b5-2pbz9\" (UID: \"baa79f53-4c43-4b71-9e2a-70033aa07c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-620c59b5-2pbz9"
Nov 01 03:48:45.896356 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:45.896341    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baa79f53-4c43-4b71-9e2a-70033aa07c73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-620c59b5-2pbz9\" (UID: \"baa79f53-4c43-4b71-9e2a-70033aa07c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-620c59b5-2pbz9"
Nov 01 03:48:45.896610 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:45.896389    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73f7e8a8-f82d-4176-be18-923cff53c4bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ca62595-4t74g\" (UID: \"73f7e8a8-f82d-4176-be18-923cff53c4bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca62595-4t74g"
Nov 01 03:48:45.896610 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:45.896477    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73f7e8a8-f82d-4176-be18-923cff53c4bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ca62595-4t74g\" (UID: \"73f7e8a8-f82d-4176-be18-923cff53c4bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca62595-4t74g"
Nov 01 03:48:45.896743 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:45.896685    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7p6s\" (UniqueName: \"kubernetes.io/projected/baa79f53-4c43-4b71-9e2a-70033aa07c73-kube-api-access-k7p6s\") pod \"maas-api-revoke-sub-620c59b5-2pbz9\" (UID: \"baa79f53-4c43-4b71-9e2a-70033aa07c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-620c59b5-2pbz9"
Nov 01 03:48:45.896743 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:45.896720    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwhxk\" (UniqueName: \"kubernetes.io/projected/73f7e8a8-f82d-4176-be18-923cff53c4bd-kube-api-access-mwhxk\") pod \"maas-api-revoke-sub-4ca62595-4t74g\" (UID: \"73f7e8a8-f82d-4176-be18-923cff53c4bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca62595-4t74g"
Nov 01 03:48:45.897140 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:45.897112    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baa79f53-4c43-4b71-9e2a-70033aa07c73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-620c59b5-2pbz9\" (UID: \"baa79f53-4c43-4b71-9e2a-70033aa07c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-620c59b5-2pbz9"
Nov 01 03:48:45.898785 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:45.898762    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baa79f53-4c43-4b71-9e2a-70033aa07c73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-620c59b5-2pbz9\" (UID: \"baa79f53-4c43-4b71-9e2a-70033aa07c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-620c59b5-2pbz9"
Nov 01 03:48:45.916198 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:45.916167    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:48:45.960065 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:45.960035    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7p6s\" (UniqueName: \"kubernetes.io/projected/baa79f53-4c43-4b71-9e2a-70033aa07c73-kube-api-access-k7p6s\") pod \"maas-api-revoke-sub-620c59b5-2pbz9\" (UID: \"baa79f53-4c43-4b71-9e2a-70033aa07c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-620c59b5-2pbz9"
Nov 01 03:48:46.010579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:45.993879    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-620c59b5-2pbz9"
Nov 01 03:48:46.010579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:45.998163    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73f7e8a8-f82d-4176-be18-923cff53c4bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ca62595-4t74g\" (UID: \"73f7e8a8-f82d-4176-be18-923cff53c4bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca62595-4t74g"
Nov 01 03:48:46.010579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:45.998240    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73f7e8a8-f82d-4176-be18-923cff53c4bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ca62595-4t74g\" (UID: \"73f7e8a8-f82d-4176-be18-923cff53c4bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca62595-4t74g"
Nov 01 03:48:46.010579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:45.998313    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwhxk\" (UniqueName: \"kubernetes.io/projected/73f7e8a8-f82d-4176-be18-923cff53c4bd-kube-api-access-mwhxk\") pod \"maas-api-revoke-sub-4ca62595-4t74g\" (UID: \"73f7e8a8-f82d-4176-be18-923cff53c4bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca62595-4t74g"
Nov 01 03:48:46.017678 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:46.011846    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73f7e8a8-f82d-4176-be18-923cff53c4bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ca62595-4t74g\" (UID: \"73f7e8a8-f82d-4176-be18-923cff53c4bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca62595-4t74g"
Nov 01 03:48:46.041441 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:46.041404    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73f7e8a8-f82d-4176-be18-923cff53c4bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ca62595-4t74g\" (UID: \"73f7e8a8-f82d-4176-be18-923cff53c4bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca62595-4t74g"
Nov 01 03:48:46.060593 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:46.060505    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwhxk\" (UniqueName: \"kubernetes.io/projected/73f7e8a8-f82d-4176-be18-923cff53c4bd-kube-api-access-mwhxk\") pod \"maas-api-revoke-sub-4ca62595-4t74g\" (UID: \"73f7e8a8-f82d-4176-be18-923cff53c4bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca62595-4t74g"
Nov 01 03:48:46.064230 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:46.064202    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:48:46.358977 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:46.357796    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca62595-4t74g"
Nov 01 03:48:46.486920 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:46.486889    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:48:46.581745 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:46.565712    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-620c59b5-2pbz9"]
Nov 01 03:48:47.148030 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:48:47.147791    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73f7e8a8_f82d_4176_be18_923cff53c4bd.slice/crio-913850b51f9ce39a92f13c021fee40d70b9744c613ea6e65220517f829307d78 WatchSource:0}: Error finding container 913850b51f9ce39a92f13c021fee40d70b9744c613ea6e65220517f829307d78: Status 404 returned error can't find the container with id 913850b51f9ce39a92f13c021fee40d70b9744c613ea6e65220517f829307d78
Nov 01 03:48:47.295629 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:47.274886    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ca62595-4t74g"]
Nov 01 03:48:47.635295 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:47.635255    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-620c59b5-2pbz9" event={"ID":"baa79f53-4c43-4b71-9e2a-70033aa07c73","Type":"ContainerStarted","Data":"ef1a1c04adb2dada316e9c49d1c6b29faad4febbfbf5a94ef56638620addd274"}
Nov 01 03:48:47.638949 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:47.638922    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca62595-4t74g" event={"ID":"73f7e8a8-f82d-4176-be18-923cff53c4bd","Type":"ContainerStarted","Data":"913850b51f9ce39a92f13c021fee40d70b9744c613ea6e65220517f829307d78"}
Nov 01 03:48:48.543823 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:48.543744    4930 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp" podUID="667240b0-64a7-45cd-84fe-9cd7e0d8a301" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:48:48.695732 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:48.695091    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-620c59b5-2pbz9" event={"ID":"baa79f53-4c43-4b71-9e2a-70033aa07c73","Type":"ContainerStarted","Data":"064580ebbfed0c67f560363dc1b62ae4007a26445674624d962cfaf01e101732"}
Nov 01 03:48:48.727591 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:48.712400    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca62595-4t74g" event={"ID":"73f7e8a8-f82d-4176-be18-923cff53c4bd","Type":"ContainerStarted","Data":"28ab47259c162955e9d731a23eba68fa32712e828627a598dd74f727d436db72"}
Nov 01 03:48:48.818181 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:48.816048    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps" podUID="d658772c-b072-4513-a639-d19a94c78525" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:48:48.834313 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:48.834243    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-620c59b5-2pbz9" podStartSLOduration=3.834225088 podStartE2EDuration="3.834225088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:48:48.83182103 +0000 UTC m=+2533.354827625" watchObservedRunningTime="2026-11-01 03:48:48.834225088 +0000 UTC m=+2533.357231679"
Nov 01 03:48:48.973578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:48.969107    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca62595-4t74g" podStartSLOduration=3.969088299 podStartE2EDuration="3.969088299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:48:48.965631875 +0000 UTC m=+2533.488638467" watchObservedRunningTime="2026-11-01 03:48:48.969088299 +0000 UTC m=+2533.492094887"
Nov 01 03:48:49.137302 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:48:49.137260    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:48:49.725892 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:49.725856    4930 generic.go:356] "Generic (PLEG): container finished" podID="baa79f53-4c43-4b71-9e2a-70033aa07c73" containerID="064580ebbfed0c67f560363dc1b62ae4007a26445674624d962cfaf01e101732" exitCode=0
Nov 01 03:48:49.726381 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:49.725954    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-620c59b5-2pbz9" event={"ID":"baa79f53-4c43-4b71-9e2a-70033aa07c73","Type":"ContainerDied","Data":"064580ebbfed0c67f560363dc1b62ae4007a26445674624d962cfaf01e101732"}
Nov 01 03:48:49.727754 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:49.727727    4930 generic.go:356] "Generic (PLEG): container finished" podID="73f7e8a8-f82d-4176-be18-923cff53c4bd" containerID="28ab47259c162955e9d731a23eba68fa32712e828627a598dd74f727d436db72" exitCode=0
Nov 01 03:48:49.727871 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:49.727811    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca62595-4t74g" event={"ID":"73f7e8a8-f82d-4176-be18-923cff53c4bd","Type":"ContainerDied","Data":"28ab47259c162955e9d731a23eba68fa32712e828627a598dd74f727d436db72"}
Nov 01 03:48:50.624799 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:50.624758    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:48:50.913591 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:50.913544    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-620c59b5-2pbz9"
Nov 01 03:48:50.945032 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:50.945006    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca62595-4t74g"
Nov 01 03:48:50.949264 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:50.949237    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73f7e8a8-f82d-4176-be18-923cff53c4bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73f7e8a8-f82d-4176-be18-923cff53c4bd-maas-api-openshift-ca\") pod \"73f7e8a8-f82d-4176-be18-923cff53c4bd\" (UID: \"73f7e8a8-f82d-4176-be18-923cff53c4bd\") "
Nov 01 03:48:50.949397 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:50.949285    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/baa79f53-4c43-4b71-9e2a-70033aa07c73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baa79f53-4c43-4b71-9e2a-70033aa07c73-maas-api-xks-ca\") pod \"baa79f53-4c43-4b71-9e2a-70033aa07c73\" (UID: \"baa79f53-4c43-4b71-9e2a-70033aa07c73\") "
Nov 01 03:48:50.949501 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:50.949484    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baa79f53-4c43-4b71-9e2a-70033aa07c73-kube-api-access-k7p6s\" (UniqueName: \"kubernetes.io/projected/baa79f53-4c43-4b71-9e2a-70033aa07c73-kube-api-access-k7p6s\") pod \"baa79f53-4c43-4b71-9e2a-70033aa07c73\" (UID: \"baa79f53-4c43-4b71-9e2a-70033aa07c73\") "
Nov 01 03:48:50.949584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:50.949534    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73f7e8a8-f82d-4176-be18-923cff53c4bd-kube-api-access-mwhxk\" (UniqueName: \"kubernetes.io/projected/73f7e8a8-f82d-4176-be18-923cff53c4bd-kube-api-access-mwhxk\") pod \"73f7e8a8-f82d-4176-be18-923cff53c4bd\" (UID: \"73f7e8a8-f82d-4176-be18-923cff53c4bd\") "
Nov 01 03:48:50.949641 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:50.949547    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73f7e8a8-f82d-4176-be18-923cff53c4bd-maas-api-openshift-ca" pod "73f7e8a8-f82d-4176-be18-923cff53c4bd" (UID: "73f7e8a8-f82d-4176-be18-923cff53c4bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:48:50.949641 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:50.949613    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baa79f53-4c43-4b71-9e2a-70033aa07c73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baa79f53-4c43-4b71-9e2a-70033aa07c73-maas-api-openshift-ca\") pod \"baa79f53-4c43-4b71-9e2a-70033aa07c73\" (UID: \"baa79f53-4c43-4b71-9e2a-70033aa07c73\") "
Nov 01 03:48:50.949762 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:50.949660    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73f7e8a8-f82d-4176-be18-923cff53c4bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73f7e8a8-f82d-4176-be18-923cff53c4bd-maas-api-xks-ca\") pod \"73f7e8a8-f82d-4176-be18-923cff53c4bd\" (UID: \"73f7e8a8-f82d-4176-be18-923cff53c4bd\") "
Nov 01 03:48:50.950061 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:50.950036    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baa79f53-4c43-4b71-9e2a-70033aa07c73-maas-api-openshift-ca" pod "baa79f53-4c43-4b71-9e2a-70033aa07c73" (UID: "baa79f53-4c43-4b71-9e2a-70033aa07c73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:48:50.950280 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:50.950255    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baa79f53-4c43-4b71-9e2a-70033aa07c73-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:48:50.950280 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:50.950282    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73f7e8a8-f82d-4176-be18-923cff53c4bd-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:48:50.952396 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:50.952069    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baa79f53-4c43-4b71-9e2a-70033aa07c73-kube-api-access-k7p6s" pod "baa79f53-4c43-4b71-9e2a-70033aa07c73" (UID: "baa79f53-4c43-4b71-9e2a-70033aa07c73"). InnerVolumeSpecName "kube-api-access-k7p6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:48:50.952396 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:50.952198    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73f7e8a8-f82d-4176-be18-923cff53c4bd-kube-api-access-mwhxk" pod "73f7e8a8-f82d-4176-be18-923cff53c4bd" (UID: "73f7e8a8-f82d-4176-be18-923cff53c4bd"). InnerVolumeSpecName "kube-api-access-mwhxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:48:50.952396 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:50.952326    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73f7e8a8-f82d-4176-be18-923cff53c4bd-maas-api-xks-ca" pod "73f7e8a8-f82d-4176-be18-923cff53c4bd" (UID: "73f7e8a8-f82d-4176-be18-923cff53c4bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:48:50.952497 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:50.952428    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baa79f53-4c43-4b71-9e2a-70033aa07c73-maas-api-xks-ca" pod "baa79f53-4c43-4b71-9e2a-70033aa07c73" (UID: "baa79f53-4c43-4b71-9e2a-70033aa07c73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:48:51.051662 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:51.051622    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baa79f53-4c43-4b71-9e2a-70033aa07c73-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:48:51.051662 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:51.051661    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7p6s\" (UniqueName: \"kubernetes.io/projected/baa79f53-4c43-4b71-9e2a-70033aa07c73-kube-api-access-k7p6s\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:48:51.051914 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:51.051678    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwhxk\" (UniqueName: \"kubernetes.io/projected/73f7e8a8-f82d-4176-be18-923cff53c4bd-kube-api-access-mwhxk\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:48:51.051914 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:51.051692    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73f7e8a8-f82d-4176-be18-923cff53c4bd-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:48:51.452258 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:51.452221    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:48:51.742869 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:51.742786    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-620c59b5-2pbz9" event={"ID":"baa79f53-4c43-4b71-9e2a-70033aa07c73","Type":"ContainerDied","Data":"ef1a1c04adb2dada316e9c49d1c6b29faad4febbfbf5a94ef56638620addd274"}
Nov 01 03:48:51.742869 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:51.742827    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-620c59b5-2pbz9"
Nov 01 03:48:51.742869 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:51.742838    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef1a1c04adb2dada316e9c49d1c6b29faad4febbfbf5a94ef56638620addd274"
Nov 01 03:48:51.744210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:51.744186    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca62595-4t74g" event={"ID":"73f7e8a8-f82d-4176-be18-923cff53c4bd","Type":"ContainerDied","Data":"913850b51f9ce39a92f13c021fee40d70b9744c613ea6e65220517f829307d78"}
Nov 01 03:48:51.744210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:51.744200    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca62595-4t74g"
Nov 01 03:48:51.744385 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:51.744232    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="913850b51f9ce39a92f13c021fee40d70b9744c613ea6e65220517f829307d78"
Nov 01 03:48:56.434057 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:48:56.434018    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:49:00.661816 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:00.661774    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:49:02.228129 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:02.228094    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:49:02.469162 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:02.469126    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:49:02.613129 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:02.613095    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:49:02.727163 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:02.727132    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c8373-lgpzq"]
Nov 01 03:49:02.775960 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:02.775932    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c8373-lgpzq"]
Nov 01 03:49:03.225433 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:03.225394    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:49:03.635365 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:03.635328    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="830483ce-0cd5-45a5-a562-201d1ba81feb" path="/var/lib/kubelet/pods/830483ce-0cd5-45a5-a562-201d1ba81feb/volumes"
Nov 01 03:49:07.575738 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:07.575707    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:49:09.765129 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:09.765096    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:49:09.937579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:09.937521    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:49:10.122781 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:10.122753    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"]
Nov 01 03:49:10.123002 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:10.122975    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw" podUID="27e7c281-1391-4406-9dfa-d770c821678e" containerName="istio-proxy" containerID="cri-o://c6aea9e714e25329afaff81b3f7cf329000a41fb084c7167a37f59b1f12f6e8a" gracePeriod=30
Nov 01 03:49:10.135281 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:10.135252    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:49:10.929793 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:10.929758    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-417aa8-69hzv"]
Nov 01 03:49:10.930309 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:10.930294    4930 cpu_manager.go:414] "containerMap: removing container" podUID="73f7e8a8-f82d-4176-be18-923cff53c4bd" containerName="revoke-keys"
Nov 01 03:49:10.930359 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:10.930310    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73f7e8a8-f82d-4176-be18-923cff53c4bd" containerName="revoke-keys"
Nov 01 03:49:10.930359 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:10.930345    4930 cpu_manager.go:414] "containerMap: removing container" podUID="baa79f53-4c43-4b71-9e2a-70033aa07c73" containerName="revoke-keys"
Nov 01 03:49:10.930359 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:10.930350    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baa79f53-4c43-4b71-9e2a-70033aa07c73" containerName="revoke-keys"
Nov 01 03:49:10.930462 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:10.930451    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="baa79f53-4c43-4b71-9e2a-70033aa07c73" containerName="revoke-keys"
Nov 01 03:49:10.930498 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:10.930464    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="73f7e8a8-f82d-4176-be18-923cff53c4bd" containerName="revoke-keys"
Nov 01 03:49:10.934923 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:10.934905    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-417aa8-69hzv"
Nov 01 03:49:10.987896 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:10.987850    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:49:11.026780 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:11.026748    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c331aa42-00f4-4819-8efe-679636066bd7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-417aa8-69hzv\" (UID: \"c331aa42-00f4-4819-8efe-679636066bd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-417aa8-69hzv"
Nov 01 03:49:11.026965 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:11.026888    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c331aa42-00f4-4819-8efe-679636066bd7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-417aa8-69hzv\" (UID: \"c331aa42-00f4-4819-8efe-679636066bd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-417aa8-69hzv"
Nov 01 03:49:11.027077 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:11.027056    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvjk5\" (UniqueName: \"kubernetes.io/projected/c331aa42-00f4-4819-8efe-679636066bd7-kube-api-access-nvjk5\") pod \"maas-api-revoke-keys-e2e-api-b-417aa8-69hzv\" (UID: \"c331aa42-00f4-4819-8efe-679636066bd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-417aa8-69hzv"
Nov 01 03:49:11.037128 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:11.037107    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:49:11.037401 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:11.037382    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:49:11.037501 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:11.037469    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:49:11.041295 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:11.041276    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-417aa8-69hzv"]
Nov 01 03:49:11.127810 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:11.127779    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c331aa42-00f4-4819-8efe-679636066bd7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-417aa8-69hzv\" (UID: \"c331aa42-00f4-4819-8efe-679636066bd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-417aa8-69hzv"
Nov 01 03:49:11.127953 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:11.127870    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvjk5\" (UniqueName: \"kubernetes.io/projected/c331aa42-00f4-4819-8efe-679636066bd7-kube-api-access-nvjk5\") pod \"maas-api-revoke-keys-e2e-api-b-417aa8-69hzv\" (UID: \"c331aa42-00f4-4819-8efe-679636066bd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-417aa8-69hzv"
Nov 01 03:49:11.127953 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:11.127916    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c331aa42-00f4-4819-8efe-679636066bd7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-417aa8-69hzv\" (UID: \"c331aa42-00f4-4819-8efe-679636066bd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-417aa8-69hzv"
Nov 01 03:49:11.128481 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:11.128461    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c331aa42-00f4-4819-8efe-679636066bd7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-417aa8-69hzv\" (UID: \"c331aa42-00f4-4819-8efe-679636066bd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-417aa8-69hzv"
Nov 01 03:49:11.130336 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:11.130313    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c331aa42-00f4-4819-8efe-679636066bd7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-417aa8-69hzv\" (UID: \"c331aa42-00f4-4819-8efe-679636066bd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-417aa8-69hzv"
Nov 01 03:49:11.189311 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:11.189251    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvjk5\" (UniqueName: \"kubernetes.io/projected/c331aa42-00f4-4819-8efe-679636066bd7-kube-api-access-nvjk5\") pod \"maas-api-revoke-keys-e2e-api-b-417aa8-69hzv\" (UID: \"c331aa42-00f4-4819-8efe-679636066bd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-417aa8-69hzv"
Nov 01 03:49:11.249336 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:11.249310    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-417aa8-69hzv"
Nov 01 03:49:11.636540 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:11.636501    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-417aa8-69hzv"]
Nov 01 03:49:11.644041 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:49:11.644010    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc331aa42_00f4_4819_8efe_679636066bd7.slice/crio-1eba764d7dbd16955a836247ea8a7a387b9787862667d9f23865db749ad60f0c WatchSource:0}: Error finding container 1eba764d7dbd16955a836247ea8a7a387b9787862667d9f23865db749ad60f0c: Status 404 returned error can't find the container with id 1eba764d7dbd16955a836247ea8a7a387b9787862667d9f23865db749ad60f0c
Nov 01 03:49:11.908128 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:11.908031    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-417aa8-69hzv" event={"ID":"c331aa42-00f4-4819-8efe-679636066bd7","Type":"ContainerStarted","Data":"1eba764d7dbd16955a836247ea8a7a387b9787862667d9f23865db749ad60f0c"}
Nov 01 03:49:12.763882 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:12.763845    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:49:12.916757 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:12.916675    4930 generic.go:356] "Generic (PLEG): container finished" podID="c331aa42-00f4-4819-8efe-679636066bd7" containerID="88749014e2c2123ad86bfd94c63964a5bf31b7b7a26c7a40f25398af5aed2a4b" exitCode=0
Nov 01 03:49:12.916905 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:12.916768    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-417aa8-69hzv" event={"ID":"c331aa42-00f4-4819-8efe-679636066bd7","Type":"ContainerDied","Data":"88749014e2c2123ad86bfd94c63964a5bf31b7b7a26c7a40f25398af5aed2a4b"}
Nov 01 03:49:13.445133 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:13.445093    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:49:13.632875 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:13.632848    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:49:14.089083 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:14.089059    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-417aa8-69hzv"
Nov 01 03:49:14.282359 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:14.282258    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c331aa42-00f4-4819-8efe-679636066bd7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c331aa42-00f4-4819-8efe-679636066bd7-maas-api-xks-ca\") pod \"c331aa42-00f4-4819-8efe-679636066bd7\" (UID: \"c331aa42-00f4-4819-8efe-679636066bd7\") "
Nov 01 03:49:14.282542 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:14.282488    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c331aa42-00f4-4819-8efe-679636066bd7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c331aa42-00f4-4819-8efe-679636066bd7-maas-api-openshift-ca\") pod \"c331aa42-00f4-4819-8efe-679636066bd7\" (UID: \"c331aa42-00f4-4819-8efe-679636066bd7\") "
Nov 01 03:49:14.282667 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:14.282632    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c331aa42-00f4-4819-8efe-679636066bd7-kube-api-access-nvjk5\" (UniqueName: \"kubernetes.io/projected/c331aa42-00f4-4819-8efe-679636066bd7-kube-api-access-nvjk5\") pod \"c331aa42-00f4-4819-8efe-679636066bd7\" (UID: \"c331aa42-00f4-4819-8efe-679636066bd7\") "
Nov 01 03:49:14.282948 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:14.282911    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c331aa42-00f4-4819-8efe-679636066bd7-maas-api-openshift-ca" pod "c331aa42-00f4-4819-8efe-679636066bd7" (UID: "c331aa42-00f4-4819-8efe-679636066bd7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:49:14.283233 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:14.283213    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c331aa42-00f4-4819-8efe-679636066bd7-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:49:14.284446 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:14.284421    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c331aa42-00f4-4819-8efe-679636066bd7-maas-api-xks-ca" pod "c331aa42-00f4-4819-8efe-679636066bd7" (UID: "c331aa42-00f4-4819-8efe-679636066bd7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:49:14.285097 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:14.285059    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c331aa42-00f4-4819-8efe-679636066bd7-kube-api-access-nvjk5" pod "c331aa42-00f4-4819-8efe-679636066bd7" (UID: "c331aa42-00f4-4819-8efe-679636066bd7"). InnerVolumeSpecName "kube-api-access-nvjk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:49:14.384385 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:14.384354    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvjk5\" (UniqueName: \"kubernetes.io/projected/c331aa42-00f4-4819-8efe-679636066bd7-kube-api-access-nvjk5\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:49:14.384546 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:14.384388    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c331aa42-00f4-4819-8efe-679636066bd7-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:49:14.964348 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:14.964189    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-417aa8-69hzv" event={"ID":"c331aa42-00f4-4819-8efe-679636066bd7","Type":"ContainerDied","Data":"1eba764d7dbd16955a836247ea8a7a387b9787862667d9f23865db749ad60f0c"}
Nov 01 03:49:14.964348 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:14.964261    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1eba764d7dbd16955a836247ea8a7a387b9787862667d9f23865db749ad60f0c"
Nov 01 03:49:14.964348 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:14.964340    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-417aa8-69hzv"
Nov 01 03:49:15.411142 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:15.411117    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"
Nov 01 03:49:15.495779 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:15.495679    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27e7c281-1391-4406-9dfa-d770c821678e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27e7c281-1391-4406-9dfa-d770c821678e-workload-socket\") pod \"27e7c281-1391-4406-9dfa-d770c821678e\" (UID: \"27e7c281-1391-4406-9dfa-d770c821678e\") "
Nov 01 03:49:15.495779 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:15.495751    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27e7c281-1391-4406-9dfa-d770c821678e-kube-api-access-dcbmj\" (UniqueName: \"kubernetes.io/projected/27e7c281-1391-4406-9dfa-d770c821678e-kube-api-access-dcbmj\") pod \"27e7c281-1391-4406-9dfa-d770c821678e\" (UID: \"27e7c281-1391-4406-9dfa-d770c821678e\") "
Nov 01 03:49:15.496050 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:15.495803    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27e7c281-1391-4406-9dfa-d770c821678e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27e7c281-1391-4406-9dfa-d770c821678e-workload-certs\") pod \"27e7c281-1391-4406-9dfa-d770c821678e\" (UID: \"27e7c281-1391-4406-9dfa-d770c821678e\") "
Nov 01 03:49:15.496050 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:15.495910    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27e7c281-1391-4406-9dfa-d770c821678e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27e7c281-1391-4406-9dfa-d770c821678e-credential-socket\") pod \"27e7c281-1391-4406-9dfa-d770c821678e\" (UID: \"27e7c281-1391-4406-9dfa-d770c821678e\") "
Nov 01 03:49:15.496050 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:15.495968    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27e7c281-1391-4406-9dfa-d770c821678e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27e7c281-1391-4406-9dfa-d770c821678e-istio-data\") pod \"27e7c281-1391-4406-9dfa-d770c821678e\" (UID: \"27e7c281-1391-4406-9dfa-d770c821678e\") "
Nov 01 03:49:15.496050 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:15.496021    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27e7c281-1391-4406-9dfa-d770c821678e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27e7c281-1391-4406-9dfa-d770c821678e-istio-podinfo\") pod \"27e7c281-1391-4406-9dfa-d770c821678e\" (UID: \"27e7c281-1391-4406-9dfa-d770c821678e\") "
Nov 01 03:49:15.496296 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:15.496080    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27e7c281-1391-4406-9dfa-d770c821678e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27e7c281-1391-4406-9dfa-d770c821678e-istio-envoy\") pod \"27e7c281-1391-4406-9dfa-d770c821678e\" (UID: \"27e7c281-1391-4406-9dfa-d770c821678e\") "
Nov 01 03:49:15.496296 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:15.496074    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27e7c281-1391-4406-9dfa-d770c821678e-workload-socket" pod "27e7c281-1391-4406-9dfa-d770c821678e" (UID: "27e7c281-1391-4406-9dfa-d770c821678e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:49:15.496296 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:15.496161    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27e7c281-1391-4406-9dfa-d770c821678e-istio-token\" (UniqueName: \"kubernetes.io/projected/27e7c281-1391-4406-9dfa-d770c821678e-istio-token\") pod \"27e7c281-1391-4406-9dfa-d770c821678e\" (UID: \"27e7c281-1391-4406-9dfa-d770c821678e\") "
Nov 01 03:49:15.496296 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:15.496248    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27e7c281-1391-4406-9dfa-d770c821678e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27e7c281-1391-4406-9dfa-d770c821678e-istiod-ca-cert\") pod \"27e7c281-1391-4406-9dfa-d770c821678e\" (UID: \"27e7c281-1391-4406-9dfa-d770c821678e\") "
Nov 01 03:49:15.496522 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:15.496358    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27e7c281-1391-4406-9dfa-d770c821678e-credential-socket" pod "27e7c281-1391-4406-9dfa-d770c821678e" (UID: "27e7c281-1391-4406-9dfa-d770c821678e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:49:15.496904 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:15.496864    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27e7c281-1391-4406-9dfa-d770c821678e-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:49:15.496904 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:15.496897    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27e7c281-1391-4406-9dfa-d770c821678e-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:49:15.497108 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:15.496941    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27e7c281-1391-4406-9dfa-d770c821678e-workload-certs" pod "27e7c281-1391-4406-9dfa-d770c821678e" (UID: "27e7c281-1391-4406-9dfa-d770c821678e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:49:15.497691 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:15.497388    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27e7c281-1391-4406-9dfa-d770c821678e-istio-data" pod "27e7c281-1391-4406-9dfa-d770c821678e" (UID: "27e7c281-1391-4406-9dfa-d770c821678e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:49:15.497691 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:15.497525    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27e7c281-1391-4406-9dfa-d770c821678e-istiod-ca-cert" pod "27e7c281-1391-4406-9dfa-d770c821678e" (UID: "27e7c281-1391-4406-9dfa-d770c821678e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:49:15.499571 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:15.499512    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27e7c281-1391-4406-9dfa-d770c821678e-kube-api-access-dcbmj" pod "27e7c281-1391-4406-9dfa-d770c821678e" (UID: "27e7c281-1391-4406-9dfa-d770c821678e"). InnerVolumeSpecName "kube-api-access-dcbmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:49:15.502920 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:15.502789    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27e7c281-1391-4406-9dfa-d770c821678e-istio-token" pod "27e7c281-1391-4406-9dfa-d770c821678e" (UID: "27e7c281-1391-4406-9dfa-d770c821678e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:49:15.503290 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:15.503263    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27e7c281-1391-4406-9dfa-d770c821678e-istio-podinfo" pod "27e7c281-1391-4406-9dfa-d770c821678e" (UID: "27e7c281-1391-4406-9dfa-d770c821678e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:49:15.503801 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:15.503766    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27e7c281-1391-4406-9dfa-d770c821678e-istio-envoy" pod "27e7c281-1391-4406-9dfa-d770c821678e" (UID: "27e7c281-1391-4406-9dfa-d770c821678e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:49:15.603456 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:15.599741    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27e7c281-1391-4406-9dfa-d770c821678e-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:49:15.603456 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:15.599786    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27e7c281-1391-4406-9dfa-d770c821678e-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:49:15.603456 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:15.599802    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27e7c281-1391-4406-9dfa-d770c821678e-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:49:15.603456 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:15.599816    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27e7c281-1391-4406-9dfa-d770c821678e-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:49:15.603456 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:15.599831    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27e7c281-1391-4406-9dfa-d770c821678e-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:49:15.603456 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:15.599847    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27e7c281-1391-4406-9dfa-d770c821678e-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:49:15.603456 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:15.599879    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcbmj\" (UniqueName: \"kubernetes.io/projected/27e7c281-1391-4406-9dfa-d770c821678e-kube-api-access-dcbmj\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:49:15.974467 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:15.974415    4930 generic.go:356] "Generic (PLEG): container finished" podID="27e7c281-1391-4406-9dfa-d770c821678e" containerID="c6aea9e714e25329afaff81b3f7cf329000a41fb084c7167a37f59b1f12f6e8a" exitCode=0
Nov 01 03:49:15.974721 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:15.974487    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"
Nov 01 03:49:15.974721 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:15.974495    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw" event={"ID":"27e7c281-1391-4406-9dfa-d770c821678e","Type":"ContainerDied","Data":"c6aea9e714e25329afaff81b3f7cf329000a41fb084c7167a37f59b1f12f6e8a"}
Nov 01 03:49:15.974721 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:15.974631    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw" event={"ID":"27e7c281-1391-4406-9dfa-d770c821678e","Type":"ContainerDied","Data":"eec9740092ce9b07e3d448b4ec0de2b89003a62251c19905e444706b174e651f"}
Nov 01 03:49:15.974721 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:15.974668    4930 scope.go:122] "RemoveContainer" containerID="c6aea9e714e25329afaff81b3f7cf329000a41fb084c7167a37f59b1f12f6e8a"
Nov 01 03:49:15.985971 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:15.985952    4930 scope.go:122] "RemoveContainer" containerID="8ea181d9ef9afc7b4206e5a29ea369170cb7f6f25bf1a8fce106fa4f75966d9b"
Nov 01 03:49:15.996089 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:15.996070    4930 scope.go:122] "RemoveContainer" containerID="c6aea9e714e25329afaff81b3f7cf329000a41fb084c7167a37f59b1f12f6e8a"
Nov 01 03:49:15.996369 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:49:15.996346    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6aea9e714e25329afaff81b3f7cf329000a41fb084c7167a37f59b1f12f6e8a\": container with ID starting with c6aea9e714e25329afaff81b3f7cf329000a41fb084c7167a37f59b1f12f6e8a not found: ID does not exist" containerID="c6aea9e714e25329afaff81b3f7cf329000a41fb084c7167a37f59b1f12f6e8a"
Nov 01 03:49:15.996472 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:15.996374    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6aea9e714e25329afaff81b3f7cf329000a41fb084c7167a37f59b1f12f6e8a"} err="failed to get container status \"c6aea9e714e25329afaff81b3f7cf329000a41fb084c7167a37f59b1f12f6e8a\": rpc error: code = NotFound desc = could not find container \"c6aea9e714e25329afaff81b3f7cf329000a41fb084c7167a37f59b1f12f6e8a\": container with ID starting with c6aea9e714e25329afaff81b3f7cf329000a41fb084c7167a37f59b1f12f6e8a not found: ID does not exist"
Nov 01 03:49:15.996472 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:15.996392    4930 scope.go:122] "RemoveContainer" containerID="8ea181d9ef9afc7b4206e5a29ea369170cb7f6f25bf1a8fce106fa4f75966d9b"
Nov 01 03:49:15.996689 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:49:15.996664    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ea181d9ef9afc7b4206e5a29ea369170cb7f6f25bf1a8fce106fa4f75966d9b\": container with ID starting with 8ea181d9ef9afc7b4206e5a29ea369170cb7f6f25bf1a8fce106fa4f75966d9b not found: ID does not exist" containerID="8ea181d9ef9afc7b4206e5a29ea369170cb7f6f25bf1a8fce106fa4f75966d9b"
Nov 01 03:49:15.996745 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:15.996698    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ea181d9ef9afc7b4206e5a29ea369170cb7f6f25bf1a8fce106fa4f75966d9b"} err="failed to get container status \"8ea181d9ef9afc7b4206e5a29ea369170cb7f6f25bf1a8fce106fa4f75966d9b\": rpc error: code = NotFound desc = could not find container \"8ea181d9ef9afc7b4206e5a29ea369170cb7f6f25bf1a8fce106fa4f75966d9b\": container with ID starting with 8ea181d9ef9afc7b4206e5a29ea369170cb7f6f25bf1a8fce106fa4f75966d9b not found: ID does not exist"
Nov 01 03:49:16.145363 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:16.145325    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"]
Nov 01 03:49:16.195228 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:16.195199    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-2c8373-openshift-default-58ff44d5d9-rs4sw"]
Nov 01 03:49:17.065546 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:17.065451    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:49:17.625921 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:17.625883    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27e7c281-1391-4406-9dfa-d770c821678e" path="/var/lib/kubelet/pods/27e7c281-1391-4406-9dfa-d770c821678e/volumes"
Nov 01 03:49:18.292599 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:18.291945    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:49:19.125841 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:49:19.125794    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:49:23.410252 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:23.410220    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:49:23.560983 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:23.560946    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:49:23.709746 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:23.709655    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:49:23.934374 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:23.934336    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:49:24.081979 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:24.081884    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:49:28.894122 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:28.894088    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:49:33.642542 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:33.642506    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:49:38.438817 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:38.438779    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:49:39.853165 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:39.853135    4930 scope.go:122] "RemoveContainer" containerID="4868f7063b8326bd6f853bf6be38c43d6c1ae43d4f17cee2bc2ade3a2bb3ee7a"
Nov 01 03:49:41.429805 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:41.429773    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:49:43.232722 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:43.232690    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:49:45.003394 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:45.003359    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:49:47.832766 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:47.832716    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:49:47.976050 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:47.976009    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:49:48.116501 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:48.116469    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:49:49.123109 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:49:49.123074    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:49:50.909716 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:50.909686    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:49:52.138477 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:52.138418    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:49:53.714660 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:53.714626    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:49:53.928189 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:53.928156    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:49:54.072717 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:54.072607    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:49:54.215750 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:54.215716    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d7f535-g7b2w"]
Nov 01 03:49:54.264660 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:54.264627    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-d7f535-g7b2w"]
Nov 01 03:49:54.621822 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:54.621784    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:49:55.623386 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:55.623339    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae793f09-0221-462d-b200-c02136887efe" path="/var/lib/kubelet/pods/ae793f09-0221-462d-b200-c02136887efe/volumes"
Nov 01 03:49:59.080398 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:49:59.080360    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:50:00.966183 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:00.966145    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:50:01.129516 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:01.129481    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:50:01.338289 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:01.338254    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq"]
Nov 01 03:50:01.338531 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:01.338505    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq" podUID="4d42cb7e-b420-4bcb-a086-8ce40cc24849" containerName="istio-proxy" containerID="cri-o://f0008cb73ee130a8a17f6189321b0f5da7589a3cd2bebb0571ca6785b0f754a3" gracePeriod=30
Nov 01 03:50:01.346701 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:01.344379    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:50:02.848027 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:02.847992    4930 patch_prober.go:29] interesting pod/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:50:02.848437 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:02.848058    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq" podUID="4d42cb7e-b420-4bcb-a086-8ce40cc24849" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:50:03.395294 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:03.395258    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:50:04.951126 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:04.951095    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:50:05.219376 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:05.219287    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:50:05.380130 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:05.380099    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:50:05.563699 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:05.563598    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fd6cf-fct5g"]
Nov 01 03:50:05.629613 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:05.629581    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fd6cf-fct5g"]
Nov 01 03:50:05.677512 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:50:05.677474    4930 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-keys-e2e-api-a-5fd6cf-fct5g\" is forbidden: User \"system:node:ip-10-0-141-122.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-141-122.us-west-1.compute.internal' and this object" podUID="8cfeed06-fada-48eb-956b-317138fa24fa" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5fd6cf-fct5g"
Nov 01 03:50:05.930761 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:05.930727    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:50:06.616819 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:06.616759    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq"
Nov 01 03:50:06.665528 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:06.665502    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d42cb7e-b420-4bcb-a086-8ce40cc24849-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d42cb7e-b420-4bcb-a086-8ce40cc24849-istiod-ca-cert\") pod \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\" (UID: \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\") "
Nov 01 03:50:06.665721 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:06.665542    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d42cb7e-b420-4bcb-a086-8ce40cc24849-istio-token\" (UniqueName: \"kubernetes.io/projected/4d42cb7e-b420-4bcb-a086-8ce40cc24849-istio-token\") pod \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\" (UID: \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\") "
Nov 01 03:50:06.665721 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:06.665611    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d42cb7e-b420-4bcb-a086-8ce40cc24849-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d42cb7e-b420-4bcb-a086-8ce40cc24849-workload-certs\") pod \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\" (UID: \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\") "
Nov 01 03:50:06.665854 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:06.665771    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d42cb7e-b420-4bcb-a086-8ce40cc24849-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d42cb7e-b420-4bcb-a086-8ce40cc24849-credential-socket\") pod \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\" (UID: \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\") "
Nov 01 03:50:06.665854 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:06.665805    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d42cb7e-b420-4bcb-a086-8ce40cc24849-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d42cb7e-b420-4bcb-a086-8ce40cc24849-istio-podinfo\") pod \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\" (UID: \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\") "
Nov 01 03:50:06.665854 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:06.665836    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d42cb7e-b420-4bcb-a086-8ce40cc24849-kube-api-access-hmgg6\" (UniqueName: \"kubernetes.io/projected/4d42cb7e-b420-4bcb-a086-8ce40cc24849-kube-api-access-hmgg6\") pod \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\" (UID: \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\") "
Nov 01 03:50:06.666029 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:06.665893    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d42cb7e-b420-4bcb-a086-8ce40cc24849-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d42cb7e-b420-4bcb-a086-8ce40cc24849-workload-socket\") pod \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\" (UID: \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\") "
Nov 01 03:50:06.666029 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:06.665927    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d42cb7e-b420-4bcb-a086-8ce40cc24849-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d42cb7e-b420-4bcb-a086-8ce40cc24849-istio-data\") pod \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\" (UID: \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\") "
Nov 01 03:50:06.666029 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:06.665980    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d42cb7e-b420-4bcb-a086-8ce40cc24849-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d42cb7e-b420-4bcb-a086-8ce40cc24849-istio-envoy\") pod \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\" (UID: \"4d42cb7e-b420-4bcb-a086-8ce40cc24849\") "
Nov 01 03:50:06.666029 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:06.665987    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d42cb7e-b420-4bcb-a086-8ce40cc24849-workload-certs" pod "4d42cb7e-b420-4bcb-a086-8ce40cc24849" (UID: "4d42cb7e-b420-4bcb-a086-8ce40cc24849"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:50:06.666029 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:06.666019    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d42cb7e-b420-4bcb-a086-8ce40cc24849-istiod-ca-cert" pod "4d42cb7e-b420-4bcb-a086-8ce40cc24849" (UID: "4d42cb7e-b420-4bcb-a086-8ce40cc24849"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:50:06.666314 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:06.666064    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d42cb7e-b420-4bcb-a086-8ce40cc24849-credential-socket" pod "4d42cb7e-b420-4bcb-a086-8ce40cc24849" (UID: "4d42cb7e-b420-4bcb-a086-8ce40cc24849"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:50:06.666314 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:06.666240    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d42cb7e-b420-4bcb-a086-8ce40cc24849-istio-data" pod "4d42cb7e-b420-4bcb-a086-8ce40cc24849" (UID: "4d42cb7e-b420-4bcb-a086-8ce40cc24849"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:50:06.666314 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:06.666254    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d42cb7e-b420-4bcb-a086-8ce40cc24849-workload-socket" pod "4d42cb7e-b420-4bcb-a086-8ce40cc24849" (UID: "4d42cb7e-b420-4bcb-a086-8ce40cc24849"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:50:06.666678 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:06.666652    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d42cb7e-b420-4bcb-a086-8ce40cc24849-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:50:06.666781 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:06.666682    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d42cb7e-b420-4bcb-a086-8ce40cc24849-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:50:06.666781 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:06.666699    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d42cb7e-b420-4bcb-a086-8ce40cc24849-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:50:06.666781 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:06.666713    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d42cb7e-b420-4bcb-a086-8ce40cc24849-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:50:06.666781 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:06.666727    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d42cb7e-b420-4bcb-a086-8ce40cc24849-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:50:06.668052 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:06.668020    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d42cb7e-b420-4bcb-a086-8ce40cc24849-istio-envoy" pod "4d42cb7e-b420-4bcb-a086-8ce40cc24849" (UID: "4d42cb7e-b420-4bcb-a086-8ce40cc24849"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:50:06.668172 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:06.668137    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d42cb7e-b420-4bcb-a086-8ce40cc24849-istio-podinfo" pod "4d42cb7e-b420-4bcb-a086-8ce40cc24849" (UID: "4d42cb7e-b420-4bcb-a086-8ce40cc24849"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:50:06.668261 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:06.668242    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d42cb7e-b420-4bcb-a086-8ce40cc24849-istio-token" pod "4d42cb7e-b420-4bcb-a086-8ce40cc24849" (UID: "4d42cb7e-b420-4bcb-a086-8ce40cc24849"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:50:06.668315 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:06.668275    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d42cb7e-b420-4bcb-a086-8ce40cc24849-kube-api-access-hmgg6" pod "4d42cb7e-b420-4bcb-a086-8ce40cc24849" (UID: "4d42cb7e-b420-4bcb-a086-8ce40cc24849"). InnerVolumeSpecName "kube-api-access-hmgg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:50:06.767961 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:06.767920    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d42cb7e-b420-4bcb-a086-8ce40cc24849-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:50:06.767961 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:06.767955    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d42cb7e-b420-4bcb-a086-8ce40cc24849-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:50:06.768165 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:06.767969    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmgg6\" (UniqueName: \"kubernetes.io/projected/4d42cb7e-b420-4bcb-a086-8ce40cc24849-kube-api-access-hmgg6\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:50:06.768165 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:06.767984    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d42cb7e-b420-4bcb-a086-8ce40cc24849-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:50:07.408663 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:07.408631    4930 generic.go:356] "Generic (PLEG): container finished" podID="4d42cb7e-b420-4bcb-a086-8ce40cc24849" containerID="f0008cb73ee130a8a17f6189321b0f5da7589a3cd2bebb0571ca6785b0f754a3" exitCode=0
Nov 01 03:50:07.408847 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:07.408673    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq"
Nov 01 03:50:07.408847 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:07.408717    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq" event={"ID":"4d42cb7e-b420-4bcb-a086-8ce40cc24849","Type":"ContainerDied","Data":"f0008cb73ee130a8a17f6189321b0f5da7589a3cd2bebb0571ca6785b0f754a3"}
Nov 01 03:50:07.408847 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:07.408774    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq" event={"ID":"4d42cb7e-b420-4bcb-a086-8ce40cc24849","Type":"ContainerDied","Data":"06c7cacbe156e70b8eb172ec4981346e65a5464a9592d45547cde933fe6cc897"}
Nov 01 03:50:07.408847 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:07.408804    4930 scope.go:122] "RemoveContainer" containerID="f0008cb73ee130a8a17f6189321b0f5da7589a3cd2bebb0571ca6785b0f754a3"
Nov 01 03:50:07.420396 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:07.420381    4930 scope.go:122] "RemoveContainer" containerID="f0008cb73ee130a8a17f6189321b0f5da7589a3cd2bebb0571ca6785b0f754a3"
Nov 01 03:50:07.420753 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:50:07.420736    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0008cb73ee130a8a17f6189321b0f5da7589a3cd2bebb0571ca6785b0f754a3\": container with ID starting with f0008cb73ee130a8a17f6189321b0f5da7589a3cd2bebb0571ca6785b0f754a3 not found: ID does not exist" containerID="f0008cb73ee130a8a17f6189321b0f5da7589a3cd2bebb0571ca6785b0f754a3"
Nov 01 03:50:07.420805 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:07.420759    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0008cb73ee130a8a17f6189321b0f5da7589a3cd2bebb0571ca6785b0f754a3"} err="failed to get container status \"f0008cb73ee130a8a17f6189321b0f5da7589a3cd2bebb0571ca6785b0f754a3\": rpc error: code = NotFound desc = could not find container \"f0008cb73ee130a8a17f6189321b0f5da7589a3cd2bebb0571ca6785b0f754a3\": container with ID starting with f0008cb73ee130a8a17f6189321b0f5da7589a3cd2bebb0571ca6785b0f754a3 not found: ID does not exist"
Nov 01 03:50:07.577296 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:07.577265    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq"]
Nov 01 03:50:07.621380 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:07.621348    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cfeed06-fada-48eb-956b-317138fa24fa" path="/var/lib/kubelet/pods/8cfeed06-fada-48eb-956b-317138fa24fa/volumes"
Nov 01 03:50:07.624544 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:07.624523    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7f535-openshift-default-856bb56cf-jzvzq"]
Nov 01 03:50:09.621778 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:09.621746    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d42cb7e-b420-4bcb-a086-8ce40cc24849" path="/var/lib/kubelet/pods/4d42cb7e-b420-4bcb-a086-8ce40cc24849/volumes"
Nov 01 03:50:10.333632 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:10.333598    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:50:12.157269 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:12.157233    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:50:12.314265 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:12.314229    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:50:12.511816 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:12.511285    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:50:13.389753 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:13.389718    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a279c2-hktzw"]
Nov 01 03:50:13.390272 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:13.390256    4930 cpu_manager.go:414] "containerMap: removing container" podUID="c331aa42-00f4-4819-8efe-679636066bd7" containerName="revoke-keys"
Nov 01 03:50:13.390272 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:13.390272    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c331aa42-00f4-4819-8efe-679636066bd7" containerName="revoke-keys"
Nov 01 03:50:13.390410 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:13.390295    4930 cpu_manager.go:414] "containerMap: removing container" podUID="4d42cb7e-b420-4bcb-a086-8ce40cc24849" containerName="istio-proxy"
Nov 01 03:50:13.390410 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:13.390300    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d42cb7e-b420-4bcb-a086-8ce40cc24849" containerName="istio-proxy"
Nov 01 03:50:13.390410 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:13.390307    4930 cpu_manager.go:414] "containerMap: removing container" podUID="27e7c281-1391-4406-9dfa-d770c821678e" containerName="istio-proxy"
Nov 01 03:50:13.390410 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:13.390313    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27e7c281-1391-4406-9dfa-d770c821678e" containerName="istio-proxy"
Nov 01 03:50:13.390410 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:13.390325    4930 cpu_manager.go:414] "containerMap: removing container" podUID="27e7c281-1391-4406-9dfa-d770c821678e" containerName="istio-proxy"
Nov 01 03:50:13.390410 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:13.390330    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27e7c281-1391-4406-9dfa-d770c821678e" containerName="istio-proxy"
Nov 01 03:50:13.390760 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:13.390415    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="c331aa42-00f4-4819-8efe-679636066bd7" containerName="revoke-keys"
Nov 01 03:50:13.390760 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:13.390428    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d42cb7e-b420-4bcb-a086-8ce40cc24849" containerName="istio-proxy"
Nov 01 03:50:13.390760 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:13.390433    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="27e7c281-1391-4406-9dfa-d770c821678e" containerName="istio-proxy"
Nov 01 03:50:13.390760 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:13.390441    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="27e7c281-1391-4406-9dfa-d770c821678e" containerName="istio-proxy"
Nov 01 03:50:13.395297 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:13.395276    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a279c2-hktzw"
Nov 01 03:50:13.447086 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:13.447062    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q6s9\" (UniqueName: \"kubernetes.io/projected/8bf9eec1-aa20-4211-8a26-9fa2058f836d-kube-api-access-9q6s9\") pod \"maas-api-revoke-keys-e2e-api-b-a279c2-hktzw\" (UID: \"8bf9eec1-aa20-4211-8a26-9fa2058f836d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a279c2-hktzw"
Nov 01 03:50:13.447249 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:13.447143    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf9eec1-aa20-4211-8a26-9fa2058f836d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a279c2-hktzw\" (UID: \"8bf9eec1-aa20-4211-8a26-9fa2058f836d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a279c2-hktzw"
Nov 01 03:50:13.447325 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:13.447291    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf9eec1-aa20-4211-8a26-9fa2058f836d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a279c2-hktzw\" (UID: \"8bf9eec1-aa20-4211-8a26-9fa2058f836d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a279c2-hktzw"
Nov 01 03:50:13.455333 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:13.455311    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a279c2-hktzw"]
Nov 01 03:50:13.499768 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:13.499741    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:50:13.499953 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:13.499821    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:50:13.499953 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:13.499868    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:50:13.500042 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:13.499961    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:50:13.548959 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:13.548935    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q6s9\" (UniqueName: \"kubernetes.io/projected/8bf9eec1-aa20-4211-8a26-9fa2058f836d-kube-api-access-9q6s9\") pod \"maas-api-revoke-keys-e2e-api-b-a279c2-hktzw\" (UID: \"8bf9eec1-aa20-4211-8a26-9fa2058f836d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a279c2-hktzw"
Nov 01 03:50:13.549112 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:13.548999    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf9eec1-aa20-4211-8a26-9fa2058f836d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a279c2-hktzw\" (UID: \"8bf9eec1-aa20-4211-8a26-9fa2058f836d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a279c2-hktzw"
Nov 01 03:50:13.549112 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:13.549087    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf9eec1-aa20-4211-8a26-9fa2058f836d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a279c2-hktzw\" (UID: \"8bf9eec1-aa20-4211-8a26-9fa2058f836d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a279c2-hktzw"
Nov 01 03:50:13.549811 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:13.549787    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf9eec1-aa20-4211-8a26-9fa2058f836d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a279c2-hktzw\" (UID: \"8bf9eec1-aa20-4211-8a26-9fa2058f836d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a279c2-hktzw"
Nov 01 03:50:13.551472 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:13.551452    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf9eec1-aa20-4211-8a26-9fa2058f836d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a279c2-hktzw\" (UID: \"8bf9eec1-aa20-4211-8a26-9fa2058f836d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a279c2-hktzw"
Nov 01 03:50:13.610222 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:13.610195    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q6s9\" (UniqueName: \"kubernetes.io/projected/8bf9eec1-aa20-4211-8a26-9fa2058f836d-kube-api-access-9q6s9\") pod \"maas-api-revoke-keys-e2e-api-b-a279c2-hktzw\" (UID: \"8bf9eec1-aa20-4211-8a26-9fa2058f836d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a279c2-hktzw"
Nov 01 03:50:13.711237 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:13.711151    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a279c2-hktzw"
Nov 01 03:50:14.100588 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:14.100544    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a279c2-hktzw"]
Nov 01 03:50:14.101173 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:50:14.101143    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf9eec1_aa20_4211_8a26_9fa2058f836d.slice/crio-c2f9605a5231add8bd08d6f0512219b1d1ad2f2437bfa907367ae5ca07022c5a WatchSource:0}: Error finding container c2f9605a5231add8bd08d6f0512219b1d1ad2f2437bfa907367ae5ca07022c5a: Status 404 returned error can't find the container with id c2f9605a5231add8bd08d6f0512219b1d1ad2f2437bfa907367ae5ca07022c5a
Nov 01 03:50:14.464263 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:14.464224    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a279c2-hktzw" event={"ID":"8bf9eec1-aa20-4211-8a26-9fa2058f836d","Type":"ContainerStarted","Data":"c2f9605a5231add8bd08d6f0512219b1d1ad2f2437bfa907367ae5ca07022c5a"}
Nov 01 03:50:14.816471 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:14.816381    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:50:15.475490 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:15.475401    4930 generic.go:356] "Generic (PLEG): container finished" podID="8bf9eec1-aa20-4211-8a26-9fa2058f836d" containerID="65e9cfbf234ed42b6b9ec7305dc013529e56f1a3869016104a316b576c178dff" exitCode=0
Nov 01 03:50:15.475490 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:15.475474    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a279c2-hktzw" event={"ID":"8bf9eec1-aa20-4211-8a26-9fa2058f836d","Type":"ContainerDied","Data":"65e9cfbf234ed42b6b9ec7305dc013529e56f1a3869016104a316b576c178dff"}
Nov 01 03:50:16.431018 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:16.430974    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:50:16.648258 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:16.648232    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a279c2-hktzw"
Nov 01 03:50:16.705493 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:16.705419    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:50:16.814624 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:16.808893    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bf9eec1-aa20-4211-8a26-9fa2058f836d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf9eec1-aa20-4211-8a26-9fa2058f836d-maas-api-openshift-ca\") pod \"8bf9eec1-aa20-4211-8a26-9fa2058f836d\" (UID: \"8bf9eec1-aa20-4211-8a26-9fa2058f836d\") "
Nov 01 03:50:16.814624 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:16.809001    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bf9eec1-aa20-4211-8a26-9fa2058f836d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf9eec1-aa20-4211-8a26-9fa2058f836d-maas-api-xks-ca\") pod \"8bf9eec1-aa20-4211-8a26-9fa2058f836d\" (UID: \"8bf9eec1-aa20-4211-8a26-9fa2058f836d\") "
Nov 01 03:50:16.814624 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:16.809212    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bf9eec1-aa20-4211-8a26-9fa2058f836d-kube-api-access-9q6s9\" (UniqueName: \"kubernetes.io/projected/8bf9eec1-aa20-4211-8a26-9fa2058f836d-kube-api-access-9q6s9\") pod \"8bf9eec1-aa20-4211-8a26-9fa2058f836d\" (UID: \"8bf9eec1-aa20-4211-8a26-9fa2058f836d\") "
Nov 01 03:50:16.814624 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:16.814118    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bf9eec1-aa20-4211-8a26-9fa2058f836d-maas-api-xks-ca" pod "8bf9eec1-aa20-4211-8a26-9fa2058f836d" (UID: "8bf9eec1-aa20-4211-8a26-9fa2058f836d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:50:16.817798 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:16.815573    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bf9eec1-aa20-4211-8a26-9fa2058f836d-maas-api-openshift-ca" pod "8bf9eec1-aa20-4211-8a26-9fa2058f836d" (UID: "8bf9eec1-aa20-4211-8a26-9fa2058f836d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:50:16.827436 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:16.821218    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf9eec1-aa20-4211-8a26-9fa2058f836d-kube-api-access-9q6s9" pod "8bf9eec1-aa20-4211-8a26-9fa2058f836d" (UID: "8bf9eec1-aa20-4211-8a26-9fa2058f836d"). InnerVolumeSpecName "kube-api-access-9q6s9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:50:16.855531 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:16.855497    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:50:16.911314 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:16.911289    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bf9eec1-aa20-4211-8a26-9fa2058f836d-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:50:16.911314 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:16.911315    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bf9eec1-aa20-4211-8a26-9fa2058f836d-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:50:16.911460 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:16.911325    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q6s9\" (UniqueName: \"kubernetes.io/projected/8bf9eec1-aa20-4211-8a26-9fa2058f836d-kube-api-access-9q6s9\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:50:16.937599 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:16.937478    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2906cb-c788n"]
Nov 01 03:50:16.939486 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:16.939460    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2906cb-c788n"]
Nov 01 03:50:17.367954 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:17.367902    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:50:17.493516 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:17.493486    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a279c2-hktzw"
Nov 01 03:50:17.493516 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:17.493512    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a279c2-hktzw" event={"ID":"8bf9eec1-aa20-4211-8a26-9fa2058f836d","Type":"ContainerDied","Data":"c2f9605a5231add8bd08d6f0512219b1d1ad2f2437bfa907367ae5ca07022c5a"}
Nov 01 03:50:17.493811 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:17.493620    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2f9605a5231add8bd08d6f0512219b1d1ad2f2437bfa907367ae5ca07022c5a"
Nov 01 03:50:17.620395 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:17.620321    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e552bc78-982d-4ce6-97c1-693fbdfa46e5" path="/var/lib/kubelet/pods/e552bc78-982d-4ce6-97c1-693fbdfa46e5/volumes"
Nov 01 03:50:19.122500 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:50:19.122463    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:50:21.875700 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:21.875660    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:50:23.615769 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:23.615736    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:50:23.775422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:23.775388    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:50:23.977337 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:23.977256    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:50:24.770045 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:24.770009    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-170ffa-wpl7d"]
Nov 01 03:50:24.770588 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:24.770570    4930 cpu_manager.go:414] "containerMap: removing container" podUID="8bf9eec1-aa20-4211-8a26-9fa2058f836d" containerName="revoke-keys"
Nov 01 03:50:24.770659 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:24.770590    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bf9eec1-aa20-4211-8a26-9fa2058f836d" containerName="revoke-keys"
Nov 01 03:50:24.770728 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:24.770718    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bf9eec1-aa20-4211-8a26-9fa2058f836d" containerName="revoke-keys"
Nov 01 03:50:24.775412 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:24.775393    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-170ffa-wpl7d"
Nov 01 03:50:24.833962 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:24.833935    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-170ffa-wpl7d"]
Nov 01 03:50:24.879622 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:24.879593    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:50:24.879622 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:24.879617    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:50:24.879622 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:24.879594    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:50:24.879846 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:24.879660    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:50:24.914167 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:24.914141    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11338423-6bc3-42d6-8384-b438393fcda7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-170ffa-wpl7d\" (UID: \"11338423-6bc3-42d6-8384-b438393fcda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-170ffa-wpl7d"
Nov 01 03:50:24.914324 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:24.914180    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbl6c\" (UniqueName: \"kubernetes.io/projected/11338423-6bc3-42d6-8384-b438393fcda7-kube-api-access-nbl6c\") pod \"maas-api-revoke-keys-e2e-api-b-170ffa-wpl7d\" (UID: \"11338423-6bc3-42d6-8384-b438393fcda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-170ffa-wpl7d"
Nov 01 03:50:24.914398 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:24.914366    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11338423-6bc3-42d6-8384-b438393fcda7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-170ffa-wpl7d\" (UID: \"11338423-6bc3-42d6-8384-b438393fcda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-170ffa-wpl7d"
Nov 01 03:50:25.015773 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:25.015737    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11338423-6bc3-42d6-8384-b438393fcda7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-170ffa-wpl7d\" (UID: \"11338423-6bc3-42d6-8384-b438393fcda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-170ffa-wpl7d"
Nov 01 03:50:25.015965 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:25.015789    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbl6c\" (UniqueName: \"kubernetes.io/projected/11338423-6bc3-42d6-8384-b438393fcda7-kube-api-access-nbl6c\") pod \"maas-api-revoke-keys-e2e-api-b-170ffa-wpl7d\" (UID: \"11338423-6bc3-42d6-8384-b438393fcda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-170ffa-wpl7d"
Nov 01 03:50:25.015965 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:25.015862    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11338423-6bc3-42d6-8384-b438393fcda7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-170ffa-wpl7d\" (UID: \"11338423-6bc3-42d6-8384-b438393fcda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-170ffa-wpl7d"
Nov 01 03:50:25.016438 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:25.016416    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11338423-6bc3-42d6-8384-b438393fcda7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-170ffa-wpl7d\" (UID: \"11338423-6bc3-42d6-8384-b438393fcda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-170ffa-wpl7d"
Nov 01 03:50:25.018300 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:25.018279    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11338423-6bc3-42d6-8384-b438393fcda7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-170ffa-wpl7d\" (UID: \"11338423-6bc3-42d6-8384-b438393fcda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-170ffa-wpl7d"
Nov 01 03:50:25.076691 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:25.076626    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbl6c\" (UniqueName: \"kubernetes.io/projected/11338423-6bc3-42d6-8384-b438393fcda7-kube-api-access-nbl6c\") pod \"maas-api-revoke-keys-e2e-api-b-170ffa-wpl7d\" (UID: \"11338423-6bc3-42d6-8384-b438393fcda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-170ffa-wpl7d"
Nov 01 03:50:25.090409 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:25.090388    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-170ffa-wpl7d"
Nov 01 03:50:25.483264 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:25.483222    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-170ffa-wpl7d"]
Nov 01 03:50:25.553529 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:25.553501    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-170ffa-wpl7d" event={"ID":"11338423-6bc3-42d6-8384-b438393fcda7","Type":"ContainerStarted","Data":"63e6cb1f097b493abfa2bbe66c1465911cf286b4a733e1c192f6ca52f5248898"}
Nov 01 03:50:26.113265 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:26.113235    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a279c2-df99cdff7-rm7hx"]
Nov 01 03:50:26.224113 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:26.224076    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a279c2-7ffb8d6599-z857m"]
Nov 01 03:50:26.495350 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:26.495297    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:50:26.574095 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:26.574054    4930 generic.go:356] "Generic (PLEG): container finished" podID="11338423-6bc3-42d6-8384-b438393fcda7" containerID="9b6263b88681d17549a2032aecf1a8f66b4f902d664c7c729e3c29b3e9273d67" exitCode=0
Nov 01 03:50:26.574474 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:26.574454    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-170ffa-wpl7d" event={"ID":"11338423-6bc3-42d6-8384-b438393fcda7","Type":"ContainerDied","Data":"9b6263b88681d17549a2032aecf1a8f66b4f902d664c7c729e3c29b3e9273d67"}
Nov 01 03:50:26.648958 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:26.648853    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:50:27.035158 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:27.035074    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:50:27.867457 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:27.867401    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-170ffa-wpl7d"
Nov 01 03:50:28.007758 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:28.007673    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11338423-6bc3-42d6-8384-b438393fcda7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11338423-6bc3-42d6-8384-b438393fcda7-maas-api-xks-ca\") pod \"11338423-6bc3-42d6-8384-b438393fcda7\" (UID: \"11338423-6bc3-42d6-8384-b438393fcda7\") "
Nov 01 03:50:28.007758 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:28.007749    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11338423-6bc3-42d6-8384-b438393fcda7-kube-api-access-nbl6c\" (UniqueName: \"kubernetes.io/projected/11338423-6bc3-42d6-8384-b438393fcda7-kube-api-access-nbl6c\") pod \"11338423-6bc3-42d6-8384-b438393fcda7\" (UID: \"11338423-6bc3-42d6-8384-b438393fcda7\") "
Nov 01 03:50:28.007998 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:28.007894    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11338423-6bc3-42d6-8384-b438393fcda7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11338423-6bc3-42d6-8384-b438393fcda7-maas-api-openshift-ca\") pod \"11338423-6bc3-42d6-8384-b438393fcda7\" (UID: \"11338423-6bc3-42d6-8384-b438393fcda7\") "
Nov 01 03:50:28.008750 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:28.008704    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11338423-6bc3-42d6-8384-b438393fcda7-maas-api-openshift-ca" pod "11338423-6bc3-42d6-8384-b438393fcda7" (UID: "11338423-6bc3-42d6-8384-b438393fcda7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:50:28.010808 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:28.010770    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11338423-6bc3-42d6-8384-b438393fcda7-maas-api-xks-ca" pod "11338423-6bc3-42d6-8384-b438393fcda7" (UID: "11338423-6bc3-42d6-8384-b438393fcda7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:50:28.018708 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:28.018679    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11338423-6bc3-42d6-8384-b438393fcda7-kube-api-access-nbl6c" pod "11338423-6bc3-42d6-8384-b438393fcda7" (UID: "11338423-6bc3-42d6-8384-b438393fcda7"). InnerVolumeSpecName "kube-api-access-nbl6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:50:28.109116 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:28.109073    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11338423-6bc3-42d6-8384-b438393fcda7-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:50:28.109116 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:28.109105    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbl6c\" (UniqueName: \"kubernetes.io/projected/11338423-6bc3-42d6-8384-b438393fcda7-kube-api-access-nbl6c\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:50:28.109116 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:28.109116    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11338423-6bc3-42d6-8384-b438393fcda7-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:50:28.643139 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:28.643057    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-170ffa-wpl7d"
Nov 01 03:50:28.643139 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:28.643089    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-170ffa-wpl7d" event={"ID":"11338423-6bc3-42d6-8384-b438393fcda7","Type":"ContainerDied","Data":"63e6cb1f097b493abfa2bbe66c1465911cf286b4a733e1c192f6ca52f5248898"}
Nov 01 03:50:28.643139 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:28.643137    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63e6cb1f097b493abfa2bbe66c1465911cf286b4a733e1c192f6ca52f5248898"
Nov 01 03:50:31.113759 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:31.113703    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-a279c2-df99cdff7-rm7hx" podUID="532b2472-d6b3-4af0-905b-98953c8b814c" containerName="payload-processing" containerID="cri-o://3d2f0b2d180c06191df3222722f56c32cf1d2927a399a0a7cd1089fd39538f3f" gracePeriod=25
Nov 01 03:50:31.224626 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:31.224592    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a279c2-7ffb8d6599-z857m" podUID="bd58da84-e12c-4bf4-a64c-d3b7155f57fa" containerName="payload-pre-processing" containerID="cri-o://5e688d8c3244479861e7209362935483f0cea38fd55e1c9e7314760d69c7906f" gracePeriod=25
Nov 01 03:50:31.385643 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:31.385613    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a279c2-df99cdff7-rm7hx"
Nov 01 03:50:31.449408 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:31.449376    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/532b2472-d6b3-4af0-905b-98953c8b814c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/532b2472-d6b3-4af0-905b-98953c8b814c-plugins-config-volume\") pod \"532b2472-d6b3-4af0-905b-98953c8b814c\" (UID: \"532b2472-d6b3-4af0-905b-98953c8b814c\") "
Nov 01 03:50:31.449598 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:31.449437    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/532b2472-d6b3-4af0-905b-98953c8b814c-kube-api-access-5srjn\" (UniqueName: \"kubernetes.io/projected/532b2472-d6b3-4af0-905b-98953c8b814c-kube-api-access-5srjn\") pod \"532b2472-d6b3-4af0-905b-98953c8b814c\" (UID: \"532b2472-d6b3-4af0-905b-98953c8b814c\") "
Nov 01 03:50:31.449837 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:31.449805    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/532b2472-d6b3-4af0-905b-98953c8b814c-plugins-config-volume" pod "532b2472-d6b3-4af0-905b-98953c8b814c" (UID: "532b2472-d6b3-4af0-905b-98953c8b814c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:50:31.449976 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:31.449957    4930 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/532b2472-d6b3-4af0-905b-98953c8b814c-plugins-config-volume\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:50:31.452630 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:31.452606    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/532b2472-d6b3-4af0-905b-98953c8b814c-kube-api-access-5srjn" pod "532b2472-d6b3-4af0-905b-98953c8b814c" (UID: "532b2472-d6b3-4af0-905b-98953c8b814c"). InnerVolumeSpecName "kube-api-access-5srjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:50:31.492035 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:31.492010    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a279c2-7ffb8d6599-z857m"
Nov 01 03:50:31.551137 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:31.551107    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd58da84-e12c-4bf4-a64c-d3b7155f57fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd58da84-e12c-4bf4-a64c-d3b7155f57fa-plugins-config-volume\") pod \"bd58da84-e12c-4bf4-a64c-d3b7155f57fa\" (UID: \"bd58da84-e12c-4bf4-a64c-d3b7155f57fa\") "
Nov 01 03:50:31.551342 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:31.551149    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd58da84-e12c-4bf4-a64c-d3b7155f57fa-kube-api-access-8bzff\" (UniqueName: \"kubernetes.io/projected/bd58da84-e12c-4bf4-a64c-d3b7155f57fa-kube-api-access-8bzff\") pod \"bd58da84-e12c-4bf4-a64c-d3b7155f57fa\" (UID: \"bd58da84-e12c-4bf4-a64c-d3b7155f57fa\") "
Nov 01 03:50:31.551547 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:31.551523    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd58da84-e12c-4bf4-a64c-d3b7155f57fa-plugins-config-volume" pod "bd58da84-e12c-4bf4-a64c-d3b7155f57fa" (UID: "bd58da84-e12c-4bf4-a64c-d3b7155f57fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:50:31.551702 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:31.551685    4930 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd58da84-e12c-4bf4-a64c-d3b7155f57fa-plugins-config-volume\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:50:31.551774 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:31.551709    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5srjn\" (UniqueName: \"kubernetes.io/projected/532b2472-d6b3-4af0-905b-98953c8b814c-kube-api-access-5srjn\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:50:31.553377 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:31.553351    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd58da84-e12c-4bf4-a64c-d3b7155f57fa-kube-api-access-8bzff" pod "bd58da84-e12c-4bf4-a64c-d3b7155f57fa" (UID: "bd58da84-e12c-4bf4-a64c-d3b7155f57fa"). InnerVolumeSpecName "kube-api-access-8bzff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:50:31.653086 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:31.653034    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bzff\" (UniqueName: \"kubernetes.io/projected/bd58da84-e12c-4bf4-a64c-d3b7155f57fa-kube-api-access-8bzff\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:50:31.667506 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:31.667475    4930 generic.go:356] "Generic (PLEG): container finished" podID="532b2472-d6b3-4af0-905b-98953c8b814c" containerID="3d2f0b2d180c06191df3222722f56c32cf1d2927a399a0a7cd1089fd39538f3f" exitCode=0
Nov 01 03:50:31.667658 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:31.667518    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a279c2-df99cdff7-rm7hx"
Nov 01 03:50:31.667658 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:31.667534    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a279c2-df99cdff7-rm7hx" event={"ID":"532b2472-d6b3-4af0-905b-98953c8b814c","Type":"ContainerDied","Data":"3d2f0b2d180c06191df3222722f56c32cf1d2927a399a0a7cd1089fd39538f3f"}
Nov 01 03:50:31.667658 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:31.667608    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a279c2-df99cdff7-rm7hx" event={"ID":"532b2472-d6b3-4af0-905b-98953c8b814c","Type":"ContainerDied","Data":"1d43fb76d6fff4535accd8b247a4f8cff6dea83276f50a35d70fb4814f93b39f"}
Nov 01 03:50:31.667658 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:31.667637    4930 scope.go:122] "RemoveContainer" containerID="3d2f0b2d180c06191df3222722f56c32cf1d2927a399a0a7cd1089fd39538f3f"
Nov 01 03:50:31.669666 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:31.669465    4930 generic.go:356] "Generic (PLEG): container finished" podID="bd58da84-e12c-4bf4-a64c-d3b7155f57fa" containerID="5e688d8c3244479861e7209362935483f0cea38fd55e1c9e7314760d69c7906f" exitCode=0
Nov 01 03:50:31.669666 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:31.669547    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a279c2-7ffb8d6599-z857m" event={"ID":"bd58da84-e12c-4bf4-a64c-d3b7155f57fa","Type":"ContainerDied","Data":"5e688d8c3244479861e7209362935483f0cea38fd55e1c9e7314760d69c7906f"}
Nov 01 03:50:31.669666 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:31.669631    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a279c2-7ffb8d6599-z857m" event={"ID":"bd58da84-e12c-4bf4-a64c-d3b7155f57fa","Type":"ContainerDied","Data":"3fd0f0a22a7590b666ea12a1349d2c85a67c1689cc4b402311bf5d3b970e655f"}
Nov 01 03:50:31.669894 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:31.669874    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a279c2-7ffb8d6599-z857m"
Nov 01 03:50:31.678908 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:31.678891    4930 scope.go:122] "RemoveContainer" containerID="3d2f0b2d180c06191df3222722f56c32cf1d2927a399a0a7cd1089fd39538f3f"
Nov 01 03:50:31.679166 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:50:31.679149    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d2f0b2d180c06191df3222722f56c32cf1d2927a399a0a7cd1089fd39538f3f\": container with ID starting with 3d2f0b2d180c06191df3222722f56c32cf1d2927a399a0a7cd1089fd39538f3f not found: ID does not exist" containerID="3d2f0b2d180c06191df3222722f56c32cf1d2927a399a0a7cd1089fd39538f3f"
Nov 01 03:50:31.679211 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:31.679174    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d2f0b2d180c06191df3222722f56c32cf1d2927a399a0a7cd1089fd39538f3f"} err="failed to get container status \"3d2f0b2d180c06191df3222722f56c32cf1d2927a399a0a7cd1089fd39538f3f\": rpc error: code = NotFound desc = could not find container \"3d2f0b2d180c06191df3222722f56c32cf1d2927a399a0a7cd1089fd39538f3f\": container with ID starting with 3d2f0b2d180c06191df3222722f56c32cf1d2927a399a0a7cd1089fd39538f3f not found: ID does not exist"
Nov 01 03:50:31.679211 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:31.679197    4930 scope.go:122] "RemoveContainer" containerID="5e688d8c3244479861e7209362935483f0cea38fd55e1c9e7314760d69c7906f"
Nov 01 03:50:31.689249 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:31.689231    4930 scope.go:122] "RemoveContainer" containerID="5e688d8c3244479861e7209362935483f0cea38fd55e1c9e7314760d69c7906f"
Nov 01 03:50:31.689544 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:50:31.689523    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e688d8c3244479861e7209362935483f0cea38fd55e1c9e7314760d69c7906f\": container with ID starting with 5e688d8c3244479861e7209362935483f0cea38fd55e1c9e7314760d69c7906f not found: ID does not exist" containerID="5e688d8c3244479861e7209362935483f0cea38fd55e1c9e7314760d69c7906f"
Nov 01 03:50:31.689629 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:31.689548    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e688d8c3244479861e7209362935483f0cea38fd55e1c9e7314760d69c7906f"} err="failed to get container status \"5e688d8c3244479861e7209362935483f0cea38fd55e1c9e7314760d69c7906f\": rpc error: code = NotFound desc = could not find container \"5e688d8c3244479861e7209362935483f0cea38fd55e1c9e7314760d69c7906f\": container with ID starting with 5e688d8c3244479861e7209362935483f0cea38fd55e1c9e7314760d69c7906f not found: ID does not exist"
Nov 01 03:50:31.832581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:31.832520    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a279c2-7ffb8d6599-z857m"]
Nov 01 03:50:31.837175 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:31.837148    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a279c2-7ffb8d6599-z857m"]
Nov 01 03:50:31.946237 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:31.946163    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a279c2-df99cdff7-rm7hx"]
Nov 01 03:50:31.950361 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:31.950322    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a279c2-df99cdff7-rm7hx"]
Nov 01 03:50:32.239388 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:32.239315    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:50:33.621214 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:33.621180    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="532b2472-d6b3-4af0-905b-98953c8b814c" path="/var/lib/kubelet/pods/532b2472-d6b3-4af0-905b-98953c8b814c/volumes"
Nov 01 03:50:33.621643 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:33.621629    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd58da84-e12c-4bf4-a64c-d3b7155f57fa" path="/var/lib/kubelet/pods/bd58da84-e12c-4bf4-a64c-d3b7155f57fa/volumes"
Nov 01 03:50:36.244992 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:36.244957    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-170ffa-75c44b9ddc-grjt8"]
Nov 01 03:50:36.609230 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:36.609190    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:50:36.810575 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:36.809917    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:50:37.236960 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:37.236913    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:50:39.887790 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:39.887761    4930 scope.go:122] "RemoveContainer" containerID="4ac4caa3b804083627610a790dcedfca430ff1b898c4dac090cd93df22347b15"
Nov 01 03:50:39.900901 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:39.900876    4930 scope.go:122] "RemoveContainer" containerID="bd1d5108a790a24b8b2cfda23be55d7fe39898f4f1f0522dfd08a9d9d03da88d"
Nov 01 03:50:39.913338 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:39.913313    4930 scope.go:122] "RemoveContainer" containerID="eead74419348327a647a938771156a9cea748891cd0efe1e8c209977f555aa20"
Nov 01 03:50:39.925548 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:39.925523    4930 scope.go:122] "RemoveContainer" containerID="e91ffd6b25481f961a4ba7d3c33a46664e02aa7f7d5feac51b586a758cd6d21d"
Nov 01 03:50:41.245486 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:41.245447    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-170ffa-75c44b9ddc-grjt8" podUID="86df4787-7837-437c-a76d-2bf797b9242f" containerName="payload-processing" containerID="cri-o://eba404c37d24cd2b883c784b477b212eea119cb2b2ef6cc004328e927d6ae774" gracePeriod=25
Nov 01 03:50:41.532573 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:41.532527    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-170ffa-75c44b9ddc-grjt8"
Nov 01 03:50:41.690801 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:41.690770    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86df4787-7837-437c-a76d-2bf797b9242f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86df4787-7837-437c-a76d-2bf797b9242f-plugins-config-volume\") pod \"86df4787-7837-437c-a76d-2bf797b9242f\" (UID: \"86df4787-7837-437c-a76d-2bf797b9242f\") "
Nov 01 03:50:41.690972 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:41.690937    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86df4787-7837-437c-a76d-2bf797b9242f-kube-api-access-4487s\" (UniqueName: \"kubernetes.io/projected/86df4787-7837-437c-a76d-2bf797b9242f-kube-api-access-4487s\") pod \"86df4787-7837-437c-a76d-2bf797b9242f\" (UID: \"86df4787-7837-437c-a76d-2bf797b9242f\") "
Nov 01 03:50:41.691209 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:41.691178    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86df4787-7837-437c-a76d-2bf797b9242f-plugins-config-volume" pod "86df4787-7837-437c-a76d-2bf797b9242f" (UID: "86df4787-7837-437c-a76d-2bf797b9242f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:50:41.691394 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:41.691369    4930 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86df4787-7837-437c-a76d-2bf797b9242f-plugins-config-volume\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:50:41.693231 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:41.693200    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86df4787-7837-437c-a76d-2bf797b9242f-kube-api-access-4487s" pod "86df4787-7837-437c-a76d-2bf797b9242f" (UID: "86df4787-7837-437c-a76d-2bf797b9242f"). InnerVolumeSpecName "kube-api-access-4487s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:50:41.771022 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:41.770937    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:50:41.789515 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:41.789488    4930 generic.go:356] "Generic (PLEG): container finished" podID="86df4787-7837-437c-a76d-2bf797b9242f" containerID="eba404c37d24cd2b883c784b477b212eea119cb2b2ef6cc004328e927d6ae774" exitCode=0
Nov 01 03:50:41.789676 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:41.789579    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-170ffa-75c44b9ddc-grjt8"
Nov 01 03:50:41.789676 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:41.789581    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-170ffa-75c44b9ddc-grjt8" event={"ID":"86df4787-7837-437c-a76d-2bf797b9242f","Type":"ContainerDied","Data":"eba404c37d24cd2b883c784b477b212eea119cb2b2ef6cc004328e927d6ae774"}
Nov 01 03:50:41.789806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:41.789700    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-170ffa-75c44b9ddc-grjt8" event={"ID":"86df4787-7837-437c-a76d-2bf797b9242f","Type":"ContainerDied","Data":"d2c864e4ff2903ead7d1d3e9a80a129ab270c60bb10dee66671bc0cc0ba986ab"}
Nov 01 03:50:41.789806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:41.789722    4930 scope.go:122] "RemoveContainer" containerID="eba404c37d24cd2b883c784b477b212eea119cb2b2ef6cc004328e927d6ae774"
Nov 01 03:50:41.792647 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:41.792623    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4487s\" (UniqueName: \"kubernetes.io/projected/86df4787-7837-437c-a76d-2bf797b9242f-kube-api-access-4487s\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:50:41.799926 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:41.799910    4930 scope.go:122] "RemoveContainer" containerID="eba404c37d24cd2b883c784b477b212eea119cb2b2ef6cc004328e927d6ae774"
Nov 01 03:50:41.800213 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:50:41.800194    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eba404c37d24cd2b883c784b477b212eea119cb2b2ef6cc004328e927d6ae774\": container with ID starting with eba404c37d24cd2b883c784b477b212eea119cb2b2ef6cc004328e927d6ae774 not found: ID does not exist" containerID="eba404c37d24cd2b883c784b477b212eea119cb2b2ef6cc004328e927d6ae774"
Nov 01 03:50:41.800267 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:41.800221    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eba404c37d24cd2b883c784b477b212eea119cb2b2ef6cc004328e927d6ae774"} err="failed to get container status \"eba404c37d24cd2b883c784b477b212eea119cb2b2ef6cc004328e927d6ae774\": rpc error: code = NotFound desc = could not find container \"eba404c37d24cd2b883c784b477b212eea119cb2b2ef6cc004328e927d6ae774\": container with ID starting with eba404c37d24cd2b883c784b477b212eea119cb2b2ef6cc004328e927d6ae774 not found: ID does not exist"
Nov 01 03:50:41.957733 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:41.957698    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-170ffa-75c44b9ddc-grjt8"]
Nov 01 03:50:41.964345 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:41.964320    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-170ffa-75c44b9ddc-grjt8"]
Nov 01 03:50:42.048741 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:42.048651    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:50:43.620906 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:43.620865    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86df4787-7837-437c-a76d-2bf797b9242f" path="/var/lib/kubelet/pods/86df4787-7837-437c-a76d-2bf797b9242f/volumes"
Nov 01 03:50:46.766768 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:46.766686    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:50:49.123252 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:50:49.123213    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:50:51.251026 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:51.250991    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:50:51.535493 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:51.535407    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:50:56.262473 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:50:56.262432    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:51:01.044849 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:01.044814    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:51:05.197291 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:05.197250    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:51:06.721570 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:06.721533    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:51:06.955273 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:06.955241    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:51:07.105666 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:07.105633    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:51:07.191889 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:07.191860    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvgwp"]
Nov 01 03:51:07.192092 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:07.192073    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvgwp" podUID="c7161bfc-1b3c-4245-a0e7-c35a7fe69be1" containerName="discovery" containerID="cri-o://6c988adf0b9527638cb030cd350cd466a7e507ebf99dbd3c3ba94f0d76905966" gracePeriod=30
Nov 01 03:51:07.240871 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:07.240839    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-417aa8-69hzv"]
Nov 01 03:51:07.290095 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:07.290068    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-417aa8-69hzv"]
Nov 01 03:51:07.469327 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:07.469301    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvgwp"
Nov 01 03:51:07.537903 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:07.537875    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-kube-api-access-m7j7l\" (UniqueName: \"kubernetes.io/projected/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-kube-api-access-m7j7l\") pod \"c7161bfc-1b3c-4245-a0e7-c35a7fe69be1\" (UID: \"c7161bfc-1b3c-4245-a0e7-c35a7fe69be1\") "
Nov 01 03:51:07.538077 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:07.537941    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-istio-token\" (UniqueName: \"kubernetes.io/projected/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-istio-token\") pod \"c7161bfc-1b3c-4245-a0e7-c35a7fe69be1\" (UID: \"c7161bfc-1b3c-4245-a0e7-c35a7fe69be1\") "
Nov 01 03:51:07.538077 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:07.538024    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-istio-csr-ca-configmap\") pod \"c7161bfc-1b3c-4245-a0e7-c35a7fe69be1\" (UID: \"c7161bfc-1b3c-4245-a0e7-c35a7fe69be1\") "
Nov 01 03:51:07.538208 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:07.538153    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-local-certs\") pod \"c7161bfc-1b3c-4245-a0e7-c35a7fe69be1\" (UID: \"c7161bfc-1b3c-4245-a0e7-c35a7fe69be1\") "
Nov 01 03:51:07.538310 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:07.538294    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-istio-csr-dns-cert\") pod \"c7161bfc-1b3c-4245-a0e7-c35a7fe69be1\" (UID: \"c7161bfc-1b3c-4245-a0e7-c35a7fe69be1\") "
Nov 01 03:51:07.538375 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:07.538326    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-cacerts\" (UniqueName: \"kubernetes.io/secret/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-cacerts\") pod \"c7161bfc-1b3c-4245-a0e7-c35a7fe69be1\" (UID: \"c7161bfc-1b3c-4245-a0e7-c35a7fe69be1\") "
Nov 01 03:51:07.538375 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:07.538357    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-istio-kubeconfig\") pod \"c7161bfc-1b3c-4245-a0e7-c35a7fe69be1\" (UID: \"c7161bfc-1b3c-4245-a0e7-c35a7fe69be1\") "
Nov 01 03:51:07.538483 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:07.538397    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-istio-csr-ca-configmap" pod "c7161bfc-1b3c-4245-a0e7-c35a7fe69be1" (UID: "c7161bfc-1b3c-4245-a0e7-c35a7fe69be1"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:51:07.539052 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:07.538922    4930 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-istio-csr-ca-configmap\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:07.542988 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:07.542948    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-local-certs" pod "c7161bfc-1b3c-4245-a0e7-c35a7fe69be1" (UID: "c7161bfc-1b3c-4245-a0e7-c35a7fe69be1"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:51:07.542988 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:07.542962    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-istio-csr-dns-cert" pod "c7161bfc-1b3c-4245-a0e7-c35a7fe69be1" (UID: "c7161bfc-1b3c-4245-a0e7-c35a7fe69be1"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:51:07.543148 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:07.542993    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-istio-kubeconfig" pod "c7161bfc-1b3c-4245-a0e7-c35a7fe69be1" (UID: "c7161bfc-1b3c-4245-a0e7-c35a7fe69be1"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:51:07.543148 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:07.543016    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-istio-token" pod "c7161bfc-1b3c-4245-a0e7-c35a7fe69be1" (UID: "c7161bfc-1b3c-4245-a0e7-c35a7fe69be1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:51:07.543148 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:07.543021    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-cacerts" pod "c7161bfc-1b3c-4245-a0e7-c35a7fe69be1" (UID: "c7161bfc-1b3c-4245-a0e7-c35a7fe69be1"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:51:07.543148 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:07.543048    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-kube-api-access-m7j7l" pod "c7161bfc-1b3c-4245-a0e7-c35a7fe69be1" (UID: "c7161bfc-1b3c-4245-a0e7-c35a7fe69be1"). InnerVolumeSpecName "kube-api-access-m7j7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:51:07.623754 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:07.623674    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c331aa42-00f4-4819-8efe-679636066bd7" path="/var/lib/kubelet/pods/c331aa42-00f4-4819-8efe-679636066bd7/volumes"
Nov 01 03:51:07.640274 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:07.640239    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7j7l\" (UniqueName: \"kubernetes.io/projected/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-kube-api-access-m7j7l\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:07.640274 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:07.640275    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:07.640456 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:07.640289    4930 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-local-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:07.640456 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:07.640304    4930 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-istio-csr-dns-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:07.640456 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:07.640318    4930 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-cacerts\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:07.640456 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:07.640342    4930 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1-istio-kubeconfig\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:07.662918 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:07.662890    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:51:07.982169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:07.982084    4930 generic.go:356] "Generic (PLEG): container finished" podID="c7161bfc-1b3c-4245-a0e7-c35a7fe69be1" containerID="6c988adf0b9527638cb030cd350cd466a7e507ebf99dbd3c3ba94f0d76905966" exitCode=0
Nov 01 03:51:07.982169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:07.982140    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvgwp"
Nov 01 03:51:07.982169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:07.982161    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvgwp" event={"ID":"c7161bfc-1b3c-4245-a0e7-c35a7fe69be1","Type":"ContainerDied","Data":"6c988adf0b9527638cb030cd350cd466a7e507ebf99dbd3c3ba94f0d76905966"}
Nov 01 03:51:07.982753 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:07.982211    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvgwp" event={"ID":"c7161bfc-1b3c-4245-a0e7-c35a7fe69be1","Type":"ContainerDied","Data":"f25bed095bd4c46595b895841f8fa4e871aeb930982e01f224cd848975af3efd"}
Nov 01 03:51:07.982753 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:07.982239    4930 scope.go:122] "RemoveContainer" containerID="6c988adf0b9527638cb030cd350cd466a7e507ebf99dbd3c3ba94f0d76905966"
Nov 01 03:51:07.993096 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:07.993075    4930 scope.go:122] "RemoveContainer" containerID="6c988adf0b9527638cb030cd350cd466a7e507ebf99dbd3c3ba94f0d76905966"
Nov 01 03:51:07.993388 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:51:07.993368    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c988adf0b9527638cb030cd350cd466a7e507ebf99dbd3c3ba94f0d76905966\": container with ID starting with 6c988adf0b9527638cb030cd350cd466a7e507ebf99dbd3c3ba94f0d76905966 not found: ID does not exist" containerID="6c988adf0b9527638cb030cd350cd466a7e507ebf99dbd3c3ba94f0d76905966"
Nov 01 03:51:07.993452 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:07.993398    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c988adf0b9527638cb030cd350cd466a7e507ebf99dbd3c3ba94f0d76905966"} err="failed to get container status \"6c988adf0b9527638cb030cd350cd466a7e507ebf99dbd3c3ba94f0d76905966\": rpc error: code = NotFound desc = could not find container \"6c988adf0b9527638cb030cd350cd466a7e507ebf99dbd3c3ba94f0d76905966\": container with ID starting with 6c988adf0b9527638cb030cd350cd466a7e507ebf99dbd3c3ba94f0d76905966 not found: ID does not exist"
Nov 01 03:51:08.149194 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:08.149160    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvgwp"]
Nov 01 03:51:08.199607 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:08.199577    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bvgwp"]
Nov 01 03:51:09.620148 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:09.620111    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7161bfc-1b3c-4245-a0e7-c35a7fe69be1" path="/var/lib/kubelet/pods/c7161bfc-1b3c-4245-a0e7-c35a7fe69be1/volumes"
Nov 01 03:51:10.510608 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:10.510549    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a95789f-fv6vv"]
Nov 01 03:51:10.511073 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:10.511060    4930 cpu_manager.go:414] "containerMap: removing container" podUID="c7161bfc-1b3c-4245-a0e7-c35a7fe69be1" containerName="discovery"
Nov 01 03:51:10.511129 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:10.511074    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7161bfc-1b3c-4245-a0e7-c35a7fe69be1" containerName="discovery"
Nov 01 03:51:10.511129 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:10.511097    4930 cpu_manager.go:414] "containerMap: removing container" podUID="86df4787-7837-437c-a76d-2bf797b9242f" containerName="payload-processing"
Nov 01 03:51:10.511129 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:10.511102    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86df4787-7837-437c-a76d-2bf797b9242f" containerName="payload-processing"
Nov 01 03:51:10.511129 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:10.511116    4930 cpu_manager.go:414] "containerMap: removing container" podUID="bd58da84-e12c-4bf4-a64c-d3b7155f57fa" containerName="payload-pre-processing"
Nov 01 03:51:10.511129 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:10.511121    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd58da84-e12c-4bf4-a64c-d3b7155f57fa" containerName="payload-pre-processing"
Nov 01 03:51:10.511302 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:10.511131    4930 cpu_manager.go:414] "containerMap: removing container" podUID="11338423-6bc3-42d6-8384-b438393fcda7" containerName="revoke-keys"
Nov 01 03:51:10.511302 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:10.511137    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11338423-6bc3-42d6-8384-b438393fcda7" containerName="revoke-keys"
Nov 01 03:51:10.511302 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:10.511153    4930 cpu_manager.go:414] "containerMap: removing container" podUID="532b2472-d6b3-4af0-905b-98953c8b814c" containerName="payload-processing"
Nov 01 03:51:10.511302 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:10.511159    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="532b2472-d6b3-4af0-905b-98953c8b814c" containerName="payload-processing"
Nov 01 03:51:10.511302 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:10.511248    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7161bfc-1b3c-4245-a0e7-c35a7fe69be1" containerName="discovery"
Nov 01 03:51:10.511302 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:10.511260    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="86df4787-7837-437c-a76d-2bf797b9242f" containerName="payload-processing"
Nov 01 03:51:10.511302 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:10.511268    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd58da84-e12c-4bf4-a64c-d3b7155f57fa" containerName="payload-pre-processing"
Nov 01 03:51:10.511302 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:10.511274    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="11338423-6bc3-42d6-8384-b438393fcda7" containerName="revoke-keys"
Nov 01 03:51:10.511302 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:10.511281    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="532b2472-d6b3-4af0-905b-98953c8b814c" containerName="payload-processing"
Nov 01 03:51:10.515847 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:10.515825    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a95789f-fv6vv"
Nov 01 03:51:10.571547 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:10.571510    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksdft\" (UniqueName: \"kubernetes.io/projected/f3a91e22-8c81-432a-957c-e4c301c5cb54-kube-api-access-ksdft\") pod \"maas-api-revoke-sub-8a95789f-fv6vv\" (UID: \"f3a91e22-8c81-432a-957c-e4c301c5cb54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a95789f-fv6vv"
Nov 01 03:51:10.571806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:10.571639    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3a91e22-8c81-432a-957c-e4c301c5cb54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a95789f-fv6vv\" (UID: \"f3a91e22-8c81-432a-957c-e4c301c5cb54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a95789f-fv6vv"
Nov 01 03:51:10.571806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:10.571725    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3a91e22-8c81-432a-957c-e4c301c5cb54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a95789f-fv6vv\" (UID: \"f3a91e22-8c81-432a-957c-e4c301c5cb54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a95789f-fv6vv"
Nov 01 03:51:10.573599 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:10.573575    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a95789f-fv6vv"]
Nov 01 03:51:10.618445 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:10.618422    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:51:10.618574 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:10.618445    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:51:10.618574 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:10.618458    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:51:10.618707 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:10.618693    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:51:10.672690 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:10.672662    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksdft\" (UniqueName: \"kubernetes.io/projected/f3a91e22-8c81-432a-957c-e4c301c5cb54-kube-api-access-ksdft\") pod \"maas-api-revoke-sub-8a95789f-fv6vv\" (UID: \"f3a91e22-8c81-432a-957c-e4c301c5cb54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a95789f-fv6vv"
Nov 01 03:51:10.673034 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:10.672757    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3a91e22-8c81-432a-957c-e4c301c5cb54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a95789f-fv6vv\" (UID: \"f3a91e22-8c81-432a-957c-e4c301c5cb54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a95789f-fv6vv"
Nov 01 03:51:10.673034 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:10.672859    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3a91e22-8c81-432a-957c-e4c301c5cb54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a95789f-fv6vv\" (UID: \"f3a91e22-8c81-432a-957c-e4c301c5cb54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a95789f-fv6vv"
Nov 01 03:51:10.673493 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:10.673473    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3a91e22-8c81-432a-957c-e4c301c5cb54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a95789f-fv6vv\" (UID: \"f3a91e22-8c81-432a-957c-e4c301c5cb54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a95789f-fv6vv"
Nov 01 03:51:10.675202 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:10.675176    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3a91e22-8c81-432a-957c-e4c301c5cb54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a95789f-fv6vv\" (UID: \"f3a91e22-8c81-432a-957c-e4c301c5cb54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a95789f-fv6vv"
Nov 01 03:51:10.736143 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:10.736112    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksdft\" (UniqueName: \"kubernetes.io/projected/f3a91e22-8c81-432a-957c-e4c301c5cb54-kube-api-access-ksdft\") pod \"maas-api-revoke-sub-8a95789f-fv6vv\" (UID: \"f3a91e22-8c81-432a-957c-e4c301c5cb54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a95789f-fv6vv"
Nov 01 03:51:10.830308 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:10.830249    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a95789f-fv6vv"
Nov 01 03:51:11.224550 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:11.224525    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a95789f-fv6vv"]
Nov 01 03:51:11.225407 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:51:11.225382    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3a91e22_8c81_432a_957c_e4c301c5cb54.slice/crio-9da2e0334768a49708bd2d054a4a2b4d3365d804a72775d11555010f214b5ad1 WatchSource:0}: Error finding container 9da2e0334768a49708bd2d054a4a2b4d3365d804a72775d11555010f214b5ad1: Status 404 returned error can't find the container with id 9da2e0334768a49708bd2d054a4a2b4d3365d804a72775d11555010f214b5ad1
Nov 01 03:51:12.014892 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:12.014853    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a95789f-fv6vv" event={"ID":"f3a91e22-8c81-432a-957c-e4c301c5cb54","Type":"ContainerStarted","Data":"b853dab771b161fb9f01071dee2eb76e1e015caea386367813bb36c8ba39a5dd"}
Nov 01 03:51:12.015381 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:12.014912    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a95789f-fv6vv" event={"ID":"f3a91e22-8c81-432a-957c-e4c301c5cb54","Type":"ContainerStarted","Data":"9da2e0334768a49708bd2d054a4a2b4d3365d804a72775d11555010f214b5ad1"}
Nov 01 03:51:12.132989 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:12.132929    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a95789f-fv6vv" podStartSLOduration=2.132911099 podStartE2EDuration="2.132911099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:51:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:51:12.131541669 +0000 UTC m=+2676.654548256" watchObservedRunningTime="2026-11-01 03:51:12.132911099 +0000 UTC m=+2676.655917690"
Nov 01 03:51:12.250568 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:12.250528    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:51:13.024159 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:13.024127    4930 generic.go:356] "Generic (PLEG): container finished" podID="f3a91e22-8c81-432a-957c-e4c301c5cb54" containerID="b853dab771b161fb9f01071dee2eb76e1e015caea386367813bb36c8ba39a5dd" exitCode=0
Nov 01 03:51:13.024641 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:13.024204    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a95789f-fv6vv" event={"ID":"f3a91e22-8c81-432a-957c-e4c301c5cb54","Type":"ContainerDied","Data":"b853dab771b161fb9f01071dee2eb76e1e015caea386367813bb36c8ba39a5dd"}
Nov 01 03:51:13.813673 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:13.813632    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:51:14.007119 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:14.007084    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:51:14.156378 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:14.156319    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:51:14.224624 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:14.224595    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a95789f-fv6vv"
Nov 01 03:51:14.316949 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:14.316702    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3a91e22-8c81-432a-957c-e4c301c5cb54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3a91e22-8c81-432a-957c-e4c301c5cb54-maas-api-openshift-ca\") pod \"f3a91e22-8c81-432a-957c-e4c301c5cb54\" (UID: \"f3a91e22-8c81-432a-957c-e4c301c5cb54\") "
Nov 01 03:51:14.316949 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:14.316751    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3a91e22-8c81-432a-957c-e4c301c5cb54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3a91e22-8c81-432a-957c-e4c301c5cb54-maas-api-xks-ca\") pod \"f3a91e22-8c81-432a-957c-e4c301c5cb54\" (UID: \"f3a91e22-8c81-432a-957c-e4c301c5cb54\") "
Nov 01 03:51:14.316949 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:14.316817    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3a91e22-8c81-432a-957c-e4c301c5cb54-kube-api-access-ksdft\" (UniqueName: \"kubernetes.io/projected/f3a91e22-8c81-432a-957c-e4c301c5cb54-kube-api-access-ksdft\") pod \"f3a91e22-8c81-432a-957c-e4c301c5cb54\" (UID: \"f3a91e22-8c81-432a-957c-e4c301c5cb54\") "
Nov 01 03:51:14.317261 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:14.317192    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3a91e22-8c81-432a-957c-e4c301c5cb54-maas-api-openshift-ca" pod "f3a91e22-8c81-432a-957c-e4c301c5cb54" (UID: "f3a91e22-8c81-432a-957c-e4c301c5cb54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:51:14.317672 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:14.317644    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3a91e22-8c81-432a-957c-e4c301c5cb54-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:14.320994 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:14.320968    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3a91e22-8c81-432a-957c-e4c301c5cb54-kube-api-access-ksdft" pod "f3a91e22-8c81-432a-957c-e4c301c5cb54" (UID: "f3a91e22-8c81-432a-957c-e4c301c5cb54"). InnerVolumeSpecName "kube-api-access-ksdft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:51:14.321096 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:14.321057    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3a91e22-8c81-432a-957c-e4c301c5cb54-maas-api-xks-ca" pod "f3a91e22-8c81-432a-957c-e4c301c5cb54" (UID: "f3a91e22-8c81-432a-957c-e4c301c5cb54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:51:14.418944 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:14.418866    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3a91e22-8c81-432a-957c-e4c301c5cb54-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:14.418944 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:14.418895    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksdft\" (UniqueName: \"kubernetes.io/projected/f3a91e22-8c81-432a-957c-e4c301c5cb54-kube-api-access-ksdft\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:14.586418 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:14.586381    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:51:15.039376 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:15.039340    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a95789f-fv6vv" event={"ID":"f3a91e22-8c81-432a-957c-e4c301c5cb54","Type":"ContainerDied","Data":"9da2e0334768a49708bd2d054a4a2b4d3365d804a72775d11555010f214b5ad1"}
Nov 01 03:51:15.039376 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:15.039366    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a95789f-fv6vv"
Nov 01 03:51:15.039637 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:15.039390    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9da2e0334768a49708bd2d054a4a2b4d3365d804a72775d11555010f214b5ad1"
Nov 01 03:51:15.096021 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:15.095985    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b12199-fnb5f"]
Nov 01 03:51:15.096538 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:15.096519    4930 cpu_manager.go:414] "containerMap: removing container" podUID="f3a91e22-8c81-432a-957c-e4c301c5cb54" containerName="revoke-keys"
Nov 01 03:51:15.096538 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:15.096536    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3a91e22-8c81-432a-957c-e4c301c5cb54" containerName="revoke-keys"
Nov 01 03:51:15.096786 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:15.096666    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3a91e22-8c81-432a-957c-e4c301c5cb54" containerName="revoke-keys"
Nov 01 03:51:15.100080 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:15.100058    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b12199-fnb5f"
Nov 01 03:51:15.157336 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:15.157304    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:51:15.157781 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:15.157383    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:51:15.157781 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:15.157391    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:51:15.157781 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:15.157462    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:51:15.161709 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:15.161686    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b12199-fnb5f"]
Nov 01 03:51:15.227249 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:15.227213    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e344a2a-f8a8-4531-809e-db674de9f556-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b12199-fnb5f\" (UID: \"9e344a2a-f8a8-4531-809e-db674de9f556\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b12199-fnb5f"
Nov 01 03:51:15.227386 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:15.227332    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v8lj\" (UniqueName: \"kubernetes.io/projected/9e344a2a-f8a8-4531-809e-db674de9f556-kube-api-access-5v8lj\") pod \"maas-api-revoke-keys-e2e-worker-w1-b12199-fnb5f\" (UID: \"9e344a2a-f8a8-4531-809e-db674de9f556\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b12199-fnb5f"
Nov 01 03:51:15.227443 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:15.227383    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e344a2a-f8a8-4531-809e-db674de9f556-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b12199-fnb5f\" (UID: \"9e344a2a-f8a8-4531-809e-db674de9f556\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b12199-fnb5f"
Nov 01 03:51:15.328344 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:15.328268    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v8lj\" (UniqueName: \"kubernetes.io/projected/9e344a2a-f8a8-4531-809e-db674de9f556-kube-api-access-5v8lj\") pod \"maas-api-revoke-keys-e2e-worker-w1-b12199-fnb5f\" (UID: \"9e344a2a-f8a8-4531-809e-db674de9f556\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b12199-fnb5f"
Nov 01 03:51:15.328344 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:15.328322    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e344a2a-f8a8-4531-809e-db674de9f556-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b12199-fnb5f\" (UID: \"9e344a2a-f8a8-4531-809e-db674de9f556\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b12199-fnb5f"
Nov 01 03:51:15.328526 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:15.328432    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e344a2a-f8a8-4531-809e-db674de9f556-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b12199-fnb5f\" (UID: \"9e344a2a-f8a8-4531-809e-db674de9f556\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b12199-fnb5f"
Nov 01 03:51:15.329158 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:15.329122    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e344a2a-f8a8-4531-809e-db674de9f556-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b12199-fnb5f\" (UID: \"9e344a2a-f8a8-4531-809e-db674de9f556\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b12199-fnb5f"
Nov 01 03:51:15.330799 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:15.330766    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e344a2a-f8a8-4531-809e-db674de9f556-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b12199-fnb5f\" (UID: \"9e344a2a-f8a8-4531-809e-db674de9f556\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b12199-fnb5f"
Nov 01 03:51:15.388748 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:15.388722    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v8lj\" (UniqueName: \"kubernetes.io/projected/9e344a2a-f8a8-4531-809e-db674de9f556-kube-api-access-5v8lj\") pod \"maas-api-revoke-keys-e2e-worker-w1-b12199-fnb5f\" (UID: \"9e344a2a-f8a8-4531-809e-db674de9f556\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b12199-fnb5f"
Nov 01 03:51:15.414391 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:15.414374    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b12199-fnb5f"
Nov 01 03:51:15.790215 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:15.790187    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b12199-fnb5f"]
Nov 01 03:51:15.790870 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:51:15.790842    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e344a2a_f8a8_4531_809e_db674de9f556.slice/crio-89496c7143df372d2dd71d84ff398ae3d6812cedbcfdf326f429e1c2d1c66bf5 WatchSource:0}: Error finding container 89496c7143df372d2dd71d84ff398ae3d6812cedbcfdf326f429e1c2d1c66bf5: Status 404 returned error can't find the container with id 89496c7143df372d2dd71d84ff398ae3d6812cedbcfdf326f429e1c2d1c66bf5
Nov 01 03:51:16.048014 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:16.047943    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b12199-fnb5f" event={"ID":"9e344a2a-f8a8-4531-809e-db674de9f556","Type":"ContainerStarted","Data":"89496c7143df372d2dd71d84ff398ae3d6812cedbcfdf326f429e1c2d1c66bf5"}
Nov 01 03:51:17.058498 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:17.058406    4930 generic.go:356] "Generic (PLEG): container finished" podID="9e344a2a-f8a8-4531-809e-db674de9f556" containerID="5ec224fbaa70d12831d661b0fb63398236c54040b96bb694d3c93025394a0833" exitCode=0
Nov 01 03:51:17.058901 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:17.058489    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b12199-fnb5f" event={"ID":"9e344a2a-f8a8-4531-809e-db674de9f556","Type":"ContainerDied","Data":"5ec224fbaa70d12831d661b0fb63398236c54040b96bb694d3c93025394a0833"}
Nov 01 03:51:17.463128 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:17.463086    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:51:18.227219 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:18.227192    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b12199-fnb5f"
Nov 01 03:51:18.365687 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:18.365655    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e344a2a-f8a8-4531-809e-db674de9f556-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e344a2a-f8a8-4531-809e-db674de9f556-maas-api-openshift-ca\") pod \"9e344a2a-f8a8-4531-809e-db674de9f556\" (UID: \"9e344a2a-f8a8-4531-809e-db674de9f556\") "
Nov 01 03:51:18.365875 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:18.365762    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e344a2a-f8a8-4531-809e-db674de9f556-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e344a2a-f8a8-4531-809e-db674de9f556-maas-api-xks-ca\") pod \"9e344a2a-f8a8-4531-809e-db674de9f556\" (UID: \"9e344a2a-f8a8-4531-809e-db674de9f556\") "
Nov 01 03:51:18.365875 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:18.365863    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e344a2a-f8a8-4531-809e-db674de9f556-kube-api-access-5v8lj\" (UniqueName: \"kubernetes.io/projected/9e344a2a-f8a8-4531-809e-db674de9f556-kube-api-access-5v8lj\") pod \"9e344a2a-f8a8-4531-809e-db674de9f556\" (UID: \"9e344a2a-f8a8-4531-809e-db674de9f556\") "
Nov 01 03:51:18.366126 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:18.366103    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e344a2a-f8a8-4531-809e-db674de9f556-maas-api-openshift-ca" pod "9e344a2a-f8a8-4531-809e-db674de9f556" (UID: "9e344a2a-f8a8-4531-809e-db674de9f556"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:51:18.366676 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:18.366647    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e344a2a-f8a8-4531-809e-db674de9f556-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:18.368043 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:18.368011    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e344a2a-f8a8-4531-809e-db674de9f556-maas-api-xks-ca" pod "9e344a2a-f8a8-4531-809e-db674de9f556" (UID: "9e344a2a-f8a8-4531-809e-db674de9f556"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:51:18.368345 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:18.368320    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e344a2a-f8a8-4531-809e-db674de9f556-kube-api-access-5v8lj" pod "9e344a2a-f8a8-4531-809e-db674de9f556" (UID: "9e344a2a-f8a8-4531-809e-db674de9f556"). InnerVolumeSpecName "kube-api-access-5v8lj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:51:18.467954 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:18.467927    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e344a2a-f8a8-4531-809e-db674de9f556-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:18.467954 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:18.467953    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v8lj\" (UniqueName: \"kubernetes.io/projected/9e344a2a-f8a8-4531-809e-db674de9f556-kube-api-access-5v8lj\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:19.075049 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:19.075021    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b12199-fnb5f"
Nov 01 03:51:19.075254 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:19.075015    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b12199-fnb5f" event={"ID":"9e344a2a-f8a8-4531-809e-db674de9f556","Type":"ContainerDied","Data":"89496c7143df372d2dd71d84ff398ae3d6812cedbcfdf326f429e1c2d1c66bf5"}
Nov 01 03:51:19.075254 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:19.075150    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89496c7143df372d2dd71d84ff398ae3d6812cedbcfdf326f429e1c2d1c66bf5"
Nov 01 03:51:19.121487 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:51:19.121454    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:51:19.590290 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:19.590256    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"]
Nov 01 03:51:19.665454 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:19.665422    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:51:22.299762 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:22.299724    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm"]
Nov 01 03:51:22.300287 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:22.299967    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm" podUID="accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd" containerName="istio-proxy" containerID="cri-o://1c365ed9d508a7b0d892d0491d5646a1d5ebdb820ae17d2c13959dd219724c50" gracePeriod=30
Nov 01 03:51:22.574438 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:22.574350    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:51:23.209926 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:23.209885    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:51:24.112273 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:24.112241    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26176dd3-jh22v"]
Nov 01 03:51:24.112861 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:24.112842    4930 cpu_manager.go:414] "containerMap: removing container" podUID="9e344a2a-f8a8-4531-809e-db674de9f556" containerName="revoke-keys"
Nov 01 03:51:24.112914 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:24.112862    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e344a2a-f8a8-4531-809e-db674de9f556" containerName="revoke-keys"
Nov 01 03:51:24.112962 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:24.112952    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e344a2a-f8a8-4531-809e-db674de9f556" containerName="revoke-keys"
Nov 01 03:51:24.116472 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:24.116451    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26176dd3-jh22v"
Nov 01 03:51:24.170208 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:24.170169    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:51:24.219849 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:24.219815    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:51:24.219849 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:24.219817    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:51:24.220056 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:24.219976    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:51:24.223684 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:24.223657    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26176dd3-jh22v"]
Nov 01 03:51:24.238547 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:24.238516    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbe4aee8-385b-4ba8-8a98-5de9aa154024-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26176dd3-jh22v\" (UID: \"dbe4aee8-385b-4ba8-8a98-5de9aa154024\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26176dd3-jh22v"
Nov 01 03:51:24.238700 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:24.238611    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9h5f\" (UniqueName: \"kubernetes.io/projected/dbe4aee8-385b-4ba8-8a98-5de9aa154024-kube-api-access-c9h5f\") pod \"maas-api-revoke-sub-26176dd3-jh22v\" (UID: \"dbe4aee8-385b-4ba8-8a98-5de9aa154024\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26176dd3-jh22v"
Nov 01 03:51:24.238700 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:24.238677    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbe4aee8-385b-4ba8-8a98-5de9aa154024-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26176dd3-jh22v\" (UID: \"dbe4aee8-385b-4ba8-8a98-5de9aa154024\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26176dd3-jh22v"
Nov 01 03:51:24.340288 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:24.340251    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbe4aee8-385b-4ba8-8a98-5de9aa154024-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26176dd3-jh22v\" (UID: \"dbe4aee8-385b-4ba8-8a98-5de9aa154024\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26176dd3-jh22v"
Nov 01 03:51:24.340498 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:24.340315    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9h5f\" (UniqueName: \"kubernetes.io/projected/dbe4aee8-385b-4ba8-8a98-5de9aa154024-kube-api-access-c9h5f\") pod \"maas-api-revoke-sub-26176dd3-jh22v\" (UID: \"dbe4aee8-385b-4ba8-8a98-5de9aa154024\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26176dd3-jh22v"
Nov 01 03:51:24.340498 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:24.340397    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbe4aee8-385b-4ba8-8a98-5de9aa154024-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26176dd3-jh22v\" (UID: \"dbe4aee8-385b-4ba8-8a98-5de9aa154024\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26176dd3-jh22v"
Nov 01 03:51:24.341278 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:24.341241    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbe4aee8-385b-4ba8-8a98-5de9aa154024-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26176dd3-jh22v\" (UID: \"dbe4aee8-385b-4ba8-8a98-5de9aa154024\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26176dd3-jh22v"
Nov 01 03:51:24.342781 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:24.342757    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbe4aee8-385b-4ba8-8a98-5de9aa154024-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26176dd3-jh22v\" (UID: \"dbe4aee8-385b-4ba8-8a98-5de9aa154024\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26176dd3-jh22v"
Nov 01 03:51:24.401686 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:24.401604    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9h5f\" (UniqueName: \"kubernetes.io/projected/dbe4aee8-385b-4ba8-8a98-5de9aa154024-kube-api-access-c9h5f\") pod \"maas-api-revoke-sub-26176dd3-jh22v\" (UID: \"dbe4aee8-385b-4ba8-8a98-5de9aa154024\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26176dd3-jh22v"
Nov 01 03:51:24.433869 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:24.433836    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26176dd3-jh22v"
Nov 01 03:51:24.825468 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:24.825440    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26176dd3-jh22v"]
Nov 01 03:51:24.826344 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:51:24.826321    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbe4aee8_385b_4ba8_8a98_5de9aa154024.slice/crio-019fa39651fca46e20f619db4018b55449855c0652a8ec734c785219528c373a WatchSource:0}: Error finding container 019fa39651fca46e20f619db4018b55449855c0652a8ec734c785219528c373a: Status 404 returned error can't find the container with id 019fa39651fca46e20f619db4018b55449855c0652a8ec734c785219528c373a
Nov 01 03:51:25.126422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:25.126380    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26176dd3-jh22v" event={"ID":"dbe4aee8-385b-4ba8-8a98-5de9aa154024","Type":"ContainerStarted","Data":"019fa39651fca46e20f619db4018b55449855c0652a8ec734c785219528c373a"}
Nov 01 03:51:26.135047 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:26.135010    4930 generic.go:356] "Generic (PLEG): container finished" podID="dbe4aee8-385b-4ba8-8a98-5de9aa154024" containerID="d1430e797cd12b6c5e4e695c05844e180b04a769997102e20b5753200c498da7" exitCode=0
Nov 01 03:51:26.135453 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:26.135096    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26176dd3-jh22v" event={"ID":"dbe4aee8-385b-4ba8-8a98-5de9aa154024","Type":"ContainerDied","Data":"d1430e797cd12b6c5e4e695c05844e180b04a769997102e20b5753200c498da7"}
Nov 01 03:51:27.285045 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:27.285018    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26176dd3-jh22v"
Nov 01 03:51:27.381755 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:27.381717    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbe4aee8-385b-4ba8-8a98-5de9aa154024-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbe4aee8-385b-4ba8-8a98-5de9aa154024-maas-api-openshift-ca\") pod \"dbe4aee8-385b-4ba8-8a98-5de9aa154024\" (UID: \"dbe4aee8-385b-4ba8-8a98-5de9aa154024\") "
Nov 01 03:51:27.381917 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:27.381816    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbe4aee8-385b-4ba8-8a98-5de9aa154024-kube-api-access-c9h5f\" (UniqueName: \"kubernetes.io/projected/dbe4aee8-385b-4ba8-8a98-5de9aa154024-kube-api-access-c9h5f\") pod \"dbe4aee8-385b-4ba8-8a98-5de9aa154024\" (UID: \"dbe4aee8-385b-4ba8-8a98-5de9aa154024\") "
Nov 01 03:51:27.381917 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:27.381886    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbe4aee8-385b-4ba8-8a98-5de9aa154024-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbe4aee8-385b-4ba8-8a98-5de9aa154024-maas-api-xks-ca\") pod \"dbe4aee8-385b-4ba8-8a98-5de9aa154024\" (UID: \"dbe4aee8-385b-4ba8-8a98-5de9aa154024\") "
Nov 01 03:51:27.382083 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:27.382059    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbe4aee8-385b-4ba8-8a98-5de9aa154024-maas-api-openshift-ca" pod "dbe4aee8-385b-4ba8-8a98-5de9aa154024" (UID: "dbe4aee8-385b-4ba8-8a98-5de9aa154024"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:51:27.382413 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:27.382390    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbe4aee8-385b-4ba8-8a98-5de9aa154024-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:27.384023 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:27.383993    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbe4aee8-385b-4ba8-8a98-5de9aa154024-kube-api-access-c9h5f" pod "dbe4aee8-385b-4ba8-8a98-5de9aa154024" (UID: "dbe4aee8-385b-4ba8-8a98-5de9aa154024"). InnerVolumeSpecName "kube-api-access-c9h5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:51:27.384147 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:27.384049    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbe4aee8-385b-4ba8-8a98-5de9aa154024-maas-api-xks-ca" pod "dbe4aee8-385b-4ba8-8a98-5de9aa154024" (UID: "dbe4aee8-385b-4ba8-8a98-5de9aa154024"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:51:27.441971 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:27.441878    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:51:27.483727 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:27.483656    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9h5f\" (UniqueName: \"kubernetes.io/projected/dbe4aee8-385b-4ba8-8a98-5de9aa154024-kube-api-access-c9h5f\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:27.483727 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:27.483686    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbe4aee8-385b-4ba8-8a98-5de9aa154024-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:27.589632 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:27.589595    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:51:27.622398 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:27.622370    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm"
Nov 01 03:51:27.730387 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:27.730298    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:51:27.787686 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:27.787648    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-istio-token\" (UniqueName: \"kubernetes.io/projected/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-istio-token\") pod \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\" (UID: \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\") "
Nov 01 03:51:27.787873 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:27.787726    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-kube-api-access-fglzl\" (UniqueName: \"kubernetes.io/projected/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-kube-api-access-fglzl\") pod \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\" (UID: \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\") "
Nov 01 03:51:27.787873 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:27.787804    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-workload-certs\") pod \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\" (UID: \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\") "
Nov 01 03:51:27.787873 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:27.787834    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-istio-podinfo\") pod \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\" (UID: \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\") "
Nov 01 03:51:27.788075 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:27.787978    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-credential-socket\") pod \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\" (UID: \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\") "
Nov 01 03:51:27.788075 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:27.788021    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-istio-data\") pod \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\" (UID: \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\") "
Nov 01 03:51:27.788197 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:27.788079    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-istio-envoy\") pod \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\" (UID: \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\") "
Nov 01 03:51:27.788197 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:27.788134    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-istiod-ca-cert\") pod \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\" (UID: \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\") "
Nov 01 03:51:27.788197 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:27.788166    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-workload-socket\") pod \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\" (UID: \"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd\") "
Nov 01 03:51:27.788539 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:27.788489    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-workload-certs" pod "accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd" (UID: "accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:51:27.788539 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:27.788510    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-istio-data" pod "accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd" (UID: "accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:51:27.788789 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:27.788765    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-workload-socket" pod "accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd" (UID: "accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:51:27.788789 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:27.788780    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-credential-socket" pod "accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd" (UID: "accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:51:27.788998 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:27.788976    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:27.789090 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:27.789003    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:27.789216 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:27.789176    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-istiod-ca-cert" pod "accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd" (UID: "accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:51:27.790251 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:27.790224    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-istio-token" pod "accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd" (UID: "accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:51:27.790549 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:27.790522    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-kube-api-access-fglzl" pod "accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd" (UID: "accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd"). InnerVolumeSpecName "kube-api-access-fglzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:51:27.791069 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:27.791047    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-istio-podinfo" pod "accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd" (UID: "accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:51:27.791520 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:27.791492    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-istio-envoy" pod "accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd" (UID: "accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:51:27.889896 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:27.889868    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:27.889896 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:27.889895    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:27.889896 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:27.889904    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:27.890123 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:27.889913    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:27.890123 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:27.889923    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:27.890123 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:27.889935    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fglzl\" (UniqueName: \"kubernetes.io/projected/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-kube-api-access-fglzl\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:27.890123 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:27.889949    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:28.152475 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:28.152442    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26176dd3-jh22v"
Nov 01 03:51:28.152677 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:28.152443    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26176dd3-jh22v" event={"ID":"dbe4aee8-385b-4ba8-8a98-5de9aa154024","Type":"ContainerDied","Data":"019fa39651fca46e20f619db4018b55449855c0652a8ec734c785219528c373a"}
Nov 01 03:51:28.152677 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:28.152621    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="019fa39651fca46e20f619db4018b55449855c0652a8ec734c785219528c373a"
Nov 01 03:51:28.154087 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:28.154059    4930 generic.go:356] "Generic (PLEG): container finished" podID="accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd" containerID="1c365ed9d508a7b0d892d0491d5646a1d5ebdb820ae17d2c13959dd219724c50" exitCode=0
Nov 01 03:51:28.154223 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:28.154122    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm"
Nov 01 03:51:28.154223 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:28.154145    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm" event={"ID":"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd","Type":"ContainerDied","Data":"1c365ed9d508a7b0d892d0491d5646a1d5ebdb820ae17d2c13959dd219724c50"}
Nov 01 03:51:28.154223 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:28.154213    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm" event={"ID":"accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd","Type":"ContainerDied","Data":"631b0e1e28cdce8cf89b08a8c16cf739edc1804969d251e89fa0117fcb4f6c6c"}
Nov 01 03:51:28.154400 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:28.154249    4930 scope.go:122] "RemoveContainer" containerID="1c365ed9d508a7b0d892d0491d5646a1d5ebdb820ae17d2c13959dd219724c50"
Nov 01 03:51:28.172709 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:28.172682    4930 scope.go:122] "RemoveContainer" containerID="1c365ed9d508a7b0d892d0491d5646a1d5ebdb820ae17d2c13959dd219724c50"
Nov 01 03:51:28.173023 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:51:28.173001    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c365ed9d508a7b0d892d0491d5646a1d5ebdb820ae17d2c13959dd219724c50\": container with ID starting with 1c365ed9d508a7b0d892d0491d5646a1d5ebdb820ae17d2c13959dd219724c50 not found: ID does not exist" containerID="1c365ed9d508a7b0d892d0491d5646a1d5ebdb820ae17d2c13959dd219724c50"
Nov 01 03:51:28.173110 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:28.173031    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c365ed9d508a7b0d892d0491d5646a1d5ebdb820ae17d2c13959dd219724c50"} err="failed to get container status \"1c365ed9d508a7b0d892d0491d5646a1d5ebdb820ae17d2c13959dd219724c50\": rpc error: code = NotFound desc = could not find container \"1c365ed9d508a7b0d892d0491d5646a1d5ebdb820ae17d2c13959dd219724c50\": container with ID starting with 1c365ed9d508a7b0d892d0491d5646a1d5ebdb820ae17d2c13959dd219724c50 not found: ID does not exist"
Nov 01 03:51:28.321365 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:28.321328    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm"]
Nov 01 03:51:28.370621 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:28.370587    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b12199-openshift-default-774cb8c567-pg6vm"]
Nov 01 03:51:29.620423 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:29.620385    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd" path="/var/lib/kubelet/pods/accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd/volumes"
Nov 01 03:51:32.184629 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:32.184596    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:51:34.616317 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:34.616268    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj" podUID="b0e32cb3-ac18-4c1f-819c-f1139a88bfea" containerName="main" containerID="cri-o://c0bccd55d992db25325dd41c86a53c6948929d08e109184a2ba8a47a651901ca" gracePeriod=45
Nov 01 03:51:34.883771 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:34.883718    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:51:34.987765 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:34.987733    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0e32cb3-ac18-4c1f-819c-f1139a88bfea-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0e32cb3-ac18-4c1f-819c-f1139a88bfea-model-cache\") pod \"b0e32cb3-ac18-4c1f-819c-f1139a88bfea\" (UID: \"b0e32cb3-ac18-4c1f-819c-f1139a88bfea\") "
Nov 01 03:51:34.988261 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:34.987810    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0e32cb3-ac18-4c1f-819c-f1139a88bfea-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0e32cb3-ac18-4c1f-819c-f1139a88bfea-tmp-dir\") pod \"b0e32cb3-ac18-4c1f-819c-f1139a88bfea\" (UID: \"b0e32cb3-ac18-4c1f-819c-f1139a88bfea\") "
Nov 01 03:51:34.988261 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:34.987869    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0e32cb3-ac18-4c1f-819c-f1139a88bfea-tls-certs\" (UniqueName: \"kubernetes.io/secret/b0e32cb3-ac18-4c1f-819c-f1139a88bfea-tls-certs\") pod \"b0e32cb3-ac18-4c1f-819c-f1139a88bfea\" (UID: \"b0e32cb3-ac18-4c1f-819c-f1139a88bfea\") "
Nov 01 03:51:34.988261 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:34.987940    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0e32cb3-ac18-4c1f-819c-f1139a88bfea-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0e32cb3-ac18-4c1f-819c-f1139a88bfea-dshm\") pod \"b0e32cb3-ac18-4c1f-819c-f1139a88bfea\" (UID: \"b0e32cb3-ac18-4c1f-819c-f1139a88bfea\") "
Nov 01 03:51:34.988261 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:34.987998    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0e32cb3-ac18-4c1f-819c-f1139a88bfea-model-cache" pod "b0e32cb3-ac18-4c1f-819c-f1139a88bfea" (UID: "b0e32cb3-ac18-4c1f-819c-f1139a88bfea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:51:34.988261 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:34.988097    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0e32cb3-ac18-4c1f-819c-f1139a88bfea-kube-api-access-xrp6z\" (UniqueName: \"kubernetes.io/projected/b0e32cb3-ac18-4c1f-819c-f1139a88bfea-kube-api-access-xrp6z\") pod \"b0e32cb3-ac18-4c1f-819c-f1139a88bfea\" (UID: \"b0e32cb3-ac18-4c1f-819c-f1139a88bfea\") "
Nov 01 03:51:34.988261 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:34.988126    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0e32cb3-ac18-4c1f-819c-f1139a88bfea-tmp-dir" pod "b0e32cb3-ac18-4c1f-819c-f1139a88bfea" (UID: "b0e32cb3-ac18-4c1f-819c-f1139a88bfea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:51:34.988261 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:34.988192    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0e32cb3-ac18-4c1f-819c-f1139a88bfea-home\" (UniqueName: \"kubernetes.io/empty-dir/b0e32cb3-ac18-4c1f-819c-f1139a88bfea-home\") pod \"b0e32cb3-ac18-4c1f-819c-f1139a88bfea\" (UID: \"b0e32cb3-ac18-4c1f-819c-f1139a88bfea\") "
Nov 01 03:51:34.988681 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:34.988502    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0e32cb3-ac18-4c1f-819c-f1139a88bfea-home" pod "b0e32cb3-ac18-4c1f-819c-f1139a88bfea" (UID: "b0e32cb3-ac18-4c1f-819c-f1139a88bfea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:51:34.988786 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:34.988757    4930 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0e32cb3-ac18-4c1f-819c-f1139a88bfea-home\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:34.988874 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:34.988785    4930 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0e32cb3-ac18-4c1f-819c-f1139a88bfea-model-cache\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:34.988874 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:34.988824    4930 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0e32cb3-ac18-4c1f-819c-f1139a88bfea-tmp-dir\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:34.991169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:34.990886    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0e32cb3-ac18-4c1f-819c-f1139a88bfea-tls-certs" pod "b0e32cb3-ac18-4c1f-819c-f1139a88bfea" (UID: "b0e32cb3-ac18-4c1f-819c-f1139a88bfea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:51:34.991169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:34.991003    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0e32cb3-ac18-4c1f-819c-f1139a88bfea-kube-api-access-xrp6z" pod "b0e32cb3-ac18-4c1f-819c-f1139a88bfea" (UID: "b0e32cb3-ac18-4c1f-819c-f1139a88bfea"). InnerVolumeSpecName "kube-api-access-xrp6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:51:34.991169 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:34.991085    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0e32cb3-ac18-4c1f-819c-f1139a88bfea-dshm" pod "b0e32cb3-ac18-4c1f-819c-f1139a88bfea" (UID: "b0e32cb3-ac18-4c1f-819c-f1139a88bfea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:51:35.089834 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:35.089804    4930 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0e32cb3-ac18-4c1f-819c-f1139a88bfea-dshm\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:35.089834 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:35.089833    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrp6z\" (UniqueName: \"kubernetes.io/projected/b0e32cb3-ac18-4c1f-819c-f1139a88bfea-kube-api-access-xrp6z\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:35.090035 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:35.089844    4930 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0e32cb3-ac18-4c1f-819c-f1139a88bfea-tls-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:35.216692 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:35.216606    4930 generic.go:356] "Generic (PLEG): container finished" podID="b0e32cb3-ac18-4c1f-819c-f1139a88bfea" containerID="c0bccd55d992db25325dd41c86a53c6948929d08e109184a2ba8a47a651901ca" exitCode=0
Nov 01 03:51:35.216692 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:35.216655    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"
Nov 01 03:51:35.216901 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:35.216696    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj" event={"ID":"b0e32cb3-ac18-4c1f-819c-f1139a88bfea","Type":"ContainerDied","Data":"c0bccd55d992db25325dd41c86a53c6948929d08e109184a2ba8a47a651901ca"}
Nov 01 03:51:35.216901 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:35.216750    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj" event={"ID":"b0e32cb3-ac18-4c1f-819c-f1139a88bfea","Type":"ContainerDied","Data":"1f307b357bbee507e31b256c0fbb1a61a07d4c123c5ae3b27bb8b7fd9ad26b11"}
Nov 01 03:51:35.216901 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:35.216774    4930 scope.go:122] "RemoveContainer" containerID="c0bccd55d992db25325dd41c86a53c6948929d08e109184a2ba8a47a651901ca"
Nov 01 03:51:35.230234 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:35.230210    4930 scope.go:122] "RemoveContainer" containerID="c0bccd55d992db25325dd41c86a53c6948929d08e109184a2ba8a47a651901ca"
Nov 01 03:51:35.230568 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:51:35.230534    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0bccd55d992db25325dd41c86a53c6948929d08e109184a2ba8a47a651901ca\": container with ID starting with c0bccd55d992db25325dd41c86a53c6948929d08e109184a2ba8a47a651901ca not found: ID does not exist" containerID="c0bccd55d992db25325dd41c86a53c6948929d08e109184a2ba8a47a651901ca"
Nov 01 03:51:35.230671 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:35.230585    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0bccd55d992db25325dd41c86a53c6948929d08e109184a2ba8a47a651901ca"} err="failed to get container status \"c0bccd55d992db25325dd41c86a53c6948929d08e109184a2ba8a47a651901ca\": rpc error: code = NotFound desc = could not find container \"c0bccd55d992db25325dd41c86a53c6948929d08e109184a2ba8a47a651901ca\": container with ID starting with c0bccd55d992db25325dd41c86a53c6948929d08e109184a2ba8a47a651901ca not found: ID does not exist"
Nov 01 03:51:35.382389 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:35.382360    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"]
Nov 01 03:51:35.388359 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:35.388333    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b12199/simulator-w1-b12199-kserve-7cfcb76798-g5hnj"]
Nov 01 03:51:35.621192 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:35.621161    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0e32cb3-ac18-4c1f-819c-f1139a88bfea" path="/var/lib/kubelet/pods/b0e32cb3-ac18-4c1f-819c-f1139a88bfea/volumes"
Nov 01 03:51:37.020506 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:37.020465    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:51:38.049592 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:38.049526    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa699d2c-hht2n"]
Nov 01 03:51:38.099272 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:38.099244    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3a679a8-45wbr"]
Nov 01 03:51:38.102361 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:38.102338    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa699d2c-hht2n"]
Nov 01 03:51:38.104965 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:38.104944    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3a679a8-45wbr"]
Nov 01 03:51:38.304792 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:38.304721    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jh955_f29ea61e-2a78-4c11-9566-f52497e135ff/console-operator/2.log"
Nov 01 03:51:38.346630 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:38.346597    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jh955_f29ea61e-2a78-4c11-9566-f52497e135ff/console-operator/2.log"
Nov 01 03:51:39.620035 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:39.619992    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fe8f9eb-df64-4911-a80d-d0416d9b48c4" path="/var/lib/kubelet/pods/2fe8f9eb-df64-4911-a80d-d0416d9b48c4/volumes"
Nov 01 03:51:39.620675 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:39.620652    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35100fb8-078b-40cf-a01a-f92b559d6a54" path="/var/lib/kubelet/pods/35100fb8-078b-40cf-a01a-f92b559d6a54/volumes"
Nov 01 03:51:41.725643 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:41.725607    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:51:42.109171 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:42.109120    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:51:43.165393 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:43.165355    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:51:43.306567 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:43.306511    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:51:46.525141 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:46.525091    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:51:49.123777 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:51:49.123734    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:51:50.657864 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:50.657805    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:51:51.606943 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:51.606906    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:51:52.192147 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:52.192111    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:51:52.423705 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:52.423662    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:51:52.573863 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:52.573771    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:51:52.708279 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:52.708249    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a279c2-hktzw"]
Nov 01 03:51:52.757220 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:52.757186    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a279c2-hktzw"]
Nov 01 03:51:53.133106 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:53.133075    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:51:53.621274 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:53.621241    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bf9eec1-aa20-4211-8a26-9fa2058f836d" path="/var/lib/kubelet/pods/8bf9eec1-aa20-4211-8a26-9fa2058f836d/volumes"
Nov 01 03:51:53.870107 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:53.870068    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:51:54.369452 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:54.369420    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22e11b-6j9ps"]
Nov 01 03:51:54.369929 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:54.369915    4930 cpu_manager.go:414] "containerMap: removing container" podUID="accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd" containerName="istio-proxy"
Nov 01 03:51:54.369982 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:54.369930    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd" containerName="istio-proxy"
Nov 01 03:51:54.369982 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:54.369969    4930 cpu_manager.go:414] "containerMap: removing container" podUID="b0e32cb3-ac18-4c1f-819c-f1139a88bfea" containerName="main"
Nov 01 03:51:54.369982 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:54.369974    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0e32cb3-ac18-4c1f-819c-f1139a88bfea" containerName="main"
Nov 01 03:51:54.370091 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:54.369987    4930 cpu_manager.go:414] "containerMap: removing container" podUID="dbe4aee8-385b-4ba8-8a98-5de9aa154024" containerName="revoke-keys"
Nov 01 03:51:54.370091 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:54.369992    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbe4aee8-385b-4ba8-8a98-5de9aa154024" containerName="revoke-keys"
Nov 01 03:51:54.370091 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:54.370067    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0e32cb3-ac18-4c1f-819c-f1139a88bfea" containerName="main"
Nov 01 03:51:54.370091 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:54.370078    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbe4aee8-385b-4ba8-8a98-5de9aa154024" containerName="revoke-keys"
Nov 01 03:51:54.370091 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:54.370085    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="accb5e6c-0cb8-46c0-a2bc-7776d28f6ccd" containerName="istio-proxy"
Nov 01 03:51:54.375050 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:54.375033    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22e11b-6j9ps"
Nov 01 03:51:54.429671 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:54.429646    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:51:54.429804 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:54.429725    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:51:54.429804 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:54.429742    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:51:54.479651 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:54.479627    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:51:54.483040 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:54.483018    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22e11b-6j9ps"]
Nov 01 03:51:54.523567 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:54.523524    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f4de4f9-c0dd-418c-8157-c2f25a2b15eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-22e11b-6j9ps\" (UID: \"2f4de4f9-c0dd-418c-8157-c2f25a2b15eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22e11b-6j9ps"
Nov 01 03:51:54.523716 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:54.523629    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4fkq\" (UniqueName: \"kubernetes.io/projected/2f4de4f9-c0dd-418c-8157-c2f25a2b15eb-kube-api-access-d4fkq\") pod \"maas-api-revoke-keys-e2e-inf-a-22e11b-6j9ps\" (UID: \"2f4de4f9-c0dd-418c-8157-c2f25a2b15eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22e11b-6j9ps"
Nov 01 03:51:54.523790 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:54.523772    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f4de4f9-c0dd-418c-8157-c2f25a2b15eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-22e11b-6j9ps\" (UID: \"2f4de4f9-c0dd-418c-8157-c2f25a2b15eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22e11b-6j9ps"
Nov 01 03:51:54.624751 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:54.624671    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f4de4f9-c0dd-418c-8157-c2f25a2b15eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-22e11b-6j9ps\" (UID: \"2f4de4f9-c0dd-418c-8157-c2f25a2b15eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22e11b-6j9ps"
Nov 01 03:51:54.625146 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:54.624779    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f4de4f9-c0dd-418c-8157-c2f25a2b15eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-22e11b-6j9ps\" (UID: \"2f4de4f9-c0dd-418c-8157-c2f25a2b15eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22e11b-6j9ps"
Nov 01 03:51:54.625146 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:54.624844    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4fkq\" (UniqueName: \"kubernetes.io/projected/2f4de4f9-c0dd-418c-8157-c2f25a2b15eb-kube-api-access-d4fkq\") pod \"maas-api-revoke-keys-e2e-inf-a-22e11b-6j9ps\" (UID: \"2f4de4f9-c0dd-418c-8157-c2f25a2b15eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22e11b-6j9ps"
Nov 01 03:51:54.625504 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:54.625485    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f4de4f9-c0dd-418c-8157-c2f25a2b15eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-22e11b-6j9ps\" (UID: \"2f4de4f9-c0dd-418c-8157-c2f25a2b15eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22e11b-6j9ps"
Nov 01 03:51:54.627094 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:54.627070    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f4de4f9-c0dd-418c-8157-c2f25a2b15eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-22e11b-6j9ps\" (UID: \"2f4de4f9-c0dd-418c-8157-c2f25a2b15eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22e11b-6j9ps"
Nov 01 03:51:54.692535 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:54.692509    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4fkq\" (UniqueName: \"kubernetes.io/projected/2f4de4f9-c0dd-418c-8157-c2f25a2b15eb-kube-api-access-d4fkq\") pod \"maas-api-revoke-keys-e2e-inf-a-22e11b-6j9ps\" (UID: \"2f4de4f9-c0dd-418c-8157-c2f25a2b15eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22e11b-6j9ps"
Nov 01 03:51:54.988467 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:54.988381    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22e11b-6j9ps"
Nov 01 03:51:55.384713 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:55.384548    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22e11b-6j9ps"]
Nov 01 03:51:55.388716 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:51:55.388676    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f4de4f9_c0dd_418c_8157_c2f25a2b15eb.slice/crio-b135f7e9611582ab67414f07481cf7dc24fcc5b06b4a9439e00f2b5fde43d444 WatchSource:0}: Error finding container b135f7e9611582ab67414f07481cf7dc24fcc5b06b4a9439e00f2b5fde43d444: Status 404 returned error can't find the container with id b135f7e9611582ab67414f07481cf7dc24fcc5b06b4a9439e00f2b5fde43d444
Nov 01 03:51:56.368750 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:56.368713    4930 generic.go:356] "Generic (PLEG): container finished" podID="2f4de4f9-c0dd-418c-8157-c2f25a2b15eb" containerID="ec6a1ccf54c90759226769d7a212a2b6e80f13e040497d4f566b7cbe252d42fa" exitCode=0
Nov 01 03:51:56.369162 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:56.368797    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22e11b-6j9ps" event={"ID":"2f4de4f9-c0dd-418c-8157-c2f25a2b15eb","Type":"ContainerDied","Data":"ec6a1ccf54c90759226769d7a212a2b6e80f13e040497d4f566b7cbe252d42fa"}
Nov 01 03:51:56.369162 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:56.368858    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22e11b-6j9ps" event={"ID":"2f4de4f9-c0dd-418c-8157-c2f25a2b15eb","Type":"ContainerStarted","Data":"b135f7e9611582ab67414f07481cf7dc24fcc5b06b4a9439e00f2b5fde43d444"}
Nov 01 03:51:57.072719 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:57.072681    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:51:57.576692 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:57.576663    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22e11b-6j9ps"
Nov 01 03:51:57.671672 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:57.671584    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f4de4f9-c0dd-418c-8157-c2f25a2b15eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f4de4f9-c0dd-418c-8157-c2f25a2b15eb-maas-api-xks-ca\") pod \"2f4de4f9-c0dd-418c-8157-c2f25a2b15eb\" (UID: \"2f4de4f9-c0dd-418c-8157-c2f25a2b15eb\") "
Nov 01 03:51:57.673710 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:57.673677    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f4de4f9-c0dd-418c-8157-c2f25a2b15eb-maas-api-xks-ca" pod "2f4de4f9-c0dd-418c-8157-c2f25a2b15eb" (UID: "2f4de4f9-c0dd-418c-8157-c2f25a2b15eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:51:57.772471 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:57.772427    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f4de4f9-c0dd-418c-8157-c2f25a2b15eb-kube-api-access-d4fkq\" (UniqueName: \"kubernetes.io/projected/2f4de4f9-c0dd-418c-8157-c2f25a2b15eb-kube-api-access-d4fkq\") pod \"2f4de4f9-c0dd-418c-8157-c2f25a2b15eb\" (UID: \"2f4de4f9-c0dd-418c-8157-c2f25a2b15eb\") "
Nov 01 03:51:57.772658 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:57.772525    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f4de4f9-c0dd-418c-8157-c2f25a2b15eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f4de4f9-c0dd-418c-8157-c2f25a2b15eb-maas-api-openshift-ca\") pod \"2f4de4f9-c0dd-418c-8157-c2f25a2b15eb\" (UID: \"2f4de4f9-c0dd-418c-8157-c2f25a2b15eb\") "
Nov 01 03:51:57.773193 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:57.773146    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f4de4f9-c0dd-418c-8157-c2f25a2b15eb-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:57.773379 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:57.773345    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f4de4f9-c0dd-418c-8157-c2f25a2b15eb-maas-api-openshift-ca" pod "2f4de4f9-c0dd-418c-8157-c2f25a2b15eb" (UID: "2f4de4f9-c0dd-418c-8157-c2f25a2b15eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:51:57.775341 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:57.775315    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f4de4f9-c0dd-418c-8157-c2f25a2b15eb-kube-api-access-d4fkq" pod "2f4de4f9-c0dd-418c-8157-c2f25a2b15eb" (UID: "2f4de4f9-c0dd-418c-8157-c2f25a2b15eb"). InnerVolumeSpecName "kube-api-access-d4fkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:51:57.874067 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:57.874040    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f4de4f9-c0dd-418c-8157-c2f25a2b15eb-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:57.874067 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:57.874065    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4fkq\" (UniqueName: \"kubernetes.io/projected/2f4de4f9-c0dd-418c-8157-c2f25a2b15eb-kube-api-access-d4fkq\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:51:57.987178 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:57.987098    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:51:58.137264 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:58.137223    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:51:58.354380 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:58.354340    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:51:58.403252 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:58.403223    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22e11b-6j9ps"
Nov 01 03:51:58.403409 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:58.403240    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22e11b-6j9ps" event={"ID":"2f4de4f9-c0dd-418c-8157-c2f25a2b15eb","Type":"ContainerDied","Data":"b135f7e9611582ab67414f07481cf7dc24fcc5b06b4a9439e00f2b5fde43d444"}
Nov 01 03:51:58.403409 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:58.403306    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b135f7e9611582ab67414f07481cf7dc24fcc5b06b4a9439e00f2b5fde43d444"
Nov 01 03:51:58.615222 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:58.615135    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:51:58.852336 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:58.852298    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:51:58.996569 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:58.996471    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:51:59.192509 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:59.192470    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-170ffa-wpl7d"]
Nov 01 03:51:59.242165 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:59.242131    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-170ffa-wpl7d"]
Nov 01 03:51:59.605156 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:59.605114    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:51:59.620088 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:51:59.620059    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11338423-6bc3-42d6-8384-b438393fcda7" path="/var/lib/kubelet/pods/11338423-6bc3-42d6-8384-b438393fcda7/volumes"
Nov 01 03:52:00.121402 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:00.121373    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:00.271291 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:00.271250    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:00.568644 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:00.568533    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:00.968055 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:00.968022    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:01.455045 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:01.455012    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ded2d7-csmzf"]
Nov 01 03:52:01.455586 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:01.455568    4930 cpu_manager.go:414] "containerMap: removing container" podUID="2f4de4f9-c0dd-418c-8157-c2f25a2b15eb" containerName="revoke-keys"
Nov 01 03:52:01.455586 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:01.455583    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f4de4f9-c0dd-418c-8157-c2f25a2b15eb" containerName="revoke-keys"
Nov 01 03:52:01.455741 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:01.455682    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f4de4f9-c0dd-418c-8157-c2f25a2b15eb" containerName="revoke-keys"
Nov 01 03:52:01.460286 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:01.460261    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ded2d7-csmzf"
Nov 01 03:52:01.520037 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:01.520002    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ded2d7-csmzf"]
Nov 01 03:52:01.522597 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:01.522541    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsvvm\" (UniqueName: \"kubernetes.io/projected/3dcac01b-ed97-47ca-aa26-ed00285418bb-kube-api-access-xsvvm\") pod \"maas-api-revoke-keys-e2e-worker-w0-ded2d7-csmzf\" (UID: \"3dcac01b-ed97-47ca-aa26-ed00285418bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ded2d7-csmzf"
Nov 01 03:52:01.522757 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:01.522690    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dcac01b-ed97-47ca-aa26-ed00285418bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ded2d7-csmzf\" (UID: \"3dcac01b-ed97-47ca-aa26-ed00285418bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ded2d7-csmzf"
Nov 01 03:52:01.522835 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:01.522814    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dcac01b-ed97-47ca-aa26-ed00285418bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ded2d7-csmzf\" (UID: \"3dcac01b-ed97-47ca-aa26-ed00285418bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ded2d7-csmzf"
Nov 01 03:52:01.565339 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:01.565306    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:52:01.565339 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:01.565331    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:52:01.565339 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:01.565311    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:52:01.565599 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:01.565380    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:52:01.624489 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:01.624453    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dcac01b-ed97-47ca-aa26-ed00285418bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ded2d7-csmzf\" (UID: \"3dcac01b-ed97-47ca-aa26-ed00285418bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ded2d7-csmzf"
Nov 01 03:52:01.624712 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:01.624614    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dcac01b-ed97-47ca-aa26-ed00285418bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ded2d7-csmzf\" (UID: \"3dcac01b-ed97-47ca-aa26-ed00285418bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ded2d7-csmzf"
Nov 01 03:52:01.624795 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:01.624738    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsvvm\" (UniqueName: \"kubernetes.io/projected/3dcac01b-ed97-47ca-aa26-ed00285418bb-kube-api-access-xsvvm\") pod \"maas-api-revoke-keys-e2e-worker-w0-ded2d7-csmzf\" (UID: \"3dcac01b-ed97-47ca-aa26-ed00285418bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ded2d7-csmzf"
Nov 01 03:52:01.625217 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:01.625191    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dcac01b-ed97-47ca-aa26-ed00285418bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ded2d7-csmzf\" (UID: \"3dcac01b-ed97-47ca-aa26-ed00285418bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ded2d7-csmzf"
Nov 01 03:52:01.626977 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:01.626955    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dcac01b-ed97-47ca-aa26-ed00285418bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ded2d7-csmzf\" (UID: \"3dcac01b-ed97-47ca-aa26-ed00285418bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ded2d7-csmzf"
Nov 01 03:52:01.697109 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:01.697074    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsvvm\" (UniqueName: \"kubernetes.io/projected/3dcac01b-ed97-47ca-aa26-ed00285418bb-kube-api-access-xsvvm\") pod \"maas-api-revoke-keys-e2e-worker-w0-ded2d7-csmzf\" (UID: \"3dcac01b-ed97-47ca-aa26-ed00285418bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ded2d7-csmzf"
Nov 01 03:52:01.775502 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:01.775423    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ded2d7-csmzf"
Nov 01 03:52:02.152180 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:02.152150    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ded2d7-csmzf"]
Nov 01 03:52:02.154149 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:52:02.154116    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dcac01b_ed97_47ca_aa26_ed00285418bb.slice/crio-b72b61f40384bd69dddec67cc5589bc815d52bdb0cddccfb2003a1105af80aad WatchSource:0}: Error finding container b72b61f40384bd69dddec67cc5589bc815d52bdb0cddccfb2003a1105af80aad: Status 404 returned error can't find the container with id b72b61f40384bd69dddec67cc5589bc815d52bdb0cddccfb2003a1105af80aad
Nov 01 03:52:02.448669 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:02.448584    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ded2d7-csmzf" event={"ID":"3dcac01b-ed97-47ca-aa26-ed00285418bb","Type":"ContainerStarted","Data":"b72b61f40384bd69dddec67cc5589bc815d52bdb0cddccfb2003a1105af80aad"}
Nov 01 03:52:03.457167 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:03.457084    4930 generic.go:356] "Generic (PLEG): container finished" podID="3dcac01b-ed97-47ca-aa26-ed00285418bb" containerID="5a8de80b665c7cf0aa520f9c7d5f02f2f10bd055ad87d5e5b6b2db18f0c7338d" exitCode=0
Nov 01 03:52:03.457534 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:03.457162    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ded2d7-csmzf" event={"ID":"3dcac01b-ed97-47ca-aa26-ed00285418bb","Type":"ContainerDied","Data":"5a8de80b665c7cf0aa520f9c7d5f02f2f10bd055ad87d5e5b6b2db18f0c7338d"}
Nov 01 03:52:04.225096 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:04.225046    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:04.623429 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:04.623401    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ded2d7-csmzf"
Nov 01 03:52:04.661844 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:04.661810    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dcac01b-ed97-47ca-aa26-ed00285418bb-kube-api-access-xsvvm\" (UniqueName: \"kubernetes.io/projected/3dcac01b-ed97-47ca-aa26-ed00285418bb-kube-api-access-xsvvm\") pod \"3dcac01b-ed97-47ca-aa26-ed00285418bb\" (UID: \"3dcac01b-ed97-47ca-aa26-ed00285418bb\") "
Nov 01 03:52:04.662006 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:04.661988    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dcac01b-ed97-47ca-aa26-ed00285418bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dcac01b-ed97-47ca-aa26-ed00285418bb-maas-api-xks-ca\") pod \"3dcac01b-ed97-47ca-aa26-ed00285418bb\" (UID: \"3dcac01b-ed97-47ca-aa26-ed00285418bb\") "
Nov 01 03:52:04.662134 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:04.662117    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dcac01b-ed97-47ca-aa26-ed00285418bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dcac01b-ed97-47ca-aa26-ed00285418bb-maas-api-openshift-ca\") pod \"3dcac01b-ed97-47ca-aa26-ed00285418bb\" (UID: \"3dcac01b-ed97-47ca-aa26-ed00285418bb\") "
Nov 01 03:52:04.662538 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:04.662509    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dcac01b-ed97-47ca-aa26-ed00285418bb-maas-api-openshift-ca" pod "3dcac01b-ed97-47ca-aa26-ed00285418bb" (UID: "3dcac01b-ed97-47ca-aa26-ed00285418bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:52:04.664168 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:04.664138    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dcac01b-ed97-47ca-aa26-ed00285418bb-kube-api-access-xsvvm" pod "3dcac01b-ed97-47ca-aa26-ed00285418bb" (UID: "3dcac01b-ed97-47ca-aa26-ed00285418bb"). InnerVolumeSpecName "kube-api-access-xsvvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:52:04.664281 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:04.664141    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dcac01b-ed97-47ca-aa26-ed00285418bb-maas-api-xks-ca" pod "3dcac01b-ed97-47ca-aa26-ed00285418bb" (UID: "3dcac01b-ed97-47ca-aa26-ed00285418bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:52:04.739351 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:04.739312    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:04.763592 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:04.763545    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dcac01b-ed97-47ca-aa26-ed00285418bb-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:04.763592 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:04.763595    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dcac01b-ed97-47ca-aa26-ed00285418bb-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:04.763786 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:04.763610    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsvvm\" (UniqueName: \"kubernetes.io/projected/3dcac01b-ed97-47ca-aa26-ed00285418bb-kube-api-access-xsvvm\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:05.474883 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:05.474850    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ded2d7-csmzf"
Nov 01 03:52:05.474883 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:05.474873    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ded2d7-csmzf" event={"ID":"3dcac01b-ed97-47ca-aa26-ed00285418bb","Type":"ContainerDied","Data":"b72b61f40384bd69dddec67cc5589bc815d52bdb0cddccfb2003a1105af80aad"}
Nov 01 03:52:05.475102 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:05.474965    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b72b61f40384bd69dddec67cc5589bc815d52bdb0cddccfb2003a1105af80aad"
Nov 01 03:52:05.877147 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:05.877113    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-22e11b-6b684b568c-jvmc9"]
Nov 01 03:52:05.917482 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:05.917447    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"]
Nov 01 03:52:05.926306 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:05.926273    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"]
Nov 01 03:52:05.969395 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:05.969362    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"]
Nov 01 03:52:05.974053 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:05.974020    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"]
Nov 01 03:52:05.983477 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:05.983444    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-22e11b-5ff5d6f995-fkzzs"]
Nov 01 03:52:06.019441 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:06.019409    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:06.362105 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:06.362072    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:06.518015 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:06.517962    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:06.676549 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:06.676467    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:06.897395 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:06.897358    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"]
Nov 01 03:52:06.897950 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:06.897639    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq" podUID="df136678-b446-4fe6-8705-48d36455f70a" containerName="istio-proxy" containerID="cri-o://60d8d43549422f0cf244a102fd44e6d5b3f36f78d8dde4d60b828c9761abc4d2" gracePeriod=30
Nov 01 03:52:06.920738 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:06.920707    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:07.855253 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:07.855217    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-7cd000-wxqql"]
Nov 01 03:52:07.855882 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:07.855860    4930 cpu_manager.go:414] "containerMap: removing container" podUID="3dcac01b-ed97-47ca-aa26-ed00285418bb" containerName="revoke-keys"
Nov 01 03:52:07.855882 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:07.855882    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dcac01b-ed97-47ca-aa26-ed00285418bb" containerName="revoke-keys"
Nov 01 03:52:07.856070 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:07.856057    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dcac01b-ed97-47ca-aa26-ed00285418bb" containerName="revoke-keys"
Nov 01 03:52:07.859595 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:07.859574    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-7cd000-wxqql"
Nov 01 03:52:07.915000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:07.914977    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:52:07.918482 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:07.918462    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:52:07.918624 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:07.918496    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:52:07.930223 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:07.930195    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcnp5\" (UniqueName: \"kubernetes.io/projected/33b901de-97ee-4363-9fac-f96499633f86-kube-api-access-zcnp5\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-7cd000-wxqql\" (UID: \"33b901de-97ee-4363-9fac-f96499633f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-7cd000-wxqql"
Nov 01 03:52:07.930326 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:07.930277    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b901de-97ee-4363-9fac-f96499633f86-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-7cd000-wxqql\" (UID: \"33b901de-97ee-4363-9fac-f96499633f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-7cd000-wxqql"
Nov 01 03:52:07.930377 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:07.930356    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b901de-97ee-4363-9fac-f96499633f86-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-7cd000-wxqql\" (UID: \"33b901de-97ee-4363-9fac-f96499633f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-7cd000-wxqql"
Nov 01 03:52:07.964511 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:07.964485    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:52:07.968245 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:07.968220    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-7cd000-wxqql"]
Nov 01 03:52:08.031787 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:08.031757    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcnp5\" (UniqueName: \"kubernetes.io/projected/33b901de-97ee-4363-9fac-f96499633f86-kube-api-access-zcnp5\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-7cd000-wxqql\" (UID: \"33b901de-97ee-4363-9fac-f96499633f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-7cd000-wxqql"
Nov 01 03:52:08.031898 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:08.031813    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b901de-97ee-4363-9fac-f96499633f86-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-7cd000-wxqql\" (UID: \"33b901de-97ee-4363-9fac-f96499633f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-7cd000-wxqql"
Nov 01 03:52:08.031898 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:08.031857    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b901de-97ee-4363-9fac-f96499633f86-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-7cd000-wxqql\" (UID: \"33b901de-97ee-4363-9fac-f96499633f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-7cd000-wxqql"
Nov 01 03:52:08.032492 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:08.032475    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b901de-97ee-4363-9fac-f96499633f86-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-7cd000-wxqql\" (UID: \"33b901de-97ee-4363-9fac-f96499633f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-7cd000-wxqql"
Nov 01 03:52:08.034248 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:08.034223    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b901de-97ee-4363-9fac-f96499633f86-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-7cd000-wxqql\" (UID: \"33b901de-97ee-4363-9fac-f96499633f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-7cd000-wxqql"
Nov 01 03:52:08.092703 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:08.092679    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcnp5\" (UniqueName: \"kubernetes.io/projected/33b901de-97ee-4363-9fac-f96499633f86-kube-api-access-zcnp5\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-7cd000-wxqql\" (UID: \"33b901de-97ee-4363-9fac-f96499633f86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-7cd000-wxqql"
Nov 01 03:52:08.174403 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:08.174341    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-7cd000-wxqql"
Nov 01 03:52:08.556869 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:08.556835    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-7cd000-wxqql"]
Nov 01 03:52:08.557117 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:52:08.557092    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b901de_97ee_4363_9fac_f96499633f86.slice/crio-73ad9d6122a29f871812443d57861c5172ee77f78b2a6a38a357c0e0a0a574ba WatchSource:0}: Error finding container 73ad9d6122a29f871812443d57861c5172ee77f78b2a6a38a357c0e0a0a574ba: Status 404 returned error can't find the container with id 73ad9d6122a29f871812443d57861c5172ee77f78b2a6a38a357c0e0a0a574ba
Nov 01 03:52:09.130055 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:09.129972    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d55d5e8-lhq9q"]
Nov 01 03:52:09.133976 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:09.133955    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d55d5e8-lhq9q"
Nov 01 03:52:09.192460 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:09.192418    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d55d5e8-lhq9q"]
Nov 01 03:52:09.243956 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:09.243928    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d55d5e8-lhq9q\" (UID: \"6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d55d5e8-lhq9q"
Nov 01 03:52:09.244104 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:09.244081    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvctt\" (UniqueName: \"kubernetes.io/projected/6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b-kube-api-access-gvctt\") pod \"maas-api-revoke-sub-2d55d5e8-lhq9q\" (UID: \"6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d55d5e8-lhq9q"
Nov 01 03:52:09.244162 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:09.244109    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d55d5e8-lhq9q\" (UID: \"6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d55d5e8-lhq9q"
Nov 01 03:52:09.344965 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:09.344925    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvctt\" (UniqueName: \"kubernetes.io/projected/6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b-kube-api-access-gvctt\") pod \"maas-api-revoke-sub-2d55d5e8-lhq9q\" (UID: \"6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d55d5e8-lhq9q"
Nov 01 03:52:09.345154 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:09.344976    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d55d5e8-lhq9q\" (UID: \"6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d55d5e8-lhq9q"
Nov 01 03:52:09.345154 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:09.345075    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d55d5e8-lhq9q\" (UID: \"6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d55d5e8-lhq9q"
Nov 01 03:52:09.345723 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:09.345704    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d55d5e8-lhq9q\" (UID: \"6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d55d5e8-lhq9q"
Nov 01 03:52:09.347460 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:09.347440    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d55d5e8-lhq9q\" (UID: \"6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d55d5e8-lhq9q"
Nov 01 03:52:09.405537 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:09.405473    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvctt\" (UniqueName: \"kubernetes.io/projected/6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b-kube-api-access-gvctt\") pod \"maas-api-revoke-sub-2d55d5e8-lhq9q\" (UID: \"6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d55d5e8-lhq9q"
Nov 01 03:52:09.449205 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:09.449176    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d55d5e8-lhq9q"
Nov 01 03:52:09.550796 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:09.550754    4930 generic.go:356] "Generic (PLEG): container finished" podID="33b901de-97ee-4363-9fac-f96499633f86" containerID="92526190e471c1557608858d0ba19fad653500ee19005c7e48a91659dbd475e5" exitCode=0
Nov 01 03:52:09.550980 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:09.550830    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-7cd000-wxqql" event={"ID":"33b901de-97ee-4363-9fac-f96499633f86","Type":"ContainerDied","Data":"92526190e471c1557608858d0ba19fad653500ee19005c7e48a91659dbd475e5"}
Nov 01 03:52:09.550980 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:09.550894    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-7cd000-wxqql" event={"ID":"33b901de-97ee-4363-9fac-f96499633f86","Type":"ContainerStarted","Data":"73ad9d6122a29f871812443d57861c5172ee77f78b2a6a38a357c0e0a0a574ba"}
Nov 01 03:52:09.829324 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:09.829293    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d55d5e8-lhq9q"]
Nov 01 03:52:09.830487 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:52:09.830459    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bcfdfce_7c5c_4aa0_91a1_0ff72f08aa1b.slice/crio-6b1e5aac67a2f3e7f2f01fe5472631606044dba8a395d5492c152c9090fd0f0f WatchSource:0}: Error finding container 6b1e5aac67a2f3e7f2f01fe5472631606044dba8a395d5492c152c9090fd0f0f: Status 404 returned error can't find the container with id 6b1e5aac67a2f3e7f2f01fe5472631606044dba8a395d5492c152c9090fd0f0f
Nov 01 03:52:10.002938 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:10.002913    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:10.560460 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:10.560347    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d55d5e8-lhq9q" event={"ID":"6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b","Type":"ContainerStarted","Data":"fa688ebb814860864c8eef6e3bd4dbd58a9e7ce3fb34a1361f43b3bf3bd3deaa"}
Nov 01 03:52:10.560460 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:10.560446    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d55d5e8-lhq9q" event={"ID":"6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b","Type":"ContainerStarted","Data":"6b1e5aac67a2f3e7f2f01fe5472631606044dba8a395d5492c152c9090fd0f0f"}
Nov 01 03:52:10.690290 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:10.690234    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d55d5e8-lhq9q" podStartSLOduration=1.690219879 podStartE2EDuration="1.690219879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:52:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:52:10.686044954 +0000 UTC m=+2735.209051543" watchObservedRunningTime="2026-11-01 03:52:10.690219879 +0000 UTC m=+2735.213226466"
Nov 01 03:52:10.706303 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:10.706276    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-7cd000-wxqql"
Nov 01 03:52:10.765880 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:10.765842    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33b901de-97ee-4363-9fac-f96499633f86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b901de-97ee-4363-9fac-f96499633f86-maas-api-openshift-ca\") pod \"33b901de-97ee-4363-9fac-f96499633f86\" (UID: \"33b901de-97ee-4363-9fac-f96499633f86\") "
Nov 01 03:52:10.766019 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:10.765886    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33b901de-97ee-4363-9fac-f96499633f86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b901de-97ee-4363-9fac-f96499633f86-maas-api-xks-ca\") pod \"33b901de-97ee-4363-9fac-f96499633f86\" (UID: \"33b901de-97ee-4363-9fac-f96499633f86\") "
Nov 01 03:52:10.766086 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:10.766062    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33b901de-97ee-4363-9fac-f96499633f86-kube-api-access-zcnp5\" (UniqueName: \"kubernetes.io/projected/33b901de-97ee-4363-9fac-f96499633f86-kube-api-access-zcnp5\") pod \"33b901de-97ee-4363-9fac-f96499633f86\" (UID: \"33b901de-97ee-4363-9fac-f96499633f86\") "
Nov 01 03:52:10.766220 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:10.766199    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33b901de-97ee-4363-9fac-f96499633f86-maas-api-openshift-ca" pod "33b901de-97ee-4363-9fac-f96499633f86" (UID: "33b901de-97ee-4363-9fac-f96499633f86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:52:10.766549 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:10.766523    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b901de-97ee-4363-9fac-f96499633f86-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:10.768012 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:10.767989    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33b901de-97ee-4363-9fac-f96499633f86-maas-api-xks-ca" pod "33b901de-97ee-4363-9fac-f96499633f86" (UID: "33b901de-97ee-4363-9fac-f96499633f86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:52:10.768180 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:10.768155    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b901de-97ee-4363-9fac-f96499633f86-kube-api-access-zcnp5" pod "33b901de-97ee-4363-9fac-f96499633f86" (UID: "33b901de-97ee-4363-9fac-f96499633f86"). InnerVolumeSpecName "kube-api-access-zcnp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:52:10.809544 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:10.809519    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:10.867964 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:10.867932    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b901de-97ee-4363-9fac-f96499633f86-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:10.867964 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:10.867964    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcnp5\" (UniqueName: \"kubernetes.io/projected/33b901de-97ee-4363-9fac-f96499633f86-kube-api-access-zcnp5\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:10.878355 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:10.878324    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-22e11b-6b684b568c-jvmc9" podUID="d91bba7f-304b-424e-a819-d78a9275f6be" containerName="payload-processing" containerID="cri-o://b1b926ecb93a81542e99b31821d658d34eca6e07c62d5ad26f68656f96488a59" gracePeriod=25
Nov 01 03:52:10.968353 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:10.968296    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:10.984110 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:10.984063    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22e11b-5ff5d6f995-fkzzs" podUID="03b489d4-3a39-4830-8629-d2a41315182d" containerName="payload-pre-processing" containerID="cri-o://8e475a0c4374bd50c56b34d778bf910a563369859d9df2ded6999f3cc97a2bba" gracePeriod=25
Nov 01 03:52:11.149139 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:11.149111    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-22e11b-6b684b568c-jvmc9"
Nov 01 03:52:11.286581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:11.272511    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d91bba7f-304b-424e-a819-d78a9275f6be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d91bba7f-304b-424e-a819-d78a9275f6be-plugins-config-volume\") pod \"d91bba7f-304b-424e-a819-d78a9275f6be\" (UID: \"d91bba7f-304b-424e-a819-d78a9275f6be\") "
Nov 01 03:52:11.286581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:11.272624    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d91bba7f-304b-424e-a819-d78a9275f6be-kube-api-access-nlj2z\" (UniqueName: \"kubernetes.io/projected/d91bba7f-304b-424e-a819-d78a9275f6be-kube-api-access-nlj2z\") pod \"d91bba7f-304b-424e-a819-d78a9275f6be\" (UID: \"d91bba7f-304b-424e-a819-d78a9275f6be\") "
Nov 01 03:52:11.286581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:11.273703    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d91bba7f-304b-424e-a819-d78a9275f6be-plugins-config-volume" pod "d91bba7f-304b-424e-a819-d78a9275f6be" (UID: "d91bba7f-304b-424e-a819-d78a9275f6be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:52:11.290772 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:11.290123    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d91bba7f-304b-424e-a819-d78a9275f6be-kube-api-access-nlj2z" pod "d91bba7f-304b-424e-a819-d78a9275f6be" (UID: "d91bba7f-304b-424e-a819-d78a9275f6be"). InnerVolumeSpecName "kube-api-access-nlj2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:52:11.374048 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:11.373972    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlj2z\" (UniqueName: \"kubernetes.io/projected/d91bba7f-304b-424e-a819-d78a9275f6be-kube-api-access-nlj2z\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:11.374048 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:11.374001    4930 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d91bba7f-304b-424e-a819-d78a9275f6be-plugins-config-volume\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:11.571949 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:11.571916    4930 generic.go:356] "Generic (PLEG): container finished" podID="d91bba7f-304b-424e-a819-d78a9275f6be" containerID="b1b926ecb93a81542e99b31821d658d34eca6e07c62d5ad26f68656f96488a59" exitCode=0
Nov 01 03:52:11.572406 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:11.571966    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-22e11b-6b684b568c-jvmc9"
Nov 01 03:52:11.572406 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:11.571997    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-22e11b-6b684b568c-jvmc9" event={"ID":"d91bba7f-304b-424e-a819-d78a9275f6be","Type":"ContainerDied","Data":"b1b926ecb93a81542e99b31821d658d34eca6e07c62d5ad26f68656f96488a59"}
Nov 01 03:52:11.572406 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:11.572060    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-22e11b-6b684b568c-jvmc9" event={"ID":"d91bba7f-304b-424e-a819-d78a9275f6be","Type":"ContainerDied","Data":"d9c41f0ca27b9bf7e42ed60391536d0c2084c6c02064fab21415e611689707a4"}
Nov 01 03:52:11.572406 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:11.572096    4930 scope.go:122] "RemoveContainer" containerID="b1b926ecb93a81542e99b31821d658d34eca6e07c62d5ad26f68656f96488a59"
Nov 01 03:52:11.573709 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:11.573687    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-7cd000-wxqql"
Nov 01 03:52:11.573851 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:11.573710    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-7cd000-wxqql" event={"ID":"33b901de-97ee-4363-9fac-f96499633f86","Type":"ContainerDied","Data":"73ad9d6122a29f871812443d57861c5172ee77f78b2a6a38a357c0e0a0a574ba"}
Nov 01 03:52:11.573851 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:11.573750    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73ad9d6122a29f871812443d57861c5172ee77f78b2a6a38a357c0e0a0a574ba"
Nov 01 03:52:11.581072 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:11.581048    4930 generic.go:356] "Generic (PLEG): container finished" podID="6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b" containerID="fa688ebb814860864c8eef6e3bd4dbd58a9e7ce3fb34a1361f43b3bf3bd3deaa" exitCode=0
Nov 01 03:52:11.581224 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:11.581130    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d55d5e8-lhq9q" event={"ID":"6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b","Type":"ContainerDied","Data":"fa688ebb814860864c8eef6e3bd4dbd58a9e7ce3fb34a1361f43b3bf3bd3deaa"}
Nov 01 03:52:11.584251 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:11.584230    4930 scope.go:122] "RemoveContainer" containerID="b1b926ecb93a81542e99b31821d658d34eca6e07c62d5ad26f68656f96488a59"
Nov 01 03:52:11.584522 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:52:11.584500    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1b926ecb93a81542e99b31821d658d34eca6e07c62d5ad26f68656f96488a59\": container with ID starting with b1b926ecb93a81542e99b31821d658d34eca6e07c62d5ad26f68656f96488a59 not found: ID does not exist" containerID="b1b926ecb93a81542e99b31821d658d34eca6e07c62d5ad26f68656f96488a59"
Nov 01 03:52:11.584620 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:11.584530    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1b926ecb93a81542e99b31821d658d34eca6e07c62d5ad26f68656f96488a59"} err="failed to get container status \"b1b926ecb93a81542e99b31821d658d34eca6e07c62d5ad26f68656f96488a59\": rpc error: code = NotFound desc = could not find container \"b1b926ecb93a81542e99b31821d658d34eca6e07c62d5ad26f68656f96488a59\": container with ID starting with b1b926ecb93a81542e99b31821d658d34eca6e07c62d5ad26f68656f96488a59 not found: ID does not exist"
Nov 01 03:52:11.910247 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:11.910182    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-22e11b-6b684b568c-jvmc9"]
Nov 01 03:52:11.914199 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:11.914175    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-22e11b-6b684b568c-jvmc9"]
Nov 01 03:52:12.044966 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.044178    4930 patch_prober.go:29] interesting pod/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Nov 01 03:52:12.044966 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.044263    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq" podUID="df136678-b446-4fe6-8705-48d36455f70a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Nov 01 03:52:12.164618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.164528    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"
Nov 01 03:52:12.289649 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.289616    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df136678-b446-4fe6-8705-48d36455f70a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df136678-b446-4fe6-8705-48d36455f70a-credential-socket\") pod \"df136678-b446-4fe6-8705-48d36455f70a\" (UID: \"df136678-b446-4fe6-8705-48d36455f70a\") "
Nov 01 03:52:12.289836 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.289725    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df136678-b446-4fe6-8705-48d36455f70a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df136678-b446-4fe6-8705-48d36455f70a-istio-podinfo\") pod \"df136678-b446-4fe6-8705-48d36455f70a\" (UID: \"df136678-b446-4fe6-8705-48d36455f70a\") "
Nov 01 03:52:12.289836 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.289762    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df136678-b446-4fe6-8705-48d36455f70a-istio-token\" (UniqueName: \"kubernetes.io/projected/df136678-b446-4fe6-8705-48d36455f70a-istio-token\") pod \"df136678-b446-4fe6-8705-48d36455f70a\" (UID: \"df136678-b446-4fe6-8705-48d36455f70a\") "
Nov 01 03:52:12.289836 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.289787    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df136678-b446-4fe6-8705-48d36455f70a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df136678-b446-4fe6-8705-48d36455f70a-workload-socket\") pod \"df136678-b446-4fe6-8705-48d36455f70a\" (UID: \"df136678-b446-4fe6-8705-48d36455f70a\") "
Nov 01 03:52:12.290017 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.289876    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df136678-b446-4fe6-8705-48d36455f70a-credential-socket" pod "df136678-b446-4fe6-8705-48d36455f70a" (UID: "df136678-b446-4fe6-8705-48d36455f70a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:52:12.290017 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.289921    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df136678-b446-4fe6-8705-48d36455f70a-kube-api-access-2dzxb\" (UniqueName: \"kubernetes.io/projected/df136678-b446-4fe6-8705-48d36455f70a-kube-api-access-2dzxb\") pod \"df136678-b446-4fe6-8705-48d36455f70a\" (UID: \"df136678-b446-4fe6-8705-48d36455f70a\") "
Nov 01 03:52:12.290017 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.289964    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df136678-b446-4fe6-8705-48d36455f70a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df136678-b446-4fe6-8705-48d36455f70a-workload-certs\") pod \"df136678-b446-4fe6-8705-48d36455f70a\" (UID: \"df136678-b446-4fe6-8705-48d36455f70a\") "
Nov 01 03:52:12.290017 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.290003    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df136678-b446-4fe6-8705-48d36455f70a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df136678-b446-4fe6-8705-48d36455f70a-istiod-ca-cert\") pod \"df136678-b446-4fe6-8705-48d36455f70a\" (UID: \"df136678-b446-4fe6-8705-48d36455f70a\") "
Nov 01 03:52:12.290546 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.290060    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df136678-b446-4fe6-8705-48d36455f70a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df136678-b446-4fe6-8705-48d36455f70a-istio-envoy\") pod \"df136678-b446-4fe6-8705-48d36455f70a\" (UID: \"df136678-b446-4fe6-8705-48d36455f70a\") "
Nov 01 03:52:12.290546 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.290091    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df136678-b446-4fe6-8705-48d36455f70a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df136678-b446-4fe6-8705-48d36455f70a-istio-data\") pod \"df136678-b446-4fe6-8705-48d36455f70a\" (UID: \"df136678-b446-4fe6-8705-48d36455f70a\") "
Nov 01 03:52:12.290546 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.290300    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df136678-b446-4fe6-8705-48d36455f70a-workload-certs" pod "df136678-b446-4fe6-8705-48d36455f70a" (UID: "df136678-b446-4fe6-8705-48d36455f70a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:52:12.291289 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.290993    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df136678-b446-4fe6-8705-48d36455f70a-workload-socket" pod "df136678-b446-4fe6-8705-48d36455f70a" (UID: "df136678-b446-4fe6-8705-48d36455f70a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:52:12.292327 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.291442    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df136678-b446-4fe6-8705-48d36455f70a-istiod-ca-cert" pod "df136678-b446-4fe6-8705-48d36455f70a" (UID: "df136678-b446-4fe6-8705-48d36455f70a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:52:12.292327 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.291583    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df136678-b446-4fe6-8705-48d36455f70a-istio-data" pod "df136678-b446-4fe6-8705-48d36455f70a" (UID: "df136678-b446-4fe6-8705-48d36455f70a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:52:12.292327 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.292036    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df136678-b446-4fe6-8705-48d36455f70a-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:12.292327 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.292060    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df136678-b446-4fe6-8705-48d36455f70a-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:12.292327 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.292075    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df136678-b446-4fe6-8705-48d36455f70a-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:12.292327 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.292090    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df136678-b446-4fe6-8705-48d36455f70a-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:12.292327 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.292103    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df136678-b446-4fe6-8705-48d36455f70a-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:12.293772 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.293745    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df136678-b446-4fe6-8705-48d36455f70a-istio-envoy" pod "df136678-b446-4fe6-8705-48d36455f70a" (UID: "df136678-b446-4fe6-8705-48d36455f70a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:52:12.294200 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.294177    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df136678-b446-4fe6-8705-48d36455f70a-kube-api-access-2dzxb" pod "df136678-b446-4fe6-8705-48d36455f70a" (UID: "df136678-b446-4fe6-8705-48d36455f70a"). InnerVolumeSpecName "kube-api-access-2dzxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:52:12.294305 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.294255    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df136678-b446-4fe6-8705-48d36455f70a-istio-podinfo" pod "df136678-b446-4fe6-8705-48d36455f70a" (UID: "df136678-b446-4fe6-8705-48d36455f70a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:52:12.297166 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.297143    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df136678-b446-4fe6-8705-48d36455f70a-istio-token" pod "df136678-b446-4fe6-8705-48d36455f70a" (UID: "df136678-b446-4fe6-8705-48d36455f70a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:52:12.393366 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.393334    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df136678-b446-4fe6-8705-48d36455f70a-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:12.393366 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.393364    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df136678-b446-4fe6-8705-48d36455f70a-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:12.393542 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.393374    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dzxb\" (UniqueName: \"kubernetes.io/projected/df136678-b446-4fe6-8705-48d36455f70a-kube-api-access-2dzxb\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:12.393542 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.393384    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df136678-b446-4fe6-8705-48d36455f70a-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:12.591481 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.591451    4930 generic.go:356] "Generic (PLEG): container finished" podID="df136678-b446-4fe6-8705-48d36455f70a" containerID="60d8d43549422f0cf244a102fd44e6d5b3f36f78d8dde4d60b828c9761abc4d2" exitCode=0
Nov 01 03:52:12.591960 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.591507    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"
Nov 01 03:52:12.591960 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.591541    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq" event={"ID":"df136678-b446-4fe6-8705-48d36455f70a","Type":"ContainerDied","Data":"60d8d43549422f0cf244a102fd44e6d5b3f36f78d8dde4d60b828c9761abc4d2"}
Nov 01 03:52:12.591960 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.591616    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq" event={"ID":"df136678-b446-4fe6-8705-48d36455f70a","Type":"ContainerDied","Data":"4392d0856b91ea02967f7cdb2da919bc2688a3c05538756e22e02f86b9424ecf"}
Nov 01 03:52:12.591960 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.591649    4930 scope.go:122] "RemoveContainer" containerID="60d8d43549422f0cf244a102fd44e6d5b3f36f78d8dde4d60b828c9761abc4d2"
Nov 01 03:52:12.603955 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.603936    4930 scope.go:122] "RemoveContainer" containerID="3e880d4951c987dca9006bff08e898da20ab4f7766a79e22143fe0e1331e9341"
Nov 01 03:52:12.616444 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.616425    4930 scope.go:122] "RemoveContainer" containerID="60d8d43549422f0cf244a102fd44e6d5b3f36f78d8dde4d60b828c9761abc4d2"
Nov 01 03:52:12.616786 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:52:12.616759    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60d8d43549422f0cf244a102fd44e6d5b3f36f78d8dde4d60b828c9761abc4d2\": container with ID starting with 60d8d43549422f0cf244a102fd44e6d5b3f36f78d8dde4d60b828c9761abc4d2 not found: ID does not exist" containerID="60d8d43549422f0cf244a102fd44e6d5b3f36f78d8dde4d60b828c9761abc4d2"
Nov 01 03:52:12.616901 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.616797    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60d8d43549422f0cf244a102fd44e6d5b3f36f78d8dde4d60b828c9761abc4d2"} err="failed to get container status \"60d8d43549422f0cf244a102fd44e6d5b3f36f78d8dde4d60b828c9761abc4d2\": rpc error: code = NotFound desc = could not find container \"60d8d43549422f0cf244a102fd44e6d5b3f36f78d8dde4d60b828c9761abc4d2\": container with ID starting with 60d8d43549422f0cf244a102fd44e6d5b3f36f78d8dde4d60b828c9761abc4d2 not found: ID does not exist"
Nov 01 03:52:12.616901 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.616824    4930 scope.go:122] "RemoveContainer" containerID="3e880d4951c987dca9006bff08e898da20ab4f7766a79e22143fe0e1331e9341"
Nov 01 03:52:12.617165 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:52:12.617136    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e880d4951c987dca9006bff08e898da20ab4f7766a79e22143fe0e1331e9341\": container with ID starting with 3e880d4951c987dca9006bff08e898da20ab4f7766a79e22143fe0e1331e9341 not found: ID does not exist" containerID="3e880d4951c987dca9006bff08e898da20ab4f7766a79e22143fe0e1331e9341"
Nov 01 03:52:12.617221 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.617173    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e880d4951c987dca9006bff08e898da20ab4f7766a79e22143fe0e1331e9341"} err="failed to get container status \"3e880d4951c987dca9006bff08e898da20ab4f7766a79e22143fe0e1331e9341\": rpc error: code = NotFound desc = could not find container \"3e880d4951c987dca9006bff08e898da20ab4f7766a79e22143fe0e1331e9341\": container with ID starting with 3e880d4951c987dca9006bff08e898da20ab4f7766a79e22143fe0e1331e9341 not found: ID does not exist"
Nov 01 03:52:12.751027 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.751003    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d55d5e8-lhq9q"
Nov 01 03:52:12.762600 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.761018    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"]
Nov 01 03:52:12.798568 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.798529    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b-maas-api-openshift-ca\") pod \"6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b\" (UID: \"6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b\") "
Nov 01 03:52:12.798699 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.798623    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b-kube-api-access-gvctt\" (UniqueName: \"kubernetes.io/projected/6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b-kube-api-access-gvctt\") pod \"6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b\" (UID: \"6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b\") "
Nov 01 03:52:12.798699 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.798681    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b-maas-api-xks-ca\") pod \"6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b\" (UID: \"6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b\") "
Nov 01 03:52:12.799013 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.798983    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b-maas-api-openshift-ca" pod "6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b" (UID: "6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:52:12.799258 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.799239    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:12.800807 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.800783    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b-maas-api-xks-ca" pod "6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b" (UID: "6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:52:12.800912 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.800882    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b-kube-api-access-gvctt" pod "6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b" (UID: "6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b"). InnerVolumeSpecName "kube-api-access-gvctt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:52:12.811865 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.811840    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-170ffa-openshift-default-5dc8c6695c-4fckq"]
Nov 01 03:52:12.901287 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.901202    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvctt\" (UniqueName: \"kubernetes.io/projected/6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b-kube-api-access-gvctt\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:12.901287 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:12.901243    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:13.352060 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:13.352028    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:13.526100 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:13.526063    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:13.603197 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:13.603103    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d55d5e8-lhq9q"
Nov 01 03:52:13.603734 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:13.603102    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d55d5e8-lhq9q" event={"ID":"6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b","Type":"ContainerDied","Data":"6b1e5aac67a2f3e7f2f01fe5472631606044dba8a395d5492c152c9090fd0f0f"}
Nov 01 03:52:13.603734 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:13.603249    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b1e5aac67a2f3e7f2f01fe5472631606044dba8a395d5492c152c9090fd0f0f"
Nov 01 03:52:13.619900 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:13.619873    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d91bba7f-304b-424e-a819-d78a9275f6be" path="/var/lib/kubelet/pods/d91bba7f-304b-424e-a819-d78a9275f6be/volumes"
Nov 01 03:52:13.620344 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:13.620330    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df136678-b446-4fe6-8705-48d36455f70a" path="/var/lib/kubelet/pods/df136678-b446-4fe6-8705-48d36455f70a/volumes"
Nov 01 03:52:15.039885 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:15.039844    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:17.074042 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:17.073956    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22e11b-5ff5d6f995-fkzzs" podUID="03b489d4-3a39-4830-8629-d2a41315182d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 03:52:19.122199 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:52:19.122159    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:52:19.359310 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:19.359274    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-7cd000-54689dbf75-74hbs"]
Nov 01 03:52:19.613386 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:19.613353    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:19.803765 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:19.803728    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:19.951747 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:19.951664    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:20.945544 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:20.945504    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7" podUID="1ce834cc-d34d-46cc-a081-4d2cc261becb" containerName="main" containerID="cri-o://d0936fbc459faadef3f261b6ac7a19503cf768b4d884dcf5a86bdb6cbd4e3c20" gracePeriod=45
Nov 01 03:52:20.947244 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:20.947215    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8" podUID="88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" containerName="main" containerID="cri-o://a07531139b08b831be771d6d57d9790db21302d081233cf78f49e3d56c68a0b7" gracePeriod=45
Nov 01 03:52:20.994801 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:20.994771    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n" podUID="d2511772-66f9-40a0-9166-87c4345485c4" containerName="main" containerID="cri-o://b972c0b7cd229c628519bb99f3aae0c1c83a5158577036c8ce6bb4401512fbff" gracePeriod=45
Nov 01 03:52:21.273989 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.273963    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"
Nov 01 03:52:21.281249 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.281225    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:52:21.310796 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.310773    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"
Nov 01 03:52:21.402832 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.402802    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ce834cc-d34d-46cc-a081-4d2cc261becb-kube-api-access-9wz65\" (UniqueName: \"kubernetes.io/projected/1ce834cc-d34d-46cc-a081-4d2cc261becb-kube-api-access-9wz65\") pod \"1ce834cc-d34d-46cc-a081-4d2cc261becb\" (UID: \"1ce834cc-d34d-46cc-a081-4d2cc261becb\") "
Nov 01 03:52:21.403035 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.402871    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88a5c7e9-2bd5-4a18-87ad-44feac2eb94e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88a5c7e9-2bd5-4a18-87ad-44feac2eb94e-model-cache\") pod \"88a5c7e9-2bd5-4a18-87ad-44feac2eb94e\" (UID: \"88a5c7e9-2bd5-4a18-87ad-44feac2eb94e\") "
Nov 01 03:52:21.403035 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.402910    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce834cc-d34d-46cc-a081-4d2cc261becb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ce834cc-d34d-46cc-a081-4d2cc261becb-tmp-dir\") pod \"1ce834cc-d34d-46cc-a081-4d2cc261becb\" (UID: \"1ce834cc-d34d-46cc-a081-4d2cc261becb\") "
Nov 01 03:52:21.403035 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.402966    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ce834cc-d34d-46cc-a081-4d2cc261becb-tls-certs\" (UniqueName: \"kubernetes.io/secret/1ce834cc-d34d-46cc-a081-4d2cc261becb-tls-certs\") pod \"1ce834cc-d34d-46cc-a081-4d2cc261becb\" (UID: \"1ce834cc-d34d-46cc-a081-4d2cc261becb\") "
Nov 01 03:52:21.403035 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.403001    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88a5c7e9-2bd5-4a18-87ad-44feac2eb94e-tls-certs\" (UniqueName: \"kubernetes.io/secret/88a5c7e9-2bd5-4a18-87ad-44feac2eb94e-tls-certs\") pod \"88a5c7e9-2bd5-4a18-87ad-44feac2eb94e\" (UID: \"88a5c7e9-2bd5-4a18-87ad-44feac2eb94e\") "
Nov 01 03:52:21.403268 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.403063    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce834cc-d34d-46cc-a081-4d2cc261becb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ce834cc-d34d-46cc-a081-4d2cc261becb-dshm\") pod \"1ce834cc-d34d-46cc-a081-4d2cc261becb\" (UID: \"1ce834cc-d34d-46cc-a081-4d2cc261becb\") "
Nov 01 03:52:21.403268 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.403099    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2511772-66f9-40a0-9166-87c4345485c4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2511772-66f9-40a0-9166-87c4345485c4-dshm\") pod \"d2511772-66f9-40a0-9166-87c4345485c4\" (UID: \"d2511772-66f9-40a0-9166-87c4345485c4\") "
Nov 01 03:52:21.403268 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.403128    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88a5c7e9-2bd5-4a18-87ad-44feac2eb94e-home\" (UniqueName: \"kubernetes.io/empty-dir/88a5c7e9-2bd5-4a18-87ad-44feac2eb94e-home\") pod \"88a5c7e9-2bd5-4a18-87ad-44feac2eb94e\" (UID: \"88a5c7e9-2bd5-4a18-87ad-44feac2eb94e\") "
Nov 01 03:52:21.403268 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.403163    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a5c7e9-2bd5-4a18-87ad-44feac2eb94e-model-cache" pod "88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" (UID: "88a5c7e9-2bd5-4a18-87ad-44feac2eb94e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:52:21.403268 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.403172    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2511772-66f9-40a0-9166-87c4345485c4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2511772-66f9-40a0-9166-87c4345485c4-model-cache\") pod \"d2511772-66f9-40a0-9166-87c4345485c4\" (UID: \"d2511772-66f9-40a0-9166-87c4345485c4\") "
Nov 01 03:52:21.404833 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.403257    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce834cc-d34d-46cc-a081-4d2cc261becb-home\" (UniqueName: \"kubernetes.io/empty-dir/1ce834cc-d34d-46cc-a081-4d2cc261becb-home\") pod \"1ce834cc-d34d-46cc-a081-4d2cc261becb\" (UID: \"1ce834cc-d34d-46cc-a081-4d2cc261becb\") "
Nov 01 03:52:21.404833 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.403390    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88a5c7e9-2bd5-4a18-87ad-44feac2eb94e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/88a5c7e9-2bd5-4a18-87ad-44feac2eb94e-dshm\") pod \"88a5c7e9-2bd5-4a18-87ad-44feac2eb94e\" (UID: \"88a5c7e9-2bd5-4a18-87ad-44feac2eb94e\") "
Nov 01 03:52:21.404833 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.403404    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2511772-66f9-40a0-9166-87c4345485c4-model-cache" pod "d2511772-66f9-40a0-9166-87c4345485c4" (UID: "d2511772-66f9-40a0-9166-87c4345485c4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:52:21.404833 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.403444    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2511772-66f9-40a0-9166-87c4345485c4-tls-certs\" (UniqueName: \"kubernetes.io/secret/d2511772-66f9-40a0-9166-87c4345485c4-tls-certs\") pod \"d2511772-66f9-40a0-9166-87c4345485c4\" (UID: \"d2511772-66f9-40a0-9166-87c4345485c4\") "
Nov 01 03:52:21.404833 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.403493    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce834cc-d34d-46cc-a081-4d2cc261becb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ce834cc-d34d-46cc-a081-4d2cc261becb-model-cache\") pod \"1ce834cc-d34d-46cc-a081-4d2cc261becb\" (UID: \"1ce834cc-d34d-46cc-a081-4d2cc261becb\") "
Nov 01 03:52:21.404833 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.403600    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88a5c7e9-2bd5-4a18-87ad-44feac2eb94e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88a5c7e9-2bd5-4a18-87ad-44feac2eb94e-tmp-dir\") pod \"88a5c7e9-2bd5-4a18-87ad-44feac2eb94e\" (UID: \"88a5c7e9-2bd5-4a18-87ad-44feac2eb94e\") "
Nov 01 03:52:21.404833 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.403641    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2511772-66f9-40a0-9166-87c4345485c4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2511772-66f9-40a0-9166-87c4345485c4-tmp-dir\") pod \"d2511772-66f9-40a0-9166-87c4345485c4\" (UID: \"d2511772-66f9-40a0-9166-87c4345485c4\") "
Nov 01 03:52:21.404833 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.403771    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a5c7e9-2bd5-4a18-87ad-44feac2eb94e-home" pod "88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" (UID: "88a5c7e9-2bd5-4a18-87ad-44feac2eb94e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:52:21.404833 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.403773    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2511772-66f9-40a0-9166-87c4345485c4-kube-api-access-d7prf\" (UniqueName: \"kubernetes.io/projected/d2511772-66f9-40a0-9166-87c4345485c4-kube-api-access-d7prf\") pod \"d2511772-66f9-40a0-9166-87c4345485c4\" (UID: \"d2511772-66f9-40a0-9166-87c4345485c4\") "
Nov 01 03:52:21.404833 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.403869    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2511772-66f9-40a0-9166-87c4345485c4-home\" (UniqueName: \"kubernetes.io/empty-dir/d2511772-66f9-40a0-9166-87c4345485c4-home\") pod \"d2511772-66f9-40a0-9166-87c4345485c4\" (UID: \"d2511772-66f9-40a0-9166-87c4345485c4\") "
Nov 01 03:52:21.404833 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.403909    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88a5c7e9-2bd5-4a18-87ad-44feac2eb94e-kube-api-access-prt4c\" (UniqueName: \"kubernetes.io/projected/88a5c7e9-2bd5-4a18-87ad-44feac2eb94e-kube-api-access-prt4c\") pod \"88a5c7e9-2bd5-4a18-87ad-44feac2eb94e\" (UID: \"88a5c7e9-2bd5-4a18-87ad-44feac2eb94e\") "
Nov 01 03:52:21.404833 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.404422    4930 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88a5c7e9-2bd5-4a18-87ad-44feac2eb94e-model-cache\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:21.404833 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.404440    4930 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88a5c7e9-2bd5-4a18-87ad-44feac2eb94e-home\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:21.404833 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.404454    4930 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2511772-66f9-40a0-9166-87c4345485c4-model-cache\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:21.405675 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.405060    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a5c7e9-2bd5-4a18-87ad-44feac2eb94e-tmp-dir" pod "88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" (UID: "88a5c7e9-2bd5-4a18-87ad-44feac2eb94e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:52:21.405675 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.405083    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2511772-66f9-40a0-9166-87c4345485c4-tmp-dir" pod "d2511772-66f9-40a0-9166-87c4345485c4" (UID: "d2511772-66f9-40a0-9166-87c4345485c4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:52:21.405675 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.405169    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce834cc-d34d-46cc-a081-4d2cc261becb-tmp-dir" pod "1ce834cc-d34d-46cc-a081-4d2cc261becb" (UID: "1ce834cc-d34d-46cc-a081-4d2cc261becb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:52:21.405675 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.405253    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce834cc-d34d-46cc-a081-4d2cc261becb-model-cache" pod "1ce834cc-d34d-46cc-a081-4d2cc261becb" (UID: "1ce834cc-d34d-46cc-a081-4d2cc261becb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:52:21.405675 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.405583    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce834cc-d34d-46cc-a081-4d2cc261becb-home" pod "1ce834cc-d34d-46cc-a081-4d2cc261becb" (UID: "1ce834cc-d34d-46cc-a081-4d2cc261becb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:52:21.405948 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.405835    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2511772-66f9-40a0-9166-87c4345485c4-home" pod "d2511772-66f9-40a0-9166-87c4345485c4" (UID: "d2511772-66f9-40a0-9166-87c4345485c4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:52:21.408040 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.408012    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2511772-66f9-40a0-9166-87c4345485c4-dshm" pod "d2511772-66f9-40a0-9166-87c4345485c4" (UID: "d2511772-66f9-40a0-9166-87c4345485c4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:52:21.408156 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.408013    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88a5c7e9-2bd5-4a18-87ad-44feac2eb94e-kube-api-access-prt4c" pod "88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" (UID: "88a5c7e9-2bd5-4a18-87ad-44feac2eb94e"). InnerVolumeSpecName "kube-api-access-prt4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:52:21.408214 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.408154    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2511772-66f9-40a0-9166-87c4345485c4-kube-api-access-d7prf" pod "d2511772-66f9-40a0-9166-87c4345485c4" (UID: "d2511772-66f9-40a0-9166-87c4345485c4"). InnerVolumeSpecName "kube-api-access-d7prf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:52:21.408214 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.408182    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88a5c7e9-2bd5-4a18-87ad-44feac2eb94e-tls-certs" pod "88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" (UID: "88a5c7e9-2bd5-4a18-87ad-44feac2eb94e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:52:21.408420 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.408398    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2511772-66f9-40a0-9166-87c4345485c4-tls-certs" pod "d2511772-66f9-40a0-9166-87c4345485c4" (UID: "d2511772-66f9-40a0-9166-87c4345485c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:52:21.408585 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.408539    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce834cc-d34d-46cc-a081-4d2cc261becb-dshm" pod "1ce834cc-d34d-46cc-a081-4d2cc261becb" (UID: "1ce834cc-d34d-46cc-a081-4d2cc261becb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:52:21.408785 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.408769    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a5c7e9-2bd5-4a18-87ad-44feac2eb94e-dshm" pod "88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" (UID: "88a5c7e9-2bd5-4a18-87ad-44feac2eb94e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:52:21.409258 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.409234    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ce834cc-d34d-46cc-a081-4d2cc261becb-kube-api-access-9wz65" pod "1ce834cc-d34d-46cc-a081-4d2cc261becb" (UID: "1ce834cc-d34d-46cc-a081-4d2cc261becb"). InnerVolumeSpecName "kube-api-access-9wz65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:52:21.409823 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.409804    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ce834cc-d34d-46cc-a081-4d2cc261becb-tls-certs" pod "1ce834cc-d34d-46cc-a081-4d2cc261becb" (UID: "1ce834cc-d34d-46cc-a081-4d2cc261becb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:52:21.505751 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.505666    4930 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ce834cc-d34d-46cc-a081-4d2cc261becb-dshm\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:21.505751 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.505699    4930 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2511772-66f9-40a0-9166-87c4345485c4-dshm\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:21.505751 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.505712    4930 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ce834cc-d34d-46cc-a081-4d2cc261becb-home\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:21.505751 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.505722    4930 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88a5c7e9-2bd5-4a18-87ad-44feac2eb94e-dshm\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:21.505751 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.505731    4930 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2511772-66f9-40a0-9166-87c4345485c4-tls-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:21.505751 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.505738    4930 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ce834cc-d34d-46cc-a081-4d2cc261becb-model-cache\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:21.505751 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.505747    4930 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88a5c7e9-2bd5-4a18-87ad-44feac2eb94e-tmp-dir\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:21.505751 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.505756    4930 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2511772-66f9-40a0-9166-87c4345485c4-tmp-dir\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:21.506306 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.505765    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7prf\" (UniqueName: \"kubernetes.io/projected/d2511772-66f9-40a0-9166-87c4345485c4-kube-api-access-d7prf\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:21.506306 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.505779    4930 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2511772-66f9-40a0-9166-87c4345485c4-home\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:21.506306 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.505790    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prt4c\" (UniqueName: \"kubernetes.io/projected/88a5c7e9-2bd5-4a18-87ad-44feac2eb94e-kube-api-access-prt4c\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:21.506306 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.505800    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wz65\" (UniqueName: \"kubernetes.io/projected/1ce834cc-d34d-46cc-a081-4d2cc261becb-kube-api-access-9wz65\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:21.506306 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.505809    4930 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ce834cc-d34d-46cc-a081-4d2cc261becb-tmp-dir\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:21.506306 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.505816    4930 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ce834cc-d34d-46cc-a081-4d2cc261becb-tls-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:21.506306 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.505824    4930 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88a5c7e9-2bd5-4a18-87ad-44feac2eb94e-tls-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:21.543897 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.543851    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm" podUID="b2637bc6-3828-4b75-9783-5415519403a0" containerName="main" containerID="cri-o://3b36a5bfcdc6efcc3187773c157d458b552ca5df74ff765a661069033055a64f" gracePeriod=45
Nov 01 03:52:21.632830 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.632790    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm" podUID="b2637bc6-3828-4b75-9783-5415519403a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": dial tcp 10.133.0.68:8000: connect: connection refused"
Nov 01 03:52:21.668835 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.668801    4930 generic.go:356] "Generic (PLEG): container finished" podID="b2637bc6-3828-4b75-9783-5415519403a0" containerID="3b36a5bfcdc6efcc3187773c157d458b552ca5df74ff765a661069033055a64f" exitCode=0
Nov 01 03:52:21.670858 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.670830    4930 generic.go:356] "Generic (PLEG): container finished" podID="d2511772-66f9-40a0-9166-87c4345485c4" containerID="b972c0b7cd229c628519bb99f3aae0c1c83a5158577036c8ce6bb4401512fbff" exitCode=0
Nov 01 03:52:21.671182 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.671032    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"
Nov 01 03:52:21.671281 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.671074    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm" event={"ID":"b2637bc6-3828-4b75-9783-5415519403a0","Type":"ContainerDied","Data":"3b36a5bfcdc6efcc3187773c157d458b552ca5df74ff765a661069033055a64f"}
Nov 01 03:52:21.671333 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.671291    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n" event={"ID":"d2511772-66f9-40a0-9166-87c4345485c4","Type":"ContainerDied","Data":"b972c0b7cd229c628519bb99f3aae0c1c83a5158577036c8ce6bb4401512fbff"}
Nov 01 03:52:21.671382 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.671315    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n" event={"ID":"d2511772-66f9-40a0-9166-87c4345485c4","Type":"ContainerDied","Data":"f26e9cd1cce9ce8b6ef92eb6b9afa19cee0d25ab41b85732827b6b5bfe8566a8"}
Nov 01 03:52:21.671422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.671380    4930 scope.go:122] "RemoveContainer" containerID="b972c0b7cd229c628519bb99f3aae0c1c83a5158577036c8ce6bb4401512fbff"
Nov 01 03:52:21.673243 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.673215    4930 generic.go:356] "Generic (PLEG): container finished" podID="88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" containerID="a07531139b08b831be771d6d57d9790db21302d081233cf78f49e3d56c68a0b7" exitCode=0
Nov 01 03:52:21.673334 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.673293    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8" event={"ID":"88a5c7e9-2bd5-4a18-87ad-44feac2eb94e","Type":"ContainerDied","Data":"a07531139b08b831be771d6d57d9790db21302d081233cf78f49e3d56c68a0b7"}
Nov 01 03:52:21.673334 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.673323    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8" event={"ID":"88a5c7e9-2bd5-4a18-87ad-44feac2eb94e","Type":"ContainerDied","Data":"08e1c3e654e3a874326829863a525666d1f95d01b88f6c785f74cb282112f63f"}
Nov 01 03:52:21.673467 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.673391    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"
Nov 01 03:52:21.675861 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.675841    4930 generic.go:356] "Generic (PLEG): container finished" podID="1ce834cc-d34d-46cc-a081-4d2cc261becb" containerID="d0936fbc459faadef3f261b6ac7a19503cf768b4d884dcf5a86bdb6cbd4e3c20" exitCode=0
Nov 01 03:52:21.675985 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.675918    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7" event={"ID":"1ce834cc-d34d-46cc-a081-4d2cc261becb","Type":"ContainerDied","Data":"d0936fbc459faadef3f261b6ac7a19503cf768b4d884dcf5a86bdb6cbd4e3c20"}
Nov 01 03:52:21.675985 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.675954    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7" event={"ID":"1ce834cc-d34d-46cc-a081-4d2cc261becb","Type":"ContainerDied","Data":"c7294e5ec3b16e452c74a01d4da0ac9e6d5514d1952036997130d71067901e2b"}
Nov 01 03:52:21.676106 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.675991    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"
Nov 01 03:52:21.685456 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.685431    4930 scope.go:122] "RemoveContainer" containerID="b972c0b7cd229c628519bb99f3aae0c1c83a5158577036c8ce6bb4401512fbff"
Nov 01 03:52:21.685873 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:52:21.685848    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b972c0b7cd229c628519bb99f3aae0c1c83a5158577036c8ce6bb4401512fbff\": container with ID starting with b972c0b7cd229c628519bb99f3aae0c1c83a5158577036c8ce6bb4401512fbff not found: ID does not exist" containerID="b972c0b7cd229c628519bb99f3aae0c1c83a5158577036c8ce6bb4401512fbff"
Nov 01 03:52:21.685960 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.685885    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b972c0b7cd229c628519bb99f3aae0c1c83a5158577036c8ce6bb4401512fbff"} err="failed to get container status \"b972c0b7cd229c628519bb99f3aae0c1c83a5158577036c8ce6bb4401512fbff\": rpc error: code = NotFound desc = could not find container \"b972c0b7cd229c628519bb99f3aae0c1c83a5158577036c8ce6bb4401512fbff\": container with ID starting with b972c0b7cd229c628519bb99f3aae0c1c83a5158577036c8ce6bb4401512fbff not found: ID does not exist"
Nov 01 03:52:21.685960 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.685909    4930 scope.go:122] "RemoveContainer" containerID="a07531139b08b831be771d6d57d9790db21302d081233cf78f49e3d56c68a0b7"
Nov 01 03:52:21.698801 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.698780    4930 scope.go:122] "RemoveContainer" containerID="a07531139b08b831be771d6d57d9790db21302d081233cf78f49e3d56c68a0b7"
Nov 01 03:52:21.699182 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:52:21.699154    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a07531139b08b831be771d6d57d9790db21302d081233cf78f49e3d56c68a0b7\": container with ID starting with a07531139b08b831be771d6d57d9790db21302d081233cf78f49e3d56c68a0b7 not found: ID does not exist" containerID="a07531139b08b831be771d6d57d9790db21302d081233cf78f49e3d56c68a0b7"
Nov 01 03:52:21.699245 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.699195    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a07531139b08b831be771d6d57d9790db21302d081233cf78f49e3d56c68a0b7"} err="failed to get container status \"a07531139b08b831be771d6d57d9790db21302d081233cf78f49e3d56c68a0b7\": rpc error: code = NotFound desc = could not find container \"a07531139b08b831be771d6d57d9790db21302d081233cf78f49e3d56c68a0b7\": container with ID starting with a07531139b08b831be771d6d57d9790db21302d081233cf78f49e3d56c68a0b7 not found: ID does not exist"
Nov 01 03:52:21.699245 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.699223    4930 scope.go:122] "RemoveContainer" containerID="d0936fbc459faadef3f261b6ac7a19503cf768b4d884dcf5a86bdb6cbd4e3c20"
Nov 01 03:52:21.711504 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.711480    4930 scope.go:122] "RemoveContainer" containerID="d0936fbc459faadef3f261b6ac7a19503cf768b4d884dcf5a86bdb6cbd4e3c20"
Nov 01 03:52:21.711837 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:52:21.711809    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0936fbc459faadef3f261b6ac7a19503cf768b4d884dcf5a86bdb6cbd4e3c20\": container with ID starting with d0936fbc459faadef3f261b6ac7a19503cf768b4d884dcf5a86bdb6cbd4e3c20 not found: ID does not exist" containerID="d0936fbc459faadef3f261b6ac7a19503cf768b4d884dcf5a86bdb6cbd4e3c20"
Nov 01 03:52:21.711936 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.711875    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0936fbc459faadef3f261b6ac7a19503cf768b4d884dcf5a86bdb6cbd4e3c20"} err="failed to get container status \"d0936fbc459faadef3f261b6ac7a19503cf768b4d884dcf5a86bdb6cbd4e3c20\": rpc error: code = NotFound desc = could not find container \"d0936fbc459faadef3f261b6ac7a19503cf768b4d884dcf5a86bdb6cbd4e3c20\": container with ID starting with d0936fbc459faadef3f261b6ac7a19503cf768b4d884dcf5a86bdb6cbd4e3c20 not found: ID does not exist"
Nov 01 03:52:21.819855 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.819827    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"
Nov 01 03:52:21.838228 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.838199    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"]
Nov 01 03:52:21.843402 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.843372    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ded2d7/simulator-w0-ded2d7-kserve-75bd769b95-zsr4n"]
Nov 01 03:52:21.909712 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.909684    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2637bc6-3828-4b75-9783-5415519403a0-tls-certs\" (UniqueName: \"kubernetes.io/secret/b2637bc6-3828-4b75-9783-5415519403a0-tls-certs\") pod \"b2637bc6-3828-4b75-9783-5415519403a0\" (UID: \"b2637bc6-3828-4b75-9783-5415519403a0\") "
Nov 01 03:52:21.909844 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.909786    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2637bc6-3828-4b75-9783-5415519403a0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2637bc6-3828-4b75-9783-5415519403a0-dshm\") pod \"b2637bc6-3828-4b75-9783-5415519403a0\" (UID: \"b2637bc6-3828-4b75-9783-5415519403a0\") "
Nov 01 03:52:21.909844 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.909832    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2637bc6-3828-4b75-9783-5415519403a0-kube-api-access-hkpwm\" (UniqueName: \"kubernetes.io/projected/b2637bc6-3828-4b75-9783-5415519403a0-kube-api-access-hkpwm\") pod \"b2637bc6-3828-4b75-9783-5415519403a0\" (UID: \"b2637bc6-3828-4b75-9783-5415519403a0\") "
Nov 01 03:52:21.909922 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.909851    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2637bc6-3828-4b75-9783-5415519403a0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2637bc6-3828-4b75-9783-5415519403a0-tmp-dir\") pod \"b2637bc6-3828-4b75-9783-5415519403a0\" (UID: \"b2637bc6-3828-4b75-9783-5415519403a0\") "
Nov 01 03:52:21.909922 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.909886    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2637bc6-3828-4b75-9783-5415519403a0-home\" (UniqueName: \"kubernetes.io/empty-dir/b2637bc6-3828-4b75-9783-5415519403a0-home\") pod \"b2637bc6-3828-4b75-9783-5415519403a0\" (UID: \"b2637bc6-3828-4b75-9783-5415519403a0\") "
Nov 01 03:52:21.910010 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.909962    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2637bc6-3828-4b75-9783-5415519403a0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2637bc6-3828-4b75-9783-5415519403a0-model-cache\") pod \"b2637bc6-3828-4b75-9783-5415519403a0\" (UID: \"b2637bc6-3828-4b75-9783-5415519403a0\") "
Nov 01 03:52:21.910231 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.910166    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2637bc6-3828-4b75-9783-5415519403a0-home" pod "b2637bc6-3828-4b75-9783-5415519403a0" (UID: "b2637bc6-3828-4b75-9783-5415519403a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:52:21.910231 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.910186    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2637bc6-3828-4b75-9783-5415519403a0-tmp-dir" pod "b2637bc6-3828-4b75-9783-5415519403a0" (UID: "b2637bc6-3828-4b75-9783-5415519403a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:52:21.910368 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.910243    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2637bc6-3828-4b75-9783-5415519403a0-model-cache" pod "b2637bc6-3828-4b75-9783-5415519403a0" (UID: "b2637bc6-3828-4b75-9783-5415519403a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:52:21.910498 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.910478    4930 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2637bc6-3828-4b75-9783-5415519403a0-model-cache\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:21.910592 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.910504    4930 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2637bc6-3828-4b75-9783-5415519403a0-tmp-dir\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:21.910592 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.910518    4930 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2637bc6-3828-4b75-9783-5415519403a0-home\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:21.911906 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.911882    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2637bc6-3828-4b75-9783-5415519403a0-dshm" pod "b2637bc6-3828-4b75-9783-5415519403a0" (UID: "b2637bc6-3828-4b75-9783-5415519403a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:52:21.912015 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.911999    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2637bc6-3828-4b75-9783-5415519403a0-tls-certs" pod "b2637bc6-3828-4b75-9783-5415519403a0" (UID: "b2637bc6-3828-4b75-9783-5415519403a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:52:21.912055 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.912025    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2637bc6-3828-4b75-9783-5415519403a0-kube-api-access-hkpwm" pod "b2637bc6-3828-4b75-9783-5415519403a0" (UID: "b2637bc6-3828-4b75-9783-5415519403a0"). InnerVolumeSpecName "kube-api-access-hkpwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:52:21.952135 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.952111    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"]
Nov 01 03:52:21.958271 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:21.958238    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ded2d7/distinct-w0-ded2d7-kserve-7d5fdbbf5c-bbbj8"]
Nov 01 03:52:22.011908 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:22.011885    4930 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2637bc6-3828-4b75-9783-5415519403a0-tls-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:22.011908 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:22.011909    4930 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2637bc6-3828-4b75-9783-5415519403a0-dshm\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:22.012071 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:22.011921    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkpwm\" (UniqueName: \"kubernetes.io/projected/b2637bc6-3828-4b75-9783-5415519403a0-kube-api-access-hkpwm\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:22.067774 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:22.067705    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"]
Nov 01 03:52:22.073946 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:22.073923    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ded2d7/distinct-2-w0-ded2d7-kserve-744b9477f8-jf8j7"]
Nov 01 03:52:22.687903 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:22.687871    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"
Nov 01 03:52:22.687903 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:22.687892    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm" event={"ID":"b2637bc6-3828-4b75-9783-5415519403a0","Type":"ContainerDied","Data":"9bbd30064d5b911d6e3721e232f06f325a8c0295bc81419808a48be71c4cf616"}
Nov 01 03:52:22.688190 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:22.687941    4930 scope.go:122] "RemoveContainer" containerID="3b36a5bfcdc6efcc3187773c157d458b552ca5df74ff765a661069033055a64f"
Nov 01 03:52:22.854061 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:22.854026    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"]
Nov 01 03:52:22.858596 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:22.858571    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ded2d7/premium-w0-ded2d7-kserve-6fb86b9cc8-9qjlm"]
Nov 01 03:52:23.620172 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:23.620141    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ce834cc-d34d-46cc-a081-4d2cc261becb" path="/var/lib/kubelet/pods/1ce834cc-d34d-46cc-a081-4d2cc261becb/volumes"
Nov 01 03:52:23.620631 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:23.620617    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" path="/var/lib/kubelet/pods/88a5c7e9-2bd5-4a18-87ad-44feac2eb94e/volumes"
Nov 01 03:52:23.621029 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:23.621016    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2637bc6-3828-4b75-9783-5415519403a0" path="/var/lib/kubelet/pods/b2637bc6-3828-4b75-9783-5415519403a0/volumes"
Nov 01 03:52:23.621724 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:23.621709    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2511772-66f9-40a0-9166-87c4345485c4" path="/var/lib/kubelet/pods/d2511772-66f9-40a0-9166-87c4345485c4/volumes"
Nov 01 03:52:24.213317 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:24.213280    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:24.359783 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:24.359739    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-7cd000-54689dbf75-74hbs" podUID="15dca7b4-1639-406e-a15e-67f7e85ba636" containerName="payload-pre-processing" containerID="cri-o://2cd937b8816e6ca3524cc7493d425a1854a70b2134dcb0b99ad992df9e243cd6" gracePeriod=25
Nov 01 03:52:24.695226 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:24.695196    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-7cd000-54689dbf75-74hbs"
Nov 01 03:52:24.718276 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:24.718248    4930 generic.go:356] "Generic (PLEG): container finished" podID="15dca7b4-1639-406e-a15e-67f7e85ba636" containerID="2cd937b8816e6ca3524cc7493d425a1854a70b2134dcb0b99ad992df9e243cd6" exitCode=0
Nov 01 03:52:24.718422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:24.718355    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-7cd000-54689dbf75-74hbs" event={"ID":"15dca7b4-1639-406e-a15e-67f7e85ba636","Type":"ContainerDied","Data":"2cd937b8816e6ca3524cc7493d425a1854a70b2134dcb0b99ad992df9e243cd6"}
Nov 01 03:52:24.718422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:24.718364    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-7cd000-54689dbf75-74hbs"
Nov 01 03:52:24.718422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:24.718397    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-7cd000-54689dbf75-74hbs" event={"ID":"15dca7b4-1639-406e-a15e-67f7e85ba636","Type":"ContainerDied","Data":"1cea0fee007568acaf3ba281858a4c9e66bd1dd004b31246fc9d5fd534c54694"}
Nov 01 03:52:24.718422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:24.718418    4930 scope.go:122] "RemoveContainer" containerID="2cd937b8816e6ca3524cc7493d425a1854a70b2134dcb0b99ad992df9e243cd6"
Nov 01 03:52:24.729793 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:24.729774    4930 scope.go:122] "RemoveContainer" containerID="2cd937b8816e6ca3524cc7493d425a1854a70b2134dcb0b99ad992df9e243cd6"
Nov 01 03:52:24.730061 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:52:24.730044    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cd937b8816e6ca3524cc7493d425a1854a70b2134dcb0b99ad992df9e243cd6\": container with ID starting with 2cd937b8816e6ca3524cc7493d425a1854a70b2134dcb0b99ad992df9e243cd6 not found: ID does not exist" containerID="2cd937b8816e6ca3524cc7493d425a1854a70b2134dcb0b99ad992df9e243cd6"
Nov 01 03:52:24.730128 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:24.730078    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cd937b8816e6ca3524cc7493d425a1854a70b2134dcb0b99ad992df9e243cd6"} err="failed to get container status \"2cd937b8816e6ca3524cc7493d425a1854a70b2134dcb0b99ad992df9e243cd6\": rpc error: code = NotFound desc = could not find container \"2cd937b8816e6ca3524cc7493d425a1854a70b2134dcb0b99ad992df9e243cd6\": container with ID starting with 2cd937b8816e6ca3524cc7493d425a1854a70b2134dcb0b99ad992df9e243cd6 not found: ID does not exist"
Nov 01 03:52:24.849406 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:24.849378    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15dca7b4-1639-406e-a15e-67f7e85ba636-kube-api-access-fvnkb\" (UniqueName: \"kubernetes.io/projected/15dca7b4-1639-406e-a15e-67f7e85ba636-kube-api-access-fvnkb\") pod \"15dca7b4-1639-406e-a15e-67f7e85ba636\" (UID: \"15dca7b4-1639-406e-a15e-67f7e85ba636\") "
Nov 01 03:52:24.849592 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:24.849455    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15dca7b4-1639-406e-a15e-67f7e85ba636-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15dca7b4-1639-406e-a15e-67f7e85ba636-plugins-config-volume\") pod \"15dca7b4-1639-406e-a15e-67f7e85ba636\" (UID: \"15dca7b4-1639-406e-a15e-67f7e85ba636\") "
Nov 01 03:52:24.849853 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:24.849821    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15dca7b4-1639-406e-a15e-67f7e85ba636-plugins-config-volume" pod "15dca7b4-1639-406e-a15e-67f7e85ba636" (UID: "15dca7b4-1639-406e-a15e-67f7e85ba636"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:52:24.850085 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:24.850067    4930 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15dca7b4-1639-406e-a15e-67f7e85ba636-plugins-config-volume\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:24.851408 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:24.851382    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15dca7b4-1639-406e-a15e-67f7e85ba636-kube-api-access-fvnkb" pod "15dca7b4-1639-406e-a15e-67f7e85ba636" (UID: "15dca7b4-1639-406e-a15e-67f7e85ba636"). InnerVolumeSpecName "kube-api-access-fvnkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:52:24.950888 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:24.950853    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvnkb\" (UniqueName: \"kubernetes.io/projected/15dca7b4-1639-406e-a15e-67f7e85ba636-kube-api-access-fvnkb\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:25.140005 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:25.139966    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-7cd000-54689dbf75-74hbs"]
Nov 01 03:52:25.145909 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:25.145883    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-7cd000-54689dbf75-74hbs"]
Nov 01 03:52:25.620778 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:25.620749    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15dca7b4-1639-406e-a15e-67f7e85ba636" path="/var/lib/kubelet/pods/15dca7b4-1639-406e-a15e-67f7e85ba636/volumes"
Nov 01 03:52:25.753179 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:25.753144    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:26.036407 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:26.036328    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:26.184041 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:26.184005    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:26.259070 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:26.259040    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22e11b-6j9ps"]
Nov 01 03:52:26.308813 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:26.308740    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-22e11b-6j9ps"]
Nov 01 03:52:26.711593 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:26.711549    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:27.073385 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:27.073292    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22e11b-5ff5d6f995-fkzzs" podUID="03b489d4-3a39-4830-8629-d2a41315182d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 03:52:27.620216 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:27.620181    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f4de4f9-c0dd-418c-8157-c2f25a2b15eb" path="/var/lib/kubelet/pods/2f4de4f9-c0dd-418c-8157-c2f25a2b15eb/volumes"
Nov 01 03:52:31.024898 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:31.024863    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:31.239960 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:31.239933    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22e11b-5ff5d6f995-fkzzs"
Nov 01 03:52:31.337768 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:31.337737    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03b489d4-3a39-4830-8629-d2a41315182d-kube-api-access-kdlc5\" (UniqueName: \"kubernetes.io/projected/03b489d4-3a39-4830-8629-d2a41315182d-kube-api-access-kdlc5\") pod \"03b489d4-3a39-4830-8629-d2a41315182d\" (UID: \"03b489d4-3a39-4830-8629-d2a41315182d\") "
Nov 01 03:52:31.337917 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:31.337782    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03b489d4-3a39-4830-8629-d2a41315182d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03b489d4-3a39-4830-8629-d2a41315182d-plugins-config-volume\") pod \"03b489d4-3a39-4830-8629-d2a41315182d\" (UID: \"03b489d4-3a39-4830-8629-d2a41315182d\") "
Nov 01 03:52:31.338167 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:31.338144    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03b489d4-3a39-4830-8629-d2a41315182d-plugins-config-volume" pod "03b489d4-3a39-4830-8629-d2a41315182d" (UID: "03b489d4-3a39-4830-8629-d2a41315182d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:52:31.338300 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:31.338284    4930 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03b489d4-3a39-4830-8629-d2a41315182d-plugins-config-volume\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:31.339835 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:31.339813    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03b489d4-3a39-4830-8629-d2a41315182d-kube-api-access-kdlc5" pod "03b489d4-3a39-4830-8629-d2a41315182d" (UID: "03b489d4-3a39-4830-8629-d2a41315182d"). InnerVolumeSpecName "kube-api-access-kdlc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:52:31.439528 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:31.439491    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdlc5\" (UniqueName: \"kubernetes.io/projected/03b489d4-3a39-4830-8629-d2a41315182d-kube-api-access-kdlc5\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:31.775435 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:31.775395    4930 generic.go:356] "Generic (PLEG): container finished" podID="03b489d4-3a39-4830-8629-d2a41315182d" containerID="8e475a0c4374bd50c56b34d778bf910a563369859d9df2ded6999f3cc97a2bba" exitCode=0
Nov 01 03:52:31.775643 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:31.775447    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22e11b-5ff5d6f995-fkzzs"
Nov 01 03:52:31.775643 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:31.775450    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22e11b-5ff5d6f995-fkzzs" event={"ID":"03b489d4-3a39-4830-8629-d2a41315182d","Type":"ContainerDied","Data":"8e475a0c4374bd50c56b34d778bf910a563369859d9df2ded6999f3cc97a2bba"}
Nov 01 03:52:31.775643 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:31.775511    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-22e11b-5ff5d6f995-fkzzs" event={"ID":"03b489d4-3a39-4830-8629-d2a41315182d","Type":"ContainerDied","Data":"bd00e1a4ea172f667468bf44950d35789afb8161c92442842870fe7a2e4f74d4"}
Nov 01 03:52:31.775643 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:31.775537    4930 scope.go:122] "RemoveContainer" containerID="8e475a0c4374bd50c56b34d778bf910a563369859d9df2ded6999f3cc97a2bba"
Nov 01 03:52:31.787232 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:31.787198    4930 scope.go:122] "RemoveContainer" containerID="8e475a0c4374bd50c56b34d778bf910a563369859d9df2ded6999f3cc97a2bba"
Nov 01 03:52:31.787531 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:52:31.787508    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e475a0c4374bd50c56b34d778bf910a563369859d9df2ded6999f3cc97a2bba\": container with ID starting with 8e475a0c4374bd50c56b34d778bf910a563369859d9df2ded6999f3cc97a2bba not found: ID does not exist" containerID="8e475a0c4374bd50c56b34d778bf910a563369859d9df2ded6999f3cc97a2bba"
Nov 01 03:52:31.787627 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:31.787543    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e475a0c4374bd50c56b34d778bf910a563369859d9df2ded6999f3cc97a2bba"} err="failed to get container status \"8e475a0c4374bd50c56b34d778bf910a563369859d9df2ded6999f3cc97a2bba\": rpc error: code = NotFound desc = could not find container \"8e475a0c4374bd50c56b34d778bf910a563369859d9df2ded6999f3cc97a2bba\": container with ID starting with 8e475a0c4374bd50c56b34d778bf910a563369859d9df2ded6999f3cc97a2bba not found: ID does not exist"
Nov 01 03:52:31.939987 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:31.939912    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-22e11b-5ff5d6f995-fkzzs"]
Nov 01 03:52:31.989568 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:31.989528    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-22e11b-5ff5d6f995-fkzzs"]
Nov 01 03:52:32.811792 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:32.811760    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:32.989924 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:32.989888    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:33.178523 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:33.178488    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"]
Nov 01 03:52:33.178780 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:33.178752    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x" podUID="7d9844bf-a8fe-4904-87fa-fb427c29dbd6" containerName="istio-proxy" containerID="cri-o://393f985c36b58e02c5a6e6aaebd4acbac702a308cda4c445dff49ff48a13ae19" gracePeriod=30
Nov 01 03:52:33.183144 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:33.183097    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:33.622467 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:33.622430    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03b489d4-3a39-4830-8629-d2a41315182d" path="/var/lib/kubelet/pods/03b489d4-3a39-4830-8629-d2a41315182d/volumes"
Nov 01 03:52:34.044202 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.044121    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84de6-k6wz2"]
Nov 01 03:52:34.044817 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.044795    4930 cpu_manager.go:414] "containerMap: removing container" podUID="1ce834cc-d34d-46cc-a081-4d2cc261becb" containerName="main"
Nov 01 03:52:34.044927 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.044819    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ce834cc-d34d-46cc-a081-4d2cc261becb" containerName="main"
Nov 01 03:52:34.044927 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.044883    4930 cpu_manager.go:414] "containerMap: removing container" podUID="d2511772-66f9-40a0-9166-87c4345485c4" containerName="main"
Nov 01 03:52:34.044927 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.044905    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2511772-66f9-40a0-9166-87c4345485c4" containerName="main"
Nov 01 03:52:34.044927 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.044915    4930 cpu_manager.go:414] "containerMap: removing container" podUID="6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b" containerName="revoke-keys"
Nov 01 03:52:34.044927 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.044923    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b" containerName="revoke-keys"
Nov 01 03:52:34.045211 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.044945    4930 cpu_manager.go:414] "containerMap: removing container" podUID="15dca7b4-1639-406e-a15e-67f7e85ba636" containerName="payload-pre-processing"
Nov 01 03:52:34.045211 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.044953    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15dca7b4-1639-406e-a15e-67f7e85ba636" containerName="payload-pre-processing"
Nov 01 03:52:34.045211 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.044992    4930 cpu_manager.go:414] "containerMap: removing container" podUID="03b489d4-3a39-4830-8629-d2a41315182d" containerName="payload-pre-processing"
Nov 01 03:52:34.045211 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.045000    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03b489d4-3a39-4830-8629-d2a41315182d" containerName="payload-pre-processing"
Nov 01 03:52:34.045211 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.045012    4930 cpu_manager.go:414] "containerMap: removing container" podUID="d91bba7f-304b-424e-a819-d78a9275f6be" containerName="payload-processing"
Nov 01 03:52:34.045211 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.045035    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d91bba7f-304b-424e-a819-d78a9275f6be" containerName="payload-processing"
Nov 01 03:52:34.045211 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.045049    4930 cpu_manager.go:414] "containerMap: removing container" podUID="88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" containerName="main"
Nov 01 03:52:34.045211 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.045055    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" containerName="main"
Nov 01 03:52:34.045211 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.045076    4930 cpu_manager.go:414] "containerMap: removing container" podUID="df136678-b446-4fe6-8705-48d36455f70a" containerName="istio-proxy"
Nov 01 03:52:34.045211 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.045097    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df136678-b446-4fe6-8705-48d36455f70a" containerName="istio-proxy"
Nov 01 03:52:34.045211 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.045114    4930 cpu_manager.go:414] "containerMap: removing container" podUID="b2637bc6-3828-4b75-9783-5415519403a0" containerName="main"
Nov 01 03:52:34.045211 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.045121    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2637bc6-3828-4b75-9783-5415519403a0" containerName="main"
Nov 01 03:52:34.045211 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.045132    4930 cpu_manager.go:414] "containerMap: removing container" podUID="33b901de-97ee-4363-9fac-f96499633f86" containerName="revoke-keys"
Nov 01 03:52:34.045211 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.045140    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33b901de-97ee-4363-9fac-f96499633f86" containerName="revoke-keys"
Nov 01 03:52:34.045211 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.045160    4930 cpu_manager.go:414] "containerMap: removing container" podUID="df136678-b446-4fe6-8705-48d36455f70a" containerName="istio-proxy"
Nov 01 03:52:34.045211 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.045167    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df136678-b446-4fe6-8705-48d36455f70a" containerName="istio-proxy"
Nov 01 03:52:34.046052 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.045311    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="15dca7b4-1639-406e-a15e-67f7e85ba636" containerName="payload-pre-processing"
Nov 01 03:52:34.046052 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.045325    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="03b489d4-3a39-4830-8629-d2a41315182d" containerName="payload-pre-processing"
Nov 01 03:52:34.046052 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.045337    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="d91bba7f-304b-424e-a819-d78a9275f6be" containerName="payload-processing"
Nov 01 03:52:34.046052 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.045347    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="88a5c7e9-2bd5-4a18-87ad-44feac2eb94e" containerName="main"
Nov 01 03:52:34.046052 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.045356    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="df136678-b446-4fe6-8705-48d36455f70a" containerName="istio-proxy"
Nov 01 03:52:34.046052 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.045365    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2637bc6-3828-4b75-9783-5415519403a0" containerName="main"
Nov 01 03:52:34.046052 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.045373    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="33b901de-97ee-4363-9fac-f96499633f86" containerName="revoke-keys"
Nov 01 03:52:34.046052 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.045386    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="df136678-b446-4fe6-8705-48d36455f70a" containerName="istio-proxy"
Nov 01 03:52:34.046052 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.045397    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ce834cc-d34d-46cc-a081-4d2cc261becb" containerName="main"
Nov 01 03:52:34.046052 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.045408    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2511772-66f9-40a0-9166-87c4345485c4" containerName="main"
Nov 01 03:52:34.046052 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.045417    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b" containerName="revoke-keys"
Nov 01 03:52:34.056168 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.056140    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84de6-k6wz2"
Nov 01 03:52:34.107483 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.107451    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84de6-k6wz2"]
Nov 01 03:52:34.108804 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.108770    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:52:34.108921 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.108808    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:52:34.108921 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.108859    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:52:34.108921 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.108887    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:52:34.171147 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.171115    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fac08faa-3e55-4106-be48-f587ea1b7e12-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d84de6-k6wz2\" (UID: \"fac08faa-3e55-4106-be48-f587ea1b7e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84de6-k6wz2"
Nov 01 03:52:34.171453 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.171419    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fac08faa-3e55-4106-be48-f587ea1b7e12-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d84de6-k6wz2\" (UID: \"fac08faa-3e55-4106-be48-f587ea1b7e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84de6-k6wz2"
Nov 01 03:52:34.171591 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.171533    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fjsb\" (UniqueName: \"kubernetes.io/projected/fac08faa-3e55-4106-be48-f587ea1b7e12-kube-api-access-9fjsb\") pod \"maas-api-revoke-keys-e2e-inf-b-d84de6-k6wz2\" (UID: \"fac08faa-3e55-4106-be48-f587ea1b7e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84de6-k6wz2"
Nov 01 03:52:34.272333 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.272302    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fac08faa-3e55-4106-be48-f587ea1b7e12-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d84de6-k6wz2\" (UID: \"fac08faa-3e55-4106-be48-f587ea1b7e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84de6-k6wz2"
Nov 01 03:52:34.272484 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.272374    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fac08faa-3e55-4106-be48-f587ea1b7e12-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d84de6-k6wz2\" (UID: \"fac08faa-3e55-4106-be48-f587ea1b7e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84de6-k6wz2"
Nov 01 03:52:34.272484 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.272405    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fjsb\" (UniqueName: \"kubernetes.io/projected/fac08faa-3e55-4106-be48-f587ea1b7e12-kube-api-access-9fjsb\") pod \"maas-api-revoke-keys-e2e-inf-b-d84de6-k6wz2\" (UID: \"fac08faa-3e55-4106-be48-f587ea1b7e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84de6-k6wz2"
Nov 01 03:52:34.273024 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.272998    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fac08faa-3e55-4106-be48-f587ea1b7e12-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d84de6-k6wz2\" (UID: \"fac08faa-3e55-4106-be48-f587ea1b7e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84de6-k6wz2"
Nov 01 03:52:34.274806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.274785    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fac08faa-3e55-4106-be48-f587ea1b7e12-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d84de6-k6wz2\" (UID: \"fac08faa-3e55-4106-be48-f587ea1b7e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84de6-k6wz2"
Nov 01 03:52:34.333443 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.333392    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fjsb\" (UniqueName: \"kubernetes.io/projected/fac08faa-3e55-4106-be48-f587ea1b7e12-kube-api-access-9fjsb\") pod \"maas-api-revoke-keys-e2e-inf-b-d84de6-k6wz2\" (UID: \"fac08faa-3e55-4106-be48-f587ea1b7e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84de6-k6wz2"
Nov 01 03:52:34.371995 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.371971    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84de6-k6wz2"
Nov 01 03:52:34.753135 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.753100    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84de6-k6wz2"]
Nov 01 03:52:34.754255 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:52:34.754229    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfac08faa_3e55_4106_be48_f587ea1b7e12.slice/crio-de7289280e14b07cec237f0d6c6231da6692c6d6c7379cc3ecd07481d4f9423d WatchSource:0}: Error finding container de7289280e14b07cec237f0d6c6231da6692c6d6c7379cc3ecd07481d4f9423d: Status 404 returned error can't find the container with id de7289280e14b07cec237f0d6c6231da6692c6d6c7379cc3ecd07481d4f9423d
Nov 01 03:52:34.799302 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:34.799272    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84de6-k6wz2" event={"ID":"fac08faa-3e55-4106-be48-f587ea1b7e12","Type":"ContainerStarted","Data":"de7289280e14b07cec237f0d6c6231da6692c6d6c7379cc3ecd07481d4f9423d"}
Nov 01 03:52:35.669514 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:35.669443    4930 patch_prober.go:29] interesting pod/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:52:35.669514 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:35.669493    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x" podUID="7d9844bf-a8fe-4904-87fa-fb427c29dbd6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:52:35.808398 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:35.808369    4930 generic.go:356] "Generic (PLEG): container finished" podID="fac08faa-3e55-4106-be48-f587ea1b7e12" containerID="d4918dec59c25f9647744d3c957d6892b52149381e3b4dad0d6232ad7c0fe355" exitCode=0
Nov 01 03:52:35.808568 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:35.808426    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84de6-k6wz2" event={"ID":"fac08faa-3e55-4106-be48-f587ea1b7e12","Type":"ContainerDied","Data":"d4918dec59c25f9647744d3c957d6892b52149381e3b4dad0d6232ad7c0fe355"}
Nov 01 03:52:36.185826 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:36.185790    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:36.959594 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:36.959569    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84de6-k6wz2"
Nov 01 03:52:37.009512 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:37.009478    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fac08faa-3e55-4106-be48-f587ea1b7e12-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fac08faa-3e55-4106-be48-f587ea1b7e12-maas-api-xks-ca\") pod \"fac08faa-3e55-4106-be48-f587ea1b7e12\" (UID: \"fac08faa-3e55-4106-be48-f587ea1b7e12\") "
Nov 01 03:52:37.009687 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:37.009588    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fac08faa-3e55-4106-be48-f587ea1b7e12-kube-api-access-9fjsb\" (UniqueName: \"kubernetes.io/projected/fac08faa-3e55-4106-be48-f587ea1b7e12-kube-api-access-9fjsb\") pod \"fac08faa-3e55-4106-be48-f587ea1b7e12\" (UID: \"fac08faa-3e55-4106-be48-f587ea1b7e12\") "
Nov 01 03:52:37.009774 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:37.009738    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fac08faa-3e55-4106-be48-f587ea1b7e12-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fac08faa-3e55-4106-be48-f587ea1b7e12-maas-api-openshift-ca\") pod \"fac08faa-3e55-4106-be48-f587ea1b7e12\" (UID: \"fac08faa-3e55-4106-be48-f587ea1b7e12\") "
Nov 01 03:52:37.010147 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:37.010119    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fac08faa-3e55-4106-be48-f587ea1b7e12-maas-api-openshift-ca" pod "fac08faa-3e55-4106-be48-f587ea1b7e12" (UID: "fac08faa-3e55-4106-be48-f587ea1b7e12"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:52:37.012593 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:37.012122    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fac08faa-3e55-4106-be48-f587ea1b7e12-kube-api-access-9fjsb" pod "fac08faa-3e55-4106-be48-f587ea1b7e12" (UID: "fac08faa-3e55-4106-be48-f587ea1b7e12"). InnerVolumeSpecName "kube-api-access-9fjsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:52:37.012593 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:37.012297    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fac08faa-3e55-4106-be48-f587ea1b7e12-maas-api-xks-ca" pod "fac08faa-3e55-4106-be48-f587ea1b7e12" (UID: "fac08faa-3e55-4106-be48-f587ea1b7e12"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:52:37.111615 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:37.111586    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fac08faa-3e55-4106-be48-f587ea1b7e12-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:37.111615 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:37.111617    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fac08faa-3e55-4106-be48-f587ea1b7e12-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:37.111816 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:37.111632    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fjsb\" (UniqueName: \"kubernetes.io/projected/fac08faa-3e55-4106-be48-f587ea1b7e12-kube-api-access-9fjsb\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:37.825465 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:37.825424    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84de6-k6wz2" event={"ID":"fac08faa-3e55-4106-be48-f587ea1b7e12","Type":"ContainerDied","Data":"de7289280e14b07cec237f0d6c6231da6692c6d6c7379cc3ecd07481d4f9423d"}
Nov 01 03:52:37.825465 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:37.825456    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84de6-k6wz2"
Nov 01 03:52:37.825754 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:37.825478    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de7289280e14b07cec237f0d6c6231da6692c6d6c7379cc3ecd07481d4f9423d"
Nov 01 03:52:38.453481 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:38.453421    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"
Nov 01 03:52:38.526292 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:38.526258    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-workload-certs\") pod \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\" (UID: \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\") "
Nov 01 03:52:38.526449 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:38.526329    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-istio-data\") pod \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\" (UID: \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\") "
Nov 01 03:52:38.526449 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:38.526363    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-istiod-ca-cert\") pod \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\" (UID: \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\") "
Nov 01 03:52:38.526449 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:38.526404    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-istio-podinfo\") pod \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\" (UID: \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\") "
Nov 01 03:52:38.526666 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:38.526458    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-credential-socket\") pod \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\" (UID: \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\") "
Nov 01 03:52:38.526666 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:38.526488    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-workload-certs" pod "7d9844bf-a8fe-4904-87fa-fb427c29dbd6" (UID: "7d9844bf-a8fe-4904-87fa-fb427c29dbd6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:52:38.526666 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:38.526497    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-kube-api-access-n6d6p\" (UniqueName: \"kubernetes.io/projected/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-kube-api-access-n6d6p\") pod \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\" (UID: \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\") "
Nov 01 03:52:38.526666 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:38.526630    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-istio-data" pod "7d9844bf-a8fe-4904-87fa-fb427c29dbd6" (UID: "7d9844bf-a8fe-4904-87fa-fb427c29dbd6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:52:38.526666 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:38.526640    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-workload-socket\") pod \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\" (UID: \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\") "
Nov 01 03:52:38.526931 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:38.526742    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-istiod-ca-cert" pod "7d9844bf-a8fe-4904-87fa-fb427c29dbd6" (UID: "7d9844bf-a8fe-4904-87fa-fb427c29dbd6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:52:38.526931 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:38.526803    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-credential-socket" pod "7d9844bf-a8fe-4904-87fa-fb427c29dbd6" (UID: "7d9844bf-a8fe-4904-87fa-fb427c29dbd6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:52:38.526931 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:38.526865    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-workload-socket" pod "7d9844bf-a8fe-4904-87fa-fb427c29dbd6" (UID: "7d9844bf-a8fe-4904-87fa-fb427c29dbd6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:52:38.527085 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:38.526971    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-istio-token\" (UniqueName: \"kubernetes.io/projected/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-istio-token\") pod \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\" (UID: \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\") "
Nov 01 03:52:38.527085 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:38.527005    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-istio-envoy\") pod \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\" (UID: \"7d9844bf-a8fe-4904-87fa-fb427c29dbd6\") "
Nov 01 03:52:38.527444 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:38.527424    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:38.527533 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:38.527447    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:38.527533 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:38.527462    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:38.527533 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:38.527476    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:38.527533 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:38.527490    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:38.529248 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:38.529224    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-istio-podinfo" pod "7d9844bf-a8fe-4904-87fa-fb427c29dbd6" (UID: "7d9844bf-a8fe-4904-87fa-fb427c29dbd6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:52:38.529330 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:38.529252    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-kube-api-access-n6d6p" pod "7d9844bf-a8fe-4904-87fa-fb427c29dbd6" (UID: "7d9844bf-a8fe-4904-87fa-fb427c29dbd6"). InnerVolumeSpecName "kube-api-access-n6d6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:52:38.529537 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:38.529518    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-istio-token" pod "7d9844bf-a8fe-4904-87fa-fb427c29dbd6" (UID: "7d9844bf-a8fe-4904-87fa-fb427c29dbd6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:52:38.529875 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:38.529861    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-istio-envoy" pod "7d9844bf-a8fe-4904-87fa-fb427c29dbd6" (UID: "7d9844bf-a8fe-4904-87fa-fb427c29dbd6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:52:38.628684 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:38.628657    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:38.628684 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:38.628682    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6d6p\" (UniqueName: \"kubernetes.io/projected/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-kube-api-access-n6d6p\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:38.628867 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:38.628693    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:38.628867 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:38.628702    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d9844bf-a8fe-4904-87fa-fb427c29dbd6-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:38.835270 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:38.835233    4930 generic.go:356] "Generic (PLEG): container finished" podID="7d9844bf-a8fe-4904-87fa-fb427c29dbd6" containerID="393f985c36b58e02c5a6e6aaebd4acbac702a308cda4c445dff49ff48a13ae19" exitCode=0
Nov 01 03:52:38.835442 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:38.835293    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"
Nov 01 03:52:38.835442 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:38.835323    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x" event={"ID":"7d9844bf-a8fe-4904-87fa-fb427c29dbd6","Type":"ContainerDied","Data":"393f985c36b58e02c5a6e6aaebd4acbac702a308cda4c445dff49ff48a13ae19"}
Nov 01 03:52:38.835442 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:38.835390    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x" event={"ID":"7d9844bf-a8fe-4904-87fa-fb427c29dbd6","Type":"ContainerDied","Data":"a26d2c5d7e74cd33411111542a6472ad81ff758fbe33dca5ff69130d6114d711"}
Nov 01 03:52:38.835442 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:38.835418    4930 scope.go:122] "RemoveContainer" containerID="393f985c36b58e02c5a6e6aaebd4acbac702a308cda4c445dff49ff48a13ae19"
Nov 01 03:52:38.846435 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:38.846417    4930 scope.go:122] "RemoveContainer" containerID="cbb49232e7db24c7465743c1aecb38f7063b1da1d6f87afac56dbc6c2f4dce11"
Nov 01 03:52:38.855448 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:38.855429    4930 scope.go:122] "RemoveContainer" containerID="393f985c36b58e02c5a6e6aaebd4acbac702a308cda4c445dff49ff48a13ae19"
Nov 01 03:52:38.855725 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:52:38.855707    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"393f985c36b58e02c5a6e6aaebd4acbac702a308cda4c445dff49ff48a13ae19\": container with ID starting with 393f985c36b58e02c5a6e6aaebd4acbac702a308cda4c445dff49ff48a13ae19 not found: ID does not exist" containerID="393f985c36b58e02c5a6e6aaebd4acbac702a308cda4c445dff49ff48a13ae19"
Nov 01 03:52:38.855765 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:38.855734    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"393f985c36b58e02c5a6e6aaebd4acbac702a308cda4c445dff49ff48a13ae19"} err="failed to get container status \"393f985c36b58e02c5a6e6aaebd4acbac702a308cda4c445dff49ff48a13ae19\": rpc error: code = NotFound desc = could not find container \"393f985c36b58e02c5a6e6aaebd4acbac702a308cda4c445dff49ff48a13ae19\": container with ID starting with 393f985c36b58e02c5a6e6aaebd4acbac702a308cda4c445dff49ff48a13ae19 not found: ID does not exist"
Nov 01 03:52:38.855765 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:38.855750    4930 scope.go:122] "RemoveContainer" containerID="cbb49232e7db24c7465743c1aecb38f7063b1da1d6f87afac56dbc6c2f4dce11"
Nov 01 03:52:38.855981 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:52:38.855964    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbb49232e7db24c7465743c1aecb38f7063b1da1d6f87afac56dbc6c2f4dce11\": container with ID starting with cbb49232e7db24c7465743c1aecb38f7063b1da1d6f87afac56dbc6c2f4dce11 not found: ID does not exist" containerID="cbb49232e7db24c7465743c1aecb38f7063b1da1d6f87afac56dbc6c2f4dce11"
Nov 01 03:52:38.856019 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:38.855989    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbb49232e7db24c7465743c1aecb38f7063b1da1d6f87afac56dbc6c2f4dce11"} err="failed to get container status \"cbb49232e7db24c7465743c1aecb38f7063b1da1d6f87afac56dbc6c2f4dce11\": rpc error: code = NotFound desc = could not find container \"cbb49232e7db24c7465743c1aecb38f7063b1da1d6f87afac56dbc6c2f4dce11\": container with ID starting with cbb49232e7db24c7465743c1aecb38f7063b1da1d6f87afac56dbc6c2f4dce11 not found: ID does not exist"
Nov 01 03:52:39.002773 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:39.002743    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"]
Nov 01 03:52:39.007533 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:39.007504    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-22e11b-openshift-default-64f8d9b9d9-gl22x"]
Nov 01 03:52:39.620373 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:39.620330    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d9844bf-a8fe-4904-87fa-fb427c29dbd6" path="/var/lib/kubelet/pods/7d9844bf-a8fe-4904-87fa-fb427c29dbd6/volumes"
Nov 01 03:52:40.019822 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:40.019733    4930 scope.go:122] "RemoveContainer" containerID="8d960751e099b7006c7f5ed8ea67b1dfa12bc03926be84ccced86c44d92a2d3d"
Nov 01 03:52:40.029221 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:40.029199    4930 scope.go:122] "RemoveContainer" containerID="178af8eef4981adaa2a84b281e9506e29c21e0701cb3c96cffa502f942e81468"
Nov 01 03:52:40.037833 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:40.037795    4930 scope.go:122] "RemoveContainer" containerID="48d7a189b781566bb304c358915d7d08ad7615eff2e76d5506e02eeb97c6f800"
Nov 01 03:52:41.192340 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:41.192298    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:42.452444 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:42.452410    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:43.165950 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:43.165911    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:43.311890 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:43.311854    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:44.306286 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:44.306254    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:45.557059 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:45.557007    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:45.879978 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:45.879943    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:46.021596 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:46.021537    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:46.719120 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:46.719087    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-541ff960-c986h"]
Nov 01 03:52:46.719541 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:46.719504    4930 cpu_manager.go:414] "containerMap: removing container" podUID="fac08faa-3e55-4106-be48-f587ea1b7e12" containerName="revoke-keys"
Nov 01 03:52:46.719541 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:46.719514    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fac08faa-3e55-4106-be48-f587ea1b7e12" containerName="revoke-keys"
Nov 01 03:52:46.719678 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:46.719548    4930 cpu_manager.go:414] "containerMap: removing container" podUID="7d9844bf-a8fe-4904-87fa-fb427c29dbd6" containerName="istio-proxy"
Nov 01 03:52:46.719678 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:46.719568    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d9844bf-a8fe-4904-87fa-fb427c29dbd6" containerName="istio-proxy"
Nov 01 03:52:46.719678 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:46.719586    4930 cpu_manager.go:414] "containerMap: removing container" podUID="7d9844bf-a8fe-4904-87fa-fb427c29dbd6" containerName="istio-proxy"
Nov 01 03:52:46.719678 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:46.719593    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d9844bf-a8fe-4904-87fa-fb427c29dbd6" containerName="istio-proxy"
Nov 01 03:52:46.719848 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:46.719680    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d9844bf-a8fe-4904-87fa-fb427c29dbd6" containerName="istio-proxy"
Nov 01 03:52:46.719848 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:46.719691    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d9844bf-a8fe-4904-87fa-fb427c29dbd6" containerName="istio-proxy"
Nov 01 03:52:46.719848 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:46.719698    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="fac08faa-3e55-4106-be48-f587ea1b7e12" containerName="revoke-keys"
Nov 01 03:52:46.734798 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:46.734765    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-541ff960-c986h"
Nov 01 03:52:46.781388 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:46.781363    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-541ff960-c986h"]
Nov 01 03:52:46.818154 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:46.818124    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw9f2\" (UniqueName: \"kubernetes.io/projected/b0068669-bf76-48d9-8cf4-fd6404431462-kube-api-access-bw9f2\") pod \"maas-api-revoke-sub-541ff960-c986h\" (UID: \"b0068669-bf76-48d9-8cf4-fd6404431462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541ff960-c986h"
Nov 01 03:52:46.818326 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:46.818305    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0068669-bf76-48d9-8cf4-fd6404431462-maas-api-xks-ca\") pod \"maas-api-revoke-sub-541ff960-c986h\" (UID: \"b0068669-bf76-48d9-8cf4-fd6404431462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541ff960-c986h"
Nov 01 03:52:46.818377 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:46.818351    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0068669-bf76-48d9-8cf4-fd6404431462-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-541ff960-c986h\" (UID: \"b0068669-bf76-48d9-8cf4-fd6404431462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541ff960-c986h"
Nov 01 03:52:46.827048 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:46.827024    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:52:46.827192 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:46.827109    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:52:46.827192 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:46.827123    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:52:46.827192 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:46.827120    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:52:46.920060 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:46.920029    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw9f2\" (UniqueName: \"kubernetes.io/projected/b0068669-bf76-48d9-8cf4-fd6404431462-kube-api-access-bw9f2\") pod \"maas-api-revoke-sub-541ff960-c986h\" (UID: \"b0068669-bf76-48d9-8cf4-fd6404431462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541ff960-c986h"
Nov 01 03:52:46.920271 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:46.920131    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0068669-bf76-48d9-8cf4-fd6404431462-maas-api-xks-ca\") pod \"maas-api-revoke-sub-541ff960-c986h\" (UID: \"b0068669-bf76-48d9-8cf4-fd6404431462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541ff960-c986h"
Nov 01 03:52:46.920271 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:46.920159    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0068669-bf76-48d9-8cf4-fd6404431462-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-541ff960-c986h\" (UID: \"b0068669-bf76-48d9-8cf4-fd6404431462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541ff960-c986h"
Nov 01 03:52:46.920779 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:46.920758    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0068669-bf76-48d9-8cf4-fd6404431462-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-541ff960-c986h\" (UID: \"b0068669-bf76-48d9-8cf4-fd6404431462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541ff960-c986h"
Nov 01 03:52:46.922412 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:46.922389    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0068669-bf76-48d9-8cf4-fd6404431462-maas-api-xks-ca\") pod \"maas-api-revoke-sub-541ff960-c986h\" (UID: \"b0068669-bf76-48d9-8cf4-fd6404431462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541ff960-c986h"
Nov 01 03:52:46.940763 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:46.940739    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:46.981234 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:46.981182    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw9f2\" (UniqueName: \"kubernetes.io/projected/b0068669-bf76-48d9-8cf4-fd6404431462-kube-api-access-bw9f2\") pod \"maas-api-revoke-sub-541ff960-c986h\" (UID: \"b0068669-bf76-48d9-8cf4-fd6404431462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541ff960-c986h"
Nov 01 03:52:47.048283 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:47.048256    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-541ff960-c986h"
Nov 01 03:52:47.086105 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:47.084529    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:47.355146 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:47.355113    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:47.467852 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:47.467818    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-541ff960-c986h"]
Nov 01 03:52:47.469119 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:52:47.469089    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0068669_bf76_48d9_8cf4_fd6404431462.slice/crio-bfb8d61a762bd2f520f5f573aaaf3a0262ae9184a51e556390ac687e6eae54ad WatchSource:0}: Error finding container bfb8d61a762bd2f520f5f573aaaf3a0262ae9184a51e556390ac687e6eae54ad: Status 404 returned error can't find the container with id bfb8d61a762bd2f520f5f573aaaf3a0262ae9184a51e556390ac687e6eae54ad
Nov 01 03:52:47.594007 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:47.593974    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b12199-fnb5f"]
Nov 01 03:52:47.619931 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:47.619856    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b12199-fnb5f"]
Nov 01 03:52:47.898887 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:47.898796    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-541ff960-c986h" event={"ID":"b0068669-bf76-48d9-8cf4-fd6404431462","Type":"ContainerStarted","Data":"f3e1ac6f52840531f0d254fe28108cc09991d35a678836245ed993fb65c7998f"}
Nov 01 03:52:47.898887 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:47.898862    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-541ff960-c986h" event={"ID":"b0068669-bf76-48d9-8cf4-fd6404431462","Type":"ContainerStarted","Data":"bfb8d61a762bd2f520f5f573aaaf3a0262ae9184a51e556390ac687e6eae54ad"}
Nov 01 03:52:48.016213 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:48.016182    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:48.017765 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:48.017720    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-541ff960-c986h" podStartSLOduration=2.017707655 podStartE2EDuration="2.017707655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:52:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:52:48.015098726 +0000 UTC m=+2772.538105315" watchObservedRunningTime="2026-11-01 03:52:48.017707655 +0000 UTC m=+2772.540714243"
Nov 01 03:52:48.615849 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:48.615811    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:48.908501 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:48.908466    4930 generic.go:356] "Generic (PLEG): container finished" podID="b0068669-bf76-48d9-8cf4-fd6404431462" containerID="f3e1ac6f52840531f0d254fe28108cc09991d35a678836245ed993fb65c7998f" exitCode=0
Nov 01 03:52:48.908909 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:48.908548    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-541ff960-c986h" event={"ID":"b0068669-bf76-48d9-8cf4-fd6404431462","Type":"ContainerDied","Data":"f3e1ac6f52840531f0d254fe28108cc09991d35a678836245ed993fb65c7998f"}
Nov 01 03:52:48.995068 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:48.995032    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:49.122155 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:52:49.122121    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:52:49.224904 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:49.224827    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-7cd000-wxqql"]
Nov 01 03:52:49.238950 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:49.238922    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-7cd000-wxqql"]
Nov 01 03:52:49.244851 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:49.244826    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:49.621217 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:49.621176    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33b901de-97ee-4363-9fac-f96499633f86" path="/var/lib/kubelet/pods/33b901de-97ee-4363-9fac-f96499633f86/volumes"
Nov 01 03:52:49.621911 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:49.621887    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e344a2a-f8a8-4531-809e-db674de9f556" path="/var/lib/kubelet/pods/9e344a2a-f8a8-4531-809e-db674de9f556/volumes"
Nov 01 03:52:49.688510 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:49.688458    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:50.065533 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:50.065500    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-541ff960-c986h"
Nov 01 03:52:50.076320 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:50.076296    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0068669-bf76-48d9-8cf4-fd6404431462-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0068669-bf76-48d9-8cf4-fd6404431462-maas-api-xks-ca\") pod \"b0068669-bf76-48d9-8cf4-fd6404431462\" (UID: \"b0068669-bf76-48d9-8cf4-fd6404431462\") "
Nov 01 03:52:50.076432 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:50.076374    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0068669-bf76-48d9-8cf4-fd6404431462-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0068669-bf76-48d9-8cf4-fd6404431462-maas-api-openshift-ca\") pod \"b0068669-bf76-48d9-8cf4-fd6404431462\" (UID: \"b0068669-bf76-48d9-8cf4-fd6404431462\") "
Nov 01 03:52:50.076432 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:50.076406    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0068669-bf76-48d9-8cf4-fd6404431462-kube-api-access-bw9f2\" (UniqueName: \"kubernetes.io/projected/b0068669-bf76-48d9-8cf4-fd6404431462-kube-api-access-bw9f2\") pod \"b0068669-bf76-48d9-8cf4-fd6404431462\" (UID: \"b0068669-bf76-48d9-8cf4-fd6404431462\") "
Nov 01 03:52:50.076804 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:50.076766    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0068669-bf76-48d9-8cf4-fd6404431462-maas-api-openshift-ca" pod "b0068669-bf76-48d9-8cf4-fd6404431462" (UID: "b0068669-bf76-48d9-8cf4-fd6404431462"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:52:50.078485 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:50.078452    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0068669-bf76-48d9-8cf4-fd6404431462-maas-api-xks-ca" pod "b0068669-bf76-48d9-8cf4-fd6404431462" (UID: "b0068669-bf76-48d9-8cf4-fd6404431462"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:52:50.078683 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:50.078604    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0068669-bf76-48d9-8cf4-fd6404431462-kube-api-access-bw9f2" pod "b0068669-bf76-48d9-8cf4-fd6404431462" (UID: "b0068669-bf76-48d9-8cf4-fd6404431462"). InnerVolumeSpecName "kube-api-access-bw9f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:52:50.178071 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:50.177999    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0068669-bf76-48d9-8cf4-fd6404431462-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:50.178071 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:50.178030    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw9f2\" (UniqueName: \"kubernetes.io/projected/b0068669-bf76-48d9-8cf4-fd6404431462-kube-api-access-bw9f2\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:50.178071 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:50.178042    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0068669-bf76-48d9-8cf4-fd6404431462-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:52:50.403662 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:50.403612    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:50.543959 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:50.543870    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:50.921688 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:50.921656    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-541ff960-c986h"
Nov 01 03:52:50.921863 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:50.921684    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-541ff960-c986h" event={"ID":"b0068669-bf76-48d9-8cf4-fd6404431462","Type":"ContainerDied","Data":"bfb8d61a762bd2f520f5f573aaaf3a0262ae9184a51e556390ac687e6eae54ad"}
Nov 01 03:52:50.921863 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:50.921732    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfb8d61a762bd2f520f5f573aaaf3a0262ae9184a51e556390ac687e6eae54ad"
Nov 01 03:52:51.976576 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:51.970189    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:53.506343 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:53.506313    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:55.203422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:55.203383    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:55.407656 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:55.407619    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:55.555238 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:55.555141    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:55.674208 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:55.674174    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84de6-k6wz2"]
Nov 01 03:52:55.759626 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:55.759588    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d84de6-k6wz2"]
Nov 01 03:52:55.833792 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:55.833689    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:56.085387 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:56.085352    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:57.224133 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:57.224079    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:57.369549 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:57.369511    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:57.559016 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:57.558916    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:52:57.569310 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:57.569278    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"]
Nov 01 03:52:57.569605 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:57.569551    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8" podUID="baec5da4-406d-4923-aa69-69820f5e04c0" containerName="istio-proxy" containerID="cri-o://7e5397471b831ab3918ce97d528cff62e209f15466024e7a6e29760e8ab7ce3a" gracePeriod=30
Nov 01 03:52:57.624170 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:57.624123    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fac08faa-3e55-4106-be48-f587ea1b7e12" path="/var/lib/kubelet/pods/fac08faa-3e55-4106-be48-f587ea1b7e12/volumes"
Nov 01 03:52:58.427419 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:58.427389    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-211380-l6w9l"]
Nov 01 03:52:58.427898 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:58.427882    4930 cpu_manager.go:414] "containerMap: removing container" podUID="b0068669-bf76-48d9-8cf4-fd6404431462" containerName="revoke-keys"
Nov 01 03:52:58.427944 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:58.427899    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0068669-bf76-48d9-8cf4-fd6404431462" containerName="revoke-keys"
Nov 01 03:52:58.427994 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:58.427985    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0068669-bf76-48d9-8cf4-fd6404431462" containerName="revoke-keys"
Nov 01 03:52:58.432369 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:58.432353    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-211380-l6w9l"
Nov 01 03:52:58.470638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:58.470603    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b73f5e95-80e1-44ad-9bd8-39e111f98fca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-211380-l6w9l\" (UID: \"b73f5e95-80e1-44ad-9bd8-39e111f98fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-211380-l6w9l"
Nov 01 03:52:58.470776 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:58.470659    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n74ls\" (UniqueName: \"kubernetes.io/projected/b73f5e95-80e1-44ad-9bd8-39e111f98fca-kube-api-access-n74ls\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-211380-l6w9l\" (UID: \"b73f5e95-80e1-44ad-9bd8-39e111f98fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-211380-l6w9l"
Nov 01 03:52:58.470855 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:58.470827    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b73f5e95-80e1-44ad-9bd8-39e111f98fca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-211380-l6w9l\" (UID: \"b73f5e95-80e1-44ad-9bd8-39e111f98fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-211380-l6w9l"
Nov 01 03:52:58.486495 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:58.486454    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:52:58.535988 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:58.535962    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:52:58.536130 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:58.536068    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:52:58.536130 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:58.536119    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:52:58.539542 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:58.539517    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-211380-l6w9l"]
Nov 01 03:52:58.572134 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:58.572106    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n74ls\" (UniqueName: \"kubernetes.io/projected/b73f5e95-80e1-44ad-9bd8-39e111f98fca-kube-api-access-n74ls\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-211380-l6w9l\" (UID: \"b73f5e95-80e1-44ad-9bd8-39e111f98fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-211380-l6w9l"
Nov 01 03:52:58.572278 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:58.572209    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b73f5e95-80e1-44ad-9bd8-39e111f98fca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-211380-l6w9l\" (UID: \"b73f5e95-80e1-44ad-9bd8-39e111f98fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-211380-l6w9l"
Nov 01 03:52:58.572373 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:58.572356    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b73f5e95-80e1-44ad-9bd8-39e111f98fca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-211380-l6w9l\" (UID: \"b73f5e95-80e1-44ad-9bd8-39e111f98fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-211380-l6w9l"
Nov 01 03:52:58.573118 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:58.573092    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b73f5e95-80e1-44ad-9bd8-39e111f98fca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-211380-l6w9l\" (UID: \"b73f5e95-80e1-44ad-9bd8-39e111f98fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-211380-l6w9l"
Nov 01 03:52:58.574677 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:58.574654    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b73f5e95-80e1-44ad-9bd8-39e111f98fca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-211380-l6w9l\" (UID: \"b73f5e95-80e1-44ad-9bd8-39e111f98fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-211380-l6w9l"
Nov 01 03:52:58.632960 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:58.632932    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n74ls\" (UniqueName: \"kubernetes.io/projected/b73f5e95-80e1-44ad-9bd8-39e111f98fca-kube-api-access-n74ls\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-211380-l6w9l\" (UID: \"b73f5e95-80e1-44ad-9bd8-39e111f98fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-211380-l6w9l"
Nov 01 03:52:58.745286 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:58.745219    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-211380-l6w9l"
Nov 01 03:52:59.129951 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:59.129923    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-211380-l6w9l"]
Nov 01 03:52:59.131212 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:52:59.131173    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb73f5e95_80e1_44ad_9bd8_39e111f98fca.slice/crio-3c9c6996ee682d4b527b9c9d88b7b98ce21378fd75800bc7798249aca5fc76b1 WatchSource:0}: Error finding container 3c9c6996ee682d4b527b9c9d88b7b98ce21378fd75800bc7798249aca5fc76b1: Status 404 returned error can't find the container with id 3c9c6996ee682d4b527b9c9d88b7b98ce21378fd75800bc7798249aca5fc76b1
Nov 01 03:52:59.985182 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:59.985147    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-211380-l6w9l" event={"ID":"b73f5e95-80e1-44ad-9bd8-39e111f98fca","Type":"ContainerStarted","Data":"3932c3aada2c330e20152084459fac969d882e2fb3085fddc5f06696dc60f87a"}
Nov 01 03:52:59.985613 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:52:59.985210    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-211380-l6w9l" event={"ID":"b73f5e95-80e1-44ad-9bd8-39e111f98fca","Type":"ContainerStarted","Data":"3c9c6996ee682d4b527b9c9d88b7b98ce21378fd75800bc7798249aca5fc76b1"}
Nov 01 03:53:00.104305 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:00.104250    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-211380-l6w9l" podStartSLOduration=2.104234331 podStartE2EDuration="2.104234331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:52:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:53:00.100964931 +0000 UTC m=+2784.623971521" watchObservedRunningTime="2026-11-01 03:53:00.104234331 +0000 UTC m=+2784.627240919"
Nov 01 03:53:00.607126 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:00.607098    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:00.993861 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:00.993775    4930 generic.go:356] "Generic (PLEG): container finished" podID="b73f5e95-80e1-44ad-9bd8-39e111f98fca" containerID="3932c3aada2c330e20152084459fac969d882e2fb3085fddc5f06696dc60f87a" exitCode=0
Nov 01 03:53:00.994248 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:00.993865    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-211380-l6w9l" event={"ID":"b73f5e95-80e1-44ad-9bd8-39e111f98fca","Type":"ContainerDied","Data":"3932c3aada2c330e20152084459fac969d882e2fb3085fddc5f06696dc60f87a"}
Nov 01 03:53:02.147261 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:02.147236    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-211380-l6w9l"
Nov 01 03:53:02.213751 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:02.213725    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b73f5e95-80e1-44ad-9bd8-39e111f98fca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b73f5e95-80e1-44ad-9bd8-39e111f98fca-maas-api-openshift-ca\") pod \"b73f5e95-80e1-44ad-9bd8-39e111f98fca\" (UID: \"b73f5e95-80e1-44ad-9bd8-39e111f98fca\") "
Nov 01 03:53:02.213895 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:02.213786    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b73f5e95-80e1-44ad-9bd8-39e111f98fca-kube-api-access-n74ls\" (UniqueName: \"kubernetes.io/projected/b73f5e95-80e1-44ad-9bd8-39e111f98fca-kube-api-access-n74ls\") pod \"b73f5e95-80e1-44ad-9bd8-39e111f98fca\" (UID: \"b73f5e95-80e1-44ad-9bd8-39e111f98fca\") "
Nov 01 03:53:02.213895 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:02.213863    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b73f5e95-80e1-44ad-9bd8-39e111f98fca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b73f5e95-80e1-44ad-9bd8-39e111f98fca-maas-api-xks-ca\") pod \"b73f5e95-80e1-44ad-9bd8-39e111f98fca\" (UID: \"b73f5e95-80e1-44ad-9bd8-39e111f98fca\") "
Nov 01 03:53:02.214060 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:02.214039    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b73f5e95-80e1-44ad-9bd8-39e111f98fca-maas-api-openshift-ca" pod "b73f5e95-80e1-44ad-9bd8-39e111f98fca" (UID: "b73f5e95-80e1-44ad-9bd8-39e111f98fca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:53:02.215937 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:02.215911    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b73f5e95-80e1-44ad-9bd8-39e111f98fca-maas-api-xks-ca" pod "b73f5e95-80e1-44ad-9bd8-39e111f98fca" (UID: "b73f5e95-80e1-44ad-9bd8-39e111f98fca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:53:02.216011 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:02.215965    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b73f5e95-80e1-44ad-9bd8-39e111f98fca-kube-api-access-n74ls" pod "b73f5e95-80e1-44ad-9bd8-39e111f98fca" (UID: "b73f5e95-80e1-44ad-9bd8-39e111f98fca"). InnerVolumeSpecName "kube-api-access-n74ls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:53:02.315282 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:02.315208    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b73f5e95-80e1-44ad-9bd8-39e111f98fca-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:02.315282 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:02.315234    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b73f5e95-80e1-44ad-9bd8-39e111f98fca-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:02.315282 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:02.315245    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n74ls\" (UniqueName: \"kubernetes.io/projected/b73f5e95-80e1-44ad-9bd8-39e111f98fca-kube-api-access-n74ls\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:02.477668 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:02.477633    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:02.668357 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:02.668324    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:02.837358 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:02.837327    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:02.841972 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:02.841949    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"
Nov 01 03:53:02.926996 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:02.926958    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baec5da4-406d-4923-aa69-69820f5e04c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baec5da4-406d-4923-aa69-69820f5e04c0-workload-certs\") pod \"baec5da4-406d-4923-aa69-69820f5e04c0\" (UID: \"baec5da4-406d-4923-aa69-69820f5e04c0\") "
Nov 01 03:53:02.927222 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:02.927061    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/baec5da4-406d-4923-aa69-69820f5e04c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baec5da4-406d-4923-aa69-69820f5e04c0-istio-podinfo\") pod \"baec5da4-406d-4923-aa69-69820f5e04c0\" (UID: \"baec5da4-406d-4923-aa69-69820f5e04c0\") "
Nov 01 03:53:02.927222 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:02.927112    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baec5da4-406d-4923-aa69-69820f5e04c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baec5da4-406d-4923-aa69-69820f5e04c0-istio-envoy\") pod \"baec5da4-406d-4923-aa69-69820f5e04c0\" (UID: \"baec5da4-406d-4923-aa69-69820f5e04c0\") "
Nov 01 03:53:02.927222 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:02.927193    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baec5da4-406d-4923-aa69-69820f5e04c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baec5da4-406d-4923-aa69-69820f5e04c0-credential-socket\") pod \"baec5da4-406d-4923-aa69-69820f5e04c0\" (UID: \"baec5da4-406d-4923-aa69-69820f5e04c0\") "
Nov 01 03:53:02.927415 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:02.927223    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baec5da4-406d-4923-aa69-69820f5e04c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baec5da4-406d-4923-aa69-69820f5e04c0-istiod-ca-cert\") pod \"baec5da4-406d-4923-aa69-69820f5e04c0\" (UID: \"baec5da4-406d-4923-aa69-69820f5e04c0\") "
Nov 01 03:53:02.927415 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:02.927307    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baec5da4-406d-4923-aa69-69820f5e04c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baec5da4-406d-4923-aa69-69820f5e04c0-istio-data\") pod \"baec5da4-406d-4923-aa69-69820f5e04c0\" (UID: \"baec5da4-406d-4923-aa69-69820f5e04c0\") "
Nov 01 03:53:02.927415 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:02.927355    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baec5da4-406d-4923-aa69-69820f5e04c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baec5da4-406d-4923-aa69-69820f5e04c0-workload-socket\") pod \"baec5da4-406d-4923-aa69-69820f5e04c0\" (UID: \"baec5da4-406d-4923-aa69-69820f5e04c0\") "
Nov 01 03:53:02.927415 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:02.927393    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baec5da4-406d-4923-aa69-69820f5e04c0-kube-api-access-9kqp8\" (UniqueName: \"kubernetes.io/projected/baec5da4-406d-4923-aa69-69820f5e04c0-kube-api-access-9kqp8\") pod \"baec5da4-406d-4923-aa69-69820f5e04c0\" (UID: \"baec5da4-406d-4923-aa69-69820f5e04c0\") "
Nov 01 03:53:02.927704 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:02.927461    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baec5da4-406d-4923-aa69-69820f5e04c0-istio-token\" (UniqueName: \"kubernetes.io/projected/baec5da4-406d-4923-aa69-69820f5e04c0-istio-token\") pod \"baec5da4-406d-4923-aa69-69820f5e04c0\" (UID: \"baec5da4-406d-4923-aa69-69820f5e04c0\") "
Nov 01 03:53:02.928999 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:02.928927    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baec5da4-406d-4923-aa69-69820f5e04c0-credential-socket" pod "baec5da4-406d-4923-aa69-69820f5e04c0" (UID: "baec5da4-406d-4923-aa69-69820f5e04c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:02.929670 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:02.929591    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baec5da4-406d-4923-aa69-69820f5e04c0-workload-certs" pod "baec5da4-406d-4923-aa69-69820f5e04c0" (UID: "baec5da4-406d-4923-aa69-69820f5e04c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:02.929919 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:02.929839    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baec5da4-406d-4923-aa69-69820f5e04c0-istiod-ca-cert" pod "baec5da4-406d-4923-aa69-69820f5e04c0" (UID: "baec5da4-406d-4923-aa69-69820f5e04c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:53:02.930416 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:02.930286    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baec5da4-406d-4923-aa69-69820f5e04c0-istio-data" pod "baec5da4-406d-4923-aa69-69820f5e04c0" (UID: "baec5da4-406d-4923-aa69-69820f5e04c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:02.930796 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:02.930594    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baec5da4-406d-4923-aa69-69820f5e04c0-workload-socket" pod "baec5da4-406d-4923-aa69-69820f5e04c0" (UID: "baec5da4-406d-4923-aa69-69820f5e04c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:02.934476 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:02.934440    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baec5da4-406d-4923-aa69-69820f5e04c0-istio-envoy" pod "baec5da4-406d-4923-aa69-69820f5e04c0" (UID: "baec5da4-406d-4923-aa69-69820f5e04c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:02.934884 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:02.934857    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baec5da4-406d-4923-aa69-69820f5e04c0-kube-api-access-9kqp8" pod "baec5da4-406d-4923-aa69-69820f5e04c0" (UID: "baec5da4-406d-4923-aa69-69820f5e04c0"). InnerVolumeSpecName "kube-api-access-9kqp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:53:02.935688 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:02.935653    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/baec5da4-406d-4923-aa69-69820f5e04c0-istio-podinfo" pod "baec5da4-406d-4923-aa69-69820f5e04c0" (UID: "baec5da4-406d-4923-aa69-69820f5e04c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:53:02.938494 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:02.938467    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baec5da4-406d-4923-aa69-69820f5e04c0-istio-token" pod "baec5da4-406d-4923-aa69-69820f5e04c0" (UID: "baec5da4-406d-4923-aa69-69820f5e04c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:53:03.008570 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.008517    4930 generic.go:356] "Generic (PLEG): container finished" podID="baec5da4-406d-4923-aa69-69820f5e04c0" containerID="7e5397471b831ab3918ce97d528cff62e209f15466024e7a6e29760e8ab7ce3a" exitCode=0
Nov 01 03:53:03.008758 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.008598    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8" event={"ID":"baec5da4-406d-4923-aa69-69820f5e04c0","Type":"ContainerDied","Data":"7e5397471b831ab3918ce97d528cff62e209f15466024e7a6e29760e8ab7ce3a"}
Nov 01 03:53:03.008758 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.008626    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"
Nov 01 03:53:03.008758 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.008662    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8" event={"ID":"baec5da4-406d-4923-aa69-69820f5e04c0","Type":"ContainerDied","Data":"25e0d6224b7bc8c13ef531f577a8a77ccb853e0357982416d6d0ce32c85f1f53"}
Nov 01 03:53:03.008758 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.008695    4930 scope.go:122] "RemoveContainer" containerID="7e5397471b831ab3918ce97d528cff62e209f15466024e7a6e29760e8ab7ce3a"
Nov 01 03:53:03.015709 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.015682    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-211380-l6w9l" event={"ID":"b73f5e95-80e1-44ad-9bd8-39e111f98fca","Type":"ContainerDied","Data":"3c9c6996ee682d4b527b9c9d88b7b98ce21378fd75800bc7798249aca5fc76b1"}
Nov 01 03:53:03.015831 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.015731    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c9c6996ee682d4b527b9c9d88b7b98ce21378fd75800bc7798249aca5fc76b1"
Nov 01 03:53:03.015831 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.015791    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-211380-l6w9l"
Nov 01 03:53:03.026363 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.026343    4930 scope.go:122] "RemoveContainer" containerID="231cb1e1808fd5e653cc3939494ddc5f5c5e544420b1a5f88cc0daab42aa7d2e"
Nov 01 03:53:03.028787 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.028769    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baec5da4-406d-4923-aa69-69820f5e04c0-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:03.028865 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.028790    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kqp8\" (UniqueName: \"kubernetes.io/projected/baec5da4-406d-4923-aa69-69820f5e04c0-kube-api-access-9kqp8\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:03.028865 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.028801    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baec5da4-406d-4923-aa69-69820f5e04c0-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:03.028865 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.028810    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baec5da4-406d-4923-aa69-69820f5e04c0-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:03.028865 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.028818    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baec5da4-406d-4923-aa69-69820f5e04c0-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:03.028865 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.028827    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baec5da4-406d-4923-aa69-69820f5e04c0-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:03.028865 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.028835    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baec5da4-406d-4923-aa69-69820f5e04c0-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:03.028865 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.028842    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baec5da4-406d-4923-aa69-69820f5e04c0-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:03.028865 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.028853    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baec5da4-406d-4923-aa69-69820f5e04c0-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:03.035262 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.035246    4930 scope.go:122] "RemoveContainer" containerID="7e5397471b831ab3918ce97d528cff62e209f15466024e7a6e29760e8ab7ce3a"
Nov 01 03:53:03.035509 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:53:03.035493    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e5397471b831ab3918ce97d528cff62e209f15466024e7a6e29760e8ab7ce3a\": container with ID starting with 7e5397471b831ab3918ce97d528cff62e209f15466024e7a6e29760e8ab7ce3a not found: ID does not exist" containerID="7e5397471b831ab3918ce97d528cff62e209f15466024e7a6e29760e8ab7ce3a"
Nov 01 03:53:03.035569 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.035515    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e5397471b831ab3918ce97d528cff62e209f15466024e7a6e29760e8ab7ce3a"} err="failed to get container status \"7e5397471b831ab3918ce97d528cff62e209f15466024e7a6e29760e8ab7ce3a\": rpc error: code = NotFound desc = could not find container \"7e5397471b831ab3918ce97d528cff62e209f15466024e7a6e29760e8ab7ce3a\": container with ID starting with 7e5397471b831ab3918ce97d528cff62e209f15466024e7a6e29760e8ab7ce3a not found: ID does not exist"
Nov 01 03:53:03.035569 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.035532    4930 scope.go:122] "RemoveContainer" containerID="231cb1e1808fd5e653cc3939494ddc5f5c5e544420b1a5f88cc0daab42aa7d2e"
Nov 01 03:53:03.035789 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:53:03.035771    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"231cb1e1808fd5e653cc3939494ddc5f5c5e544420b1a5f88cc0daab42aa7d2e\": container with ID starting with 231cb1e1808fd5e653cc3939494ddc5f5c5e544420b1a5f88cc0daab42aa7d2e not found: ID does not exist" containerID="231cb1e1808fd5e653cc3939494ddc5f5c5e544420b1a5f88cc0daab42aa7d2e"
Nov 01 03:53:03.035834 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.035798    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"231cb1e1808fd5e653cc3939494ddc5f5c5e544420b1a5f88cc0daab42aa7d2e"} err="failed to get container status \"231cb1e1808fd5e653cc3939494ddc5f5c5e544420b1a5f88cc0daab42aa7d2e\": rpc error: code = NotFound desc = could not find container \"231cb1e1808fd5e653cc3939494ddc5f5c5e544420b1a5f88cc0daab42aa7d2e\": container with ID starting with 231cb1e1808fd5e653cc3939494ddc5f5c5e544420b1a5f88cc0daab42aa7d2e not found: ID does not exist"
Nov 01 03:53:03.195930 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.195895    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"]
Nov 01 03:53:03.225020 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.224990    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-7cd000-openshift-default-69db569f5-nmbg8"]
Nov 01 03:53:03.419618 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.419529    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:03.620070 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.620036    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baec5da4-406d-4923-aa69-69820f5e04c0" path="/var/lib/kubelet/pods/baec5da4-406d-4923-aa69-69820f5e04c0/volumes"
Nov 01 03:53:03.799713 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.799635    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c22dd8-gq8v5"]
Nov 01 03:53:03.800123 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.800107    4930 cpu_manager.go:414] "containerMap: removing container" podUID="baec5da4-406d-4923-aa69-69820f5e04c0" containerName="istio-proxy"
Nov 01 03:53:03.800166 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.800125    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baec5da4-406d-4923-aa69-69820f5e04c0" containerName="istio-proxy"
Nov 01 03:53:03.800166 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.800152    4930 cpu_manager.go:414] "containerMap: removing container" podUID="b73f5e95-80e1-44ad-9bd8-39e111f98fca" containerName="revoke-keys"
Nov 01 03:53:03.800166 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.800158    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b73f5e95-80e1-44ad-9bd8-39e111f98fca" containerName="revoke-keys"
Nov 01 03:53:03.800268 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.800248    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="baec5da4-406d-4923-aa69-69820f5e04c0" containerName="istio-proxy"
Nov 01 03:53:03.800268 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.800261    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="baec5da4-406d-4923-aa69-69820f5e04c0" containerName="istio-proxy"
Nov 01 03:53:03.800268 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.800268    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="b73f5e95-80e1-44ad-9bd8-39e111f98fca" containerName="revoke-keys"
Nov 01 03:53:03.805159 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.805136    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c22dd8-gq8v5"
Nov 01 03:53:03.837011 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.836977    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eabff457-a65e-48de-8a76-214dea2161ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-c22dd8-gq8v5\" (UID: \"eabff457-a65e-48de-8a76-214dea2161ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c22dd8-gq8v5"
Nov 01 03:53:03.837155 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.837101    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzllt\" (UniqueName: \"kubernetes.io/projected/eabff457-a65e-48de-8a76-214dea2161ae-kube-api-access-jzllt\") pod \"maas-api-revoke-keys-e2e-worker-w5-c22dd8-gq8v5\" (UID: \"eabff457-a65e-48de-8a76-214dea2161ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c22dd8-gq8v5"
Nov 01 03:53:03.837205 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.837171    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eabff457-a65e-48de-8a76-214dea2161ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-c22dd8-gq8v5\" (UID: \"eabff457-a65e-48de-8a76-214dea2161ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c22dd8-gq8v5"
Nov 01 03:53:03.860095 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.860066    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:53:03.860296 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.860073    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:53:03.860296 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.860124    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:53:03.860296 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.860135    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:53:03.863957 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.863929    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c22dd8-gq8v5"]
Nov 01 03:53:03.938312 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.938282    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eabff457-a65e-48de-8a76-214dea2161ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-c22dd8-gq8v5\" (UID: \"eabff457-a65e-48de-8a76-214dea2161ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c22dd8-gq8v5"
Nov 01 03:53:03.938474 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.938353    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eabff457-a65e-48de-8a76-214dea2161ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-c22dd8-gq8v5\" (UID: \"eabff457-a65e-48de-8a76-214dea2161ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c22dd8-gq8v5"
Nov 01 03:53:03.938474 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.938390    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzllt\" (UniqueName: \"kubernetes.io/projected/eabff457-a65e-48de-8a76-214dea2161ae-kube-api-access-jzllt\") pod \"maas-api-revoke-keys-e2e-worker-w5-c22dd8-gq8v5\" (UID: \"eabff457-a65e-48de-8a76-214dea2161ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c22dd8-gq8v5"
Nov 01 03:53:03.939102 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.939078    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eabff457-a65e-48de-8a76-214dea2161ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-c22dd8-gq8v5\" (UID: \"eabff457-a65e-48de-8a76-214dea2161ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c22dd8-gq8v5"
Nov 01 03:53:03.940600 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.940577    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eabff457-a65e-48de-8a76-214dea2161ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-c22dd8-gq8v5\" (UID: \"eabff457-a65e-48de-8a76-214dea2161ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c22dd8-gq8v5"
Nov 01 03:53:03.999257 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:03.999223    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzllt\" (UniqueName: \"kubernetes.io/projected/eabff457-a65e-48de-8a76-214dea2161ae-kube-api-access-jzllt\") pod \"maas-api-revoke-keys-e2e-worker-w5-c22dd8-gq8v5\" (UID: \"eabff457-a65e-48de-8a76-214dea2161ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c22dd8-gq8v5"
Nov 01 03:53:04.121681 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:04.121652    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c22dd8-gq8v5"
Nov 01 03:53:04.501523 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:04.501492    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c22dd8-gq8v5"]
Nov 01 03:53:04.502260 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:53:04.502230    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeabff457_a65e_48de_8a76_214dea2161ae.slice/crio-377c7044f2df3de7b994eac7e816b0e1a8731099601eed178592a7cbc08e92be WatchSource:0}: Error finding container 377c7044f2df3de7b994eac7e816b0e1a8731099601eed178592a7cbc08e92be: Status 404 returned error can't find the container with id 377c7044f2df3de7b994eac7e816b0e1a8731099601eed178592a7cbc08e92be
Nov 01 03:53:05.031382 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:05.031289    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c22dd8-gq8v5" event={"ID":"eabff457-a65e-48de-8a76-214dea2161ae","Type":"ContainerStarted","Data":"179c9fa30c155cffa31bc0240c1a0dbffdc44a505a2911082ce33e4a4f650a5c"}
Nov 01 03:53:05.031382 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:05.031341    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c22dd8-gq8v5" event={"ID":"eabff457-a65e-48de-8a76-214dea2161ae","Type":"ContainerStarted","Data":"377c7044f2df3de7b994eac7e816b0e1a8731099601eed178592a7cbc08e92be"}
Nov 01 03:53:05.149785 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:05.149730    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c22dd8-gq8v5" podStartSLOduration=2.14971637 podStartE2EDuration="2.14971637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:53:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:53:05.14831221 +0000 UTC m=+2789.671318799" watchObservedRunningTime="2026-11-01 03:53:05.14971637 +0000 UTC m=+2789.672722958"
Nov 01 03:53:05.885498 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:05.885464    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:06.038904 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:06.038870    4930 generic.go:356] "Generic (PLEG): container finished" podID="eabff457-a65e-48de-8a76-214dea2161ae" containerID="179c9fa30c155cffa31bc0240c1a0dbffdc44a505a2911082ce33e4a4f650a5c" exitCode=0
Nov 01 03:53:06.039087 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:06.038954    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c22dd8-gq8v5" event={"ID":"eabff457-a65e-48de-8a76-214dea2161ae","Type":"ContainerDied","Data":"179c9fa30c155cffa31bc0240c1a0dbffdc44a505a2911082ce33e4a4f650a5c"}
Nov 01 03:53:07.188546 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:07.188519    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c22dd8-gq8v5"
Nov 01 03:53:07.271840 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:07.271812    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eabff457-a65e-48de-8a76-214dea2161ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eabff457-a65e-48de-8a76-214dea2161ae-maas-api-xks-ca\") pod \"eabff457-a65e-48de-8a76-214dea2161ae\" (UID: \"eabff457-a65e-48de-8a76-214dea2161ae\") "
Nov 01 03:53:07.271994 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:07.271914    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eabff457-a65e-48de-8a76-214dea2161ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eabff457-a65e-48de-8a76-214dea2161ae-maas-api-openshift-ca\") pod \"eabff457-a65e-48de-8a76-214dea2161ae\" (UID: \"eabff457-a65e-48de-8a76-214dea2161ae\") "
Nov 01 03:53:07.271994 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:07.271948    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eabff457-a65e-48de-8a76-214dea2161ae-kube-api-access-jzllt\" (UniqueName: \"kubernetes.io/projected/eabff457-a65e-48de-8a76-214dea2161ae-kube-api-access-jzllt\") pod \"eabff457-a65e-48de-8a76-214dea2161ae\" (UID: \"eabff457-a65e-48de-8a76-214dea2161ae\") "
Nov 01 03:53:07.272251 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:07.272230    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eabff457-a65e-48de-8a76-214dea2161ae-maas-api-openshift-ca" pod "eabff457-a65e-48de-8a76-214dea2161ae" (UID: "eabff457-a65e-48de-8a76-214dea2161ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:53:07.273956 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:07.273929    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eabff457-a65e-48de-8a76-214dea2161ae-kube-api-access-jzllt" pod "eabff457-a65e-48de-8a76-214dea2161ae" (UID: "eabff457-a65e-48de-8a76-214dea2161ae"). InnerVolumeSpecName "kube-api-access-jzllt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:53:07.274101 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:07.274079    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eabff457-a65e-48de-8a76-214dea2161ae-maas-api-xks-ca" pod "eabff457-a65e-48de-8a76-214dea2161ae" (UID: "eabff457-a65e-48de-8a76-214dea2161ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:53:07.374168 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:07.374132    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eabff457-a65e-48de-8a76-214dea2161ae-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:07.374318 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:07.374181    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eabff457-a65e-48de-8a76-214dea2161ae-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:07.374318 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:07.374194    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzllt\" (UniqueName: \"kubernetes.io/projected/eabff457-a65e-48de-8a76-214dea2161ae-kube-api-access-jzllt\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:08.052069 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:08.051976    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c22dd8-gq8v5"
Nov 01 03:53:08.052069 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:08.052003    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c22dd8-gq8v5" event={"ID":"eabff457-a65e-48de-8a76-214dea2161ae","Type":"ContainerDied","Data":"377c7044f2df3de7b994eac7e816b0e1a8731099601eed178592a7cbc08e92be"}
Nov 01 03:53:08.052069 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:08.052060    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="377c7044f2df3de7b994eac7e816b0e1a8731099601eed178592a7cbc08e92be"
Nov 01 03:53:08.280106 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:08.280074    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"]
Nov 01 03:53:08.310443 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:08.310349    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"]
Nov 01 03:53:08.441072 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:08.441033    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:10.392659 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:10.392525    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:10.551196 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:10.551158    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:11.728578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:11.728516    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:12.016152 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:12.016071    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:12.290585 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:12.290482    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:12.533686 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:12.533650    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ded2d7-csmzf"]
Nov 01 03:53:12.584178 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:12.584121    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ded2d7-csmzf"]
Nov 01 03:53:12.904452 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:12.904363    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:13.619530 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:13.619499    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dcac01b-ed97-47ca-aa26-ed00285418bb" path="/var/lib/kubelet/pods/3dcac01b-ed97-47ca-aa26-ed00285418bb/volumes"
Nov 01 03:53:13.741937 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:13.741900    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:15.687202 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:15.687166    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:15.863442 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:15.863406    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:18.863277 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:18.863238    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:19.018784 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:19.018735    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:19.121313 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:53:19.121233    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:53:19.188040 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:19.188000    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:19.380233 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:19.380151    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:23.194800 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.194764    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f1690f3-9r4sv"]
Nov 01 03:53:23.197373 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.195197    4930 cpu_manager.go:414] "containerMap: removing container" podUID="baec5da4-406d-4923-aa69-69820f5e04c0" containerName="istio-proxy"
Nov 01 03:53:23.197373 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.195210    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baec5da4-406d-4923-aa69-69820f5e04c0" containerName="istio-proxy"
Nov 01 03:53:23.197373 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.195234    4930 cpu_manager.go:414] "containerMap: removing container" podUID="eabff457-a65e-48de-8a76-214dea2161ae" containerName="revoke-keys"
Nov 01 03:53:23.197373 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.195239    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eabff457-a65e-48de-8a76-214dea2161ae" containerName="revoke-keys"
Nov 01 03:53:23.197373 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.195315    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="eabff457-a65e-48de-8a76-214dea2161ae" containerName="revoke-keys"
Nov 01 03:53:23.198766 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.198744    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1690f3-9r4sv"
Nov 01 03:53:23.254721 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.254634    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:53:23.300865 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.300802    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2" podUID="e2483a06-2f4b-4a1e-af4e-a63eb50d29bf" containerName="main" containerID="cri-o://3dbdeb37dc708acde94d3845b87620f01fe594185a9c10e2eea1d6721e1c535c" gracePeriod=45
Nov 01 03:53:23.304008 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.303977    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:53:23.307329 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.307305    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f1690f3-9r4sv"]
Nov 01 03:53:23.315001 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.314976    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:53:23.315206 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.315188    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:53:23.330916 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.330889    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw" podUID="b6d688b8-b911-4a6a-a842-2b0fb7a75e83" containerName="main" containerID="cri-o://b4f6bd1a6e51c6e7806faf07477a12394b9750967483c5ddfae03692cfeda71e" gracePeriod=45
Nov 01 03:53:23.360572 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.360533    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22002d07-ffff-4f2f-b9d7-2751eae471d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f1690f3-9r4sv\" (UID: \"22002d07-ffff-4f2f-b9d7-2751eae471d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1690f3-9r4sv"
Nov 01 03:53:23.360682 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.360632    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22002d07-ffff-4f2f-b9d7-2751eae471d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f1690f3-9r4sv\" (UID: \"22002d07-ffff-4f2f-b9d7-2751eae471d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1690f3-9r4sv"
Nov 01 03:53:23.360751 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.360702    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5wh9\" (UniqueName: \"kubernetes.io/projected/22002d07-ffff-4f2f-b9d7-2751eae471d1-kube-api-access-g5wh9\") pod \"maas-api-revoke-sub-0f1690f3-9r4sv\" (UID: \"22002d07-ffff-4f2f-b9d7-2751eae471d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1690f3-9r4sv"
Nov 01 03:53:23.462304 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.462228    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22002d07-ffff-4f2f-b9d7-2751eae471d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f1690f3-9r4sv\" (UID: \"22002d07-ffff-4f2f-b9d7-2751eae471d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1690f3-9r4sv"
Nov 01 03:53:23.462304 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.462280    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5wh9\" (UniqueName: \"kubernetes.io/projected/22002d07-ffff-4f2f-b9d7-2751eae471d1-kube-api-access-g5wh9\") pod \"maas-api-revoke-sub-0f1690f3-9r4sv\" (UID: \"22002d07-ffff-4f2f-b9d7-2751eae471d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1690f3-9r4sv"
Nov 01 03:53:23.462528 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.462506    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22002d07-ffff-4f2f-b9d7-2751eae471d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f1690f3-9r4sv\" (UID: \"22002d07-ffff-4f2f-b9d7-2751eae471d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1690f3-9r4sv"
Nov 01 03:53:23.463780 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.463745    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22002d07-ffff-4f2f-b9d7-2751eae471d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f1690f3-9r4sv\" (UID: \"22002d07-ffff-4f2f-b9d7-2751eae471d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1690f3-9r4sv"
Nov 01 03:53:23.465259 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.465229    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22002d07-ffff-4f2f-b9d7-2751eae471d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f1690f3-9r4sv\" (UID: \"22002d07-ffff-4f2f-b9d7-2751eae471d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1690f3-9r4sv"
Nov 01 03:53:23.525798 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.525735    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5wh9\" (UniqueName: \"kubernetes.io/projected/22002d07-ffff-4f2f-b9d7-2751eae471d1-kube-api-access-g5wh9\") pod \"maas-api-revoke-sub-0f1690f3-9r4sv\" (UID: \"22002d07-ffff-4f2f-b9d7-2751eae471d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1690f3-9r4sv"
Nov 01 03:53:23.581696 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.581671    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"
Nov 01 03:53:23.612390 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.612362    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:53:23.765787 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.765710    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2483a06-2f4b-4a1e-af4e-a63eb50d29bf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2483a06-2f4b-4a1e-af4e-a63eb50d29bf-dshm\") pod \"e2483a06-2f4b-4a1e-af4e-a63eb50d29bf\" (UID: \"e2483a06-2f4b-4a1e-af4e-a63eb50d29bf\") "
Nov 01 03:53:23.765787 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.765751    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2483a06-2f4b-4a1e-af4e-a63eb50d29bf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2483a06-2f4b-4a1e-af4e-a63eb50d29bf-model-cache\") pod \"e2483a06-2f4b-4a1e-af4e-a63eb50d29bf\" (UID: \"e2483a06-2f4b-4a1e-af4e-a63eb50d29bf\") "
Nov 01 03:53:23.766019 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.765798    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6d688b8-b911-4a6a-a842-2b0fb7a75e83-kube-api-access-xltfx\" (UniqueName: \"kubernetes.io/projected/b6d688b8-b911-4a6a-a842-2b0fb7a75e83-kube-api-access-xltfx\") pod \"b6d688b8-b911-4a6a-a842-2b0fb7a75e83\" (UID: \"b6d688b8-b911-4a6a-a842-2b0fb7a75e83\") "
Nov 01 03:53:23.766019 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.765834    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6d688b8-b911-4a6a-a842-2b0fb7a75e83-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6d688b8-b911-4a6a-a842-2b0fb7a75e83-model-cache\") pod \"b6d688b8-b911-4a6a-a842-2b0fb7a75e83\" (UID: \"b6d688b8-b911-4a6a-a842-2b0fb7a75e83\") "
Nov 01 03:53:23.766019 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.765883    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6d688b8-b911-4a6a-a842-2b0fb7a75e83-tls-certs\" (UniqueName: \"kubernetes.io/secret/b6d688b8-b911-4a6a-a842-2b0fb7a75e83-tls-certs\") pod \"b6d688b8-b911-4a6a-a842-2b0fb7a75e83\" (UID: \"b6d688b8-b911-4a6a-a842-2b0fb7a75e83\") "
Nov 01 03:53:23.766019 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.765951    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6d688b8-b911-4a6a-a842-2b0fb7a75e83-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6d688b8-b911-4a6a-a842-2b0fb7a75e83-tmp-dir\") pod \"b6d688b8-b911-4a6a-a842-2b0fb7a75e83\" (UID: \"b6d688b8-b911-4a6a-a842-2b0fb7a75e83\") "
Nov 01 03:53:23.766019 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.765977    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2483a06-2f4b-4a1e-af4e-a63eb50d29bf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2483a06-2f4b-4a1e-af4e-a63eb50d29bf-tmp-dir\") pod \"e2483a06-2f4b-4a1e-af4e-a63eb50d29bf\" (UID: \"e2483a06-2f4b-4a1e-af4e-a63eb50d29bf\") "
Nov 01 03:53:23.766303 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.766190    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2483a06-2f4b-4a1e-af4e-a63eb50d29bf-tls-certs\" (UniqueName: \"kubernetes.io/secret/e2483a06-2f4b-4a1e-af4e-a63eb50d29bf-tls-certs\") pod \"e2483a06-2f4b-4a1e-af4e-a63eb50d29bf\" (UID: \"e2483a06-2f4b-4a1e-af4e-a63eb50d29bf\") "
Nov 01 03:53:23.766367 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.766300    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6d688b8-b911-4a6a-a842-2b0fb7a75e83-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6d688b8-b911-4a6a-a842-2b0fb7a75e83-dshm\") pod \"b6d688b8-b911-4a6a-a842-2b0fb7a75e83\" (UID: \"b6d688b8-b911-4a6a-a842-2b0fb7a75e83\") "
Nov 01 03:53:23.766367 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.766356    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6d688b8-b911-4a6a-a842-2b0fb7a75e83-home\" (UniqueName: \"kubernetes.io/empty-dir/b6d688b8-b911-4a6a-a842-2b0fb7a75e83-home\") pod \"b6d688b8-b911-4a6a-a842-2b0fb7a75e83\" (UID: \"b6d688b8-b911-4a6a-a842-2b0fb7a75e83\") "
Nov 01 03:53:23.766482 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.766398    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2483a06-2f4b-4a1e-af4e-a63eb50d29bf-kube-api-access-8qtlb\" (UniqueName: \"kubernetes.io/projected/e2483a06-2f4b-4a1e-af4e-a63eb50d29bf-kube-api-access-8qtlb\") pod \"e2483a06-2f4b-4a1e-af4e-a63eb50d29bf\" (UID: \"e2483a06-2f4b-4a1e-af4e-a63eb50d29bf\") "
Nov 01 03:53:23.766482 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.766452    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2483a06-2f4b-4a1e-af4e-a63eb50d29bf-home\" (UniqueName: \"kubernetes.io/empty-dir/e2483a06-2f4b-4a1e-af4e-a63eb50d29bf-home\") pod \"e2483a06-2f4b-4a1e-af4e-a63eb50d29bf\" (UID: \"e2483a06-2f4b-4a1e-af4e-a63eb50d29bf\") "
Nov 01 03:53:23.767303 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.766906    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2483a06-2f4b-4a1e-af4e-a63eb50d29bf-model-cache" pod "e2483a06-2f4b-4a1e-af4e-a63eb50d29bf" (UID: "e2483a06-2f4b-4a1e-af4e-a63eb50d29bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:23.767303 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.766932    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6d688b8-b911-4a6a-a842-2b0fb7a75e83-model-cache" pod "b6d688b8-b911-4a6a-a842-2b0fb7a75e83" (UID: "b6d688b8-b911-4a6a-a842-2b0fb7a75e83"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:23.767303 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.767059    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2483a06-2f4b-4a1e-af4e-a63eb50d29bf-tmp-dir" pod "e2483a06-2f4b-4a1e-af4e-a63eb50d29bf" (UID: "e2483a06-2f4b-4a1e-af4e-a63eb50d29bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:23.767303 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.767158    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6d688b8-b911-4a6a-a842-2b0fb7a75e83-tmp-dir" pod "b6d688b8-b911-4a6a-a842-2b0fb7a75e83" (UID: "b6d688b8-b911-4a6a-a842-2b0fb7a75e83"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:23.767303 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.767256    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2483a06-2f4b-4a1e-af4e-a63eb50d29bf-home" pod "e2483a06-2f4b-4a1e-af4e-a63eb50d29bf" (UID: "e2483a06-2f4b-4a1e-af4e-a63eb50d29bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:23.767303 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.767266    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6d688b8-b911-4a6a-a842-2b0fb7a75e83-home" pod "b6d688b8-b911-4a6a-a842-2b0fb7a75e83" (UID: "b6d688b8-b911-4a6a-a842-2b0fb7a75e83"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:23.768541 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.768511    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2483a06-2f4b-4a1e-af4e-a63eb50d29bf-dshm" pod "e2483a06-2f4b-4a1e-af4e-a63eb50d29bf" (UID: "e2483a06-2f4b-4a1e-af4e-a63eb50d29bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:23.768680 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.768609    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6d688b8-b911-4a6a-a842-2b0fb7a75e83-kube-api-access-xltfx" pod "b6d688b8-b911-4a6a-a842-2b0fb7a75e83" (UID: "b6d688b8-b911-4a6a-a842-2b0fb7a75e83"). InnerVolumeSpecName "kube-api-access-xltfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:53:23.769035 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.769009    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6d688b8-b911-4a6a-a842-2b0fb7a75e83-dshm" pod "b6d688b8-b911-4a6a-a842-2b0fb7a75e83" (UID: "b6d688b8-b911-4a6a-a842-2b0fb7a75e83"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:23.769603 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.769580    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2483a06-2f4b-4a1e-af4e-a63eb50d29bf-tls-certs" pod "e2483a06-2f4b-4a1e-af4e-a63eb50d29bf" (UID: "e2483a06-2f4b-4a1e-af4e-a63eb50d29bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:53:23.769758 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.769730    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6d688b8-b911-4a6a-a842-2b0fb7a75e83-tls-certs" pod "b6d688b8-b911-4a6a-a842-2b0fb7a75e83" (UID: "b6d688b8-b911-4a6a-a842-2b0fb7a75e83"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:53:23.769850 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.769763    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2483a06-2f4b-4a1e-af4e-a63eb50d29bf-kube-api-access-8qtlb" pod "e2483a06-2f4b-4a1e-af4e-a63eb50d29bf" (UID: "e2483a06-2f4b-4a1e-af4e-a63eb50d29bf"). InnerVolumeSpecName "kube-api-access-8qtlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:53:23.815745 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.815714    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1690f3-9r4sv"
Nov 01 03:53:23.868811 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.868770    4930 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6d688b8-b911-4a6a-a842-2b0fb7a75e83-dshm\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:23.868811 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.868804    4930 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6d688b8-b911-4a6a-a842-2b0fb7a75e83-home\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:23.869011 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.868820    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qtlb\" (UniqueName: \"kubernetes.io/projected/e2483a06-2f4b-4a1e-af4e-a63eb50d29bf-kube-api-access-8qtlb\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:23.869011 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.868837    4930 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2483a06-2f4b-4a1e-af4e-a63eb50d29bf-home\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:23.869011 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.868852    4930 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2483a06-2f4b-4a1e-af4e-a63eb50d29bf-dshm\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:23.869011 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.868865    4930 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2483a06-2f4b-4a1e-af4e-a63eb50d29bf-model-cache\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:23.869011 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.868880    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xltfx\" (UniqueName: \"kubernetes.io/projected/b6d688b8-b911-4a6a-a842-2b0fb7a75e83-kube-api-access-xltfx\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:23.869011 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.868896    4930 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6d688b8-b911-4a6a-a842-2b0fb7a75e83-model-cache\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:23.869011 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.868906    4930 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6d688b8-b911-4a6a-a842-2b0fb7a75e83-tls-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:23.869011 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.868914    4930 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6d688b8-b911-4a6a-a842-2b0fb7a75e83-tmp-dir\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:23.869011 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.868922    4930 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2483a06-2f4b-4a1e-af4e-a63eb50d29bf-tmp-dir\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:23.869011 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:23.868929    4930 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2483a06-2f4b-4a1e-af4e-a63eb50d29bf-tls-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:24.188318 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:24.188281    4930 generic.go:356] "Generic (PLEG): container finished" podID="e2483a06-2f4b-4a1e-af4e-a63eb50d29bf" containerID="3dbdeb37dc708acde94d3845b87620f01fe594185a9c10e2eea1d6721e1c535c" exitCode=0
Nov 01 03:53:24.188529 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:24.188346    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"
Nov 01 03:53:24.188529 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:24.188360    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2" event={"ID":"e2483a06-2f4b-4a1e-af4e-a63eb50d29bf","Type":"ContainerDied","Data":"3dbdeb37dc708acde94d3845b87620f01fe594185a9c10e2eea1d6721e1c535c"}
Nov 01 03:53:24.188529 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:24.188415    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2" event={"ID":"e2483a06-2f4b-4a1e-af4e-a63eb50d29bf","Type":"ContainerDied","Data":"b6351225d45fceaa736af763b252889d2783faef265e49f2e9869686ddc2b791"}
Nov 01 03:53:24.188529 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:24.188438    4930 scope.go:122] "RemoveContainer" containerID="3dbdeb37dc708acde94d3845b87620f01fe594185a9c10e2eea1d6721e1c535c"
Nov 01 03:53:24.190207 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:24.190178    4930 generic.go:356] "Generic (PLEG): container finished" podID="b6d688b8-b911-4a6a-a842-2b0fb7a75e83" containerID="b4f6bd1a6e51c6e7806faf07477a12394b9750967483c5ddfae03692cfeda71e" exitCode=0
Nov 01 03:53:24.190518 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:24.190270    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw" event={"ID":"b6d688b8-b911-4a6a-a842-2b0fb7a75e83","Type":"ContainerDied","Data":"b4f6bd1a6e51c6e7806faf07477a12394b9750967483c5ddfae03692cfeda71e"}
Nov 01 03:53:24.190518 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:24.190300    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw" event={"ID":"b6d688b8-b911-4a6a-a842-2b0fb7a75e83","Type":"ContainerDied","Data":"6991638945d170b5ed0f0d57cfa48e5471551477e1c2c425411864f4e39acfa5"}
Nov 01 03:53:24.190518 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:24.190367    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"
Nov 01 03:53:24.201086 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:24.201069    4930 scope.go:122] "RemoveContainer" containerID="3dbdeb37dc708acde94d3845b87620f01fe594185a9c10e2eea1d6721e1c535c"
Nov 01 03:53:24.201410 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:53:24.201393    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dbdeb37dc708acde94d3845b87620f01fe594185a9c10e2eea1d6721e1c535c\": container with ID starting with 3dbdeb37dc708acde94d3845b87620f01fe594185a9c10e2eea1d6721e1c535c not found: ID does not exist" containerID="3dbdeb37dc708acde94d3845b87620f01fe594185a9c10e2eea1d6721e1c535c"
Nov 01 03:53:24.201457 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:24.201419    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dbdeb37dc708acde94d3845b87620f01fe594185a9c10e2eea1d6721e1c535c"} err="failed to get container status \"3dbdeb37dc708acde94d3845b87620f01fe594185a9c10e2eea1d6721e1c535c\": rpc error: code = NotFound desc = could not find container \"3dbdeb37dc708acde94d3845b87620f01fe594185a9c10e2eea1d6721e1c535c\": container with ID starting with 3dbdeb37dc708acde94d3845b87620f01fe594185a9c10e2eea1d6721e1c535c not found: ID does not exist"
Nov 01 03:53:24.201457 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:24.201435    4930 scope.go:122] "RemoveContainer" containerID="b4f6bd1a6e51c6e7806faf07477a12394b9750967483c5ddfae03692cfeda71e"
Nov 01 03:53:24.208328 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:24.208306    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f1690f3-9r4sv"]
Nov 01 03:53:24.209442 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:53:24.209420    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22002d07_ffff_4f2f_b9d7_2751eae471d1.slice/crio-3ec2171757798ed5b3581797a8147e2cf1bc02d1ec933e13f35334eef1e12535 WatchSource:0}: Error finding container 3ec2171757798ed5b3581797a8147e2cf1bc02d1ec933e13f35334eef1e12535: Status 404 returned error can't find the container with id 3ec2171757798ed5b3581797a8147e2cf1bc02d1ec933e13f35334eef1e12535
Nov 01 03:53:24.213037 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:24.213020    4930 scope.go:122] "RemoveContainer" containerID="b4f6bd1a6e51c6e7806faf07477a12394b9750967483c5ddfae03692cfeda71e"
Nov 01 03:53:24.213345 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:53:24.213271    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4f6bd1a6e51c6e7806faf07477a12394b9750967483c5ddfae03692cfeda71e\": container with ID starting with b4f6bd1a6e51c6e7806faf07477a12394b9750967483c5ddfae03692cfeda71e not found: ID does not exist" containerID="b4f6bd1a6e51c6e7806faf07477a12394b9750967483c5ddfae03692cfeda71e"
Nov 01 03:53:24.213345 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:24.213293    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4f6bd1a6e51c6e7806faf07477a12394b9750967483c5ddfae03692cfeda71e"} err="failed to get container status \"b4f6bd1a6e51c6e7806faf07477a12394b9750967483c5ddfae03692cfeda71e\": rpc error: code = NotFound desc = could not find container \"b4f6bd1a6e51c6e7806faf07477a12394b9750967483c5ddfae03692cfeda71e\": container with ID starting with b4f6bd1a6e51c6e7806faf07477a12394b9750967483c5ddfae03692cfeda71e not found: ID does not exist"
Nov 01 03:53:24.357368 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:24.357336    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"]
Nov 01 03:53:24.362347 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:24.362316    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-c22dd8/simulator-w5-c22dd8-kserve-66547dfc77-8bhmw"]
Nov 01 03:53:24.476065 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:24.476040    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"]
Nov 01 03:53:24.483849 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:24.483823    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-c22dd8/distinct-2-w5-c22dd8-kserve-694dff9d68-pf5n2"]
Nov 01 03:53:24.825244 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:24.825164    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:25.043000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:25.042972    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:25.198737 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:25.198705    4930 generic.go:356] "Generic (PLEG): container finished" podID="22002d07-ffff-4f2f-b9d7-2751eae471d1" containerID="369b41775f0bb02752a5a9ec10fc5de9f01b28fe0271d74fba154f45b5e73f41" exitCode=0
Nov 01 03:53:25.198961 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:25.198793    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1690f3-9r4sv" event={"ID":"22002d07-ffff-4f2f-b9d7-2751eae471d1","Type":"ContainerDied","Data":"369b41775f0bb02752a5a9ec10fc5de9f01b28fe0271d74fba154f45b5e73f41"}
Nov 01 03:53:25.198961 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:25.198854    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1690f3-9r4sv" event={"ID":"22002d07-ffff-4f2f-b9d7-2751eae471d1","Type":"ContainerStarted","Data":"3ec2171757798ed5b3581797a8147e2cf1bc02d1ec933e13f35334eef1e12535"}
Nov 01 03:53:25.200635 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:25.200599    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:25.324405 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:25.324337    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-211380-l6w9l"]
Nov 01 03:53:25.366192 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:25.366163    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-211380-l6w9l"]
Nov 01 03:53:25.619470 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:25.619437    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6d688b8-b911-4a6a-a842-2b0fb7a75e83" path="/var/lib/kubelet/pods/b6d688b8-b911-4a6a-a842-2b0fb7a75e83/volumes"
Nov 01 03:53:25.619884 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:25.619868    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b73f5e95-80e1-44ad-9bd8-39e111f98fca" path="/var/lib/kubelet/pods/b73f5e95-80e1-44ad-9bd8-39e111f98fca/volumes"
Nov 01 03:53:25.620249 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:25.620235    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2483a06-2f4b-4a1e-af4e-a63eb50d29bf" path="/var/lib/kubelet/pods/e2483a06-2f4b-4a1e-af4e-a63eb50d29bf/volumes"
Nov 01 03:53:25.764064 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:25.764016    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:26.052375 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:26.052288    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfd0f261-4s25s"]
Nov 01 03:53:26.056144 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:26.056115    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfd0f261-4s25s"]
Nov 01 03:53:26.348857 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:26.348830    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1690f3-9r4sv"
Nov 01 03:53:26.497542 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:26.497508    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22002d07-ffff-4f2f-b9d7-2751eae471d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22002d07-ffff-4f2f-b9d7-2751eae471d1-maas-api-openshift-ca\") pod \"22002d07-ffff-4f2f-b9d7-2751eae471d1\" (UID: \"22002d07-ffff-4f2f-b9d7-2751eae471d1\") "
Nov 01 03:53:26.497730 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:26.497570    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22002d07-ffff-4f2f-b9d7-2751eae471d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22002d07-ffff-4f2f-b9d7-2751eae471d1-maas-api-xks-ca\") pod \"22002d07-ffff-4f2f-b9d7-2751eae471d1\" (UID: \"22002d07-ffff-4f2f-b9d7-2751eae471d1\") "
Nov 01 03:53:26.497730 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:26.497651    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22002d07-ffff-4f2f-b9d7-2751eae471d1-kube-api-access-g5wh9\" (UniqueName: \"kubernetes.io/projected/22002d07-ffff-4f2f-b9d7-2751eae471d1-kube-api-access-g5wh9\") pod \"22002d07-ffff-4f2f-b9d7-2751eae471d1\" (UID: \"22002d07-ffff-4f2f-b9d7-2751eae471d1\") "
Nov 01 03:53:26.497939 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:26.497917    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22002d07-ffff-4f2f-b9d7-2751eae471d1-maas-api-openshift-ca" pod "22002d07-ffff-4f2f-b9d7-2751eae471d1" (UID: "22002d07-ffff-4f2f-b9d7-2751eae471d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:53:26.498061 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:26.498044    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22002d07-ffff-4f2f-b9d7-2751eae471d1-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:26.499996 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:26.499959    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22002d07-ffff-4f2f-b9d7-2751eae471d1-kube-api-access-g5wh9" pod "22002d07-ffff-4f2f-b9d7-2751eae471d1" (UID: "22002d07-ffff-4f2f-b9d7-2751eae471d1"). InnerVolumeSpecName "kube-api-access-g5wh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:53:26.500128 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:26.500094    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22002d07-ffff-4f2f-b9d7-2751eae471d1-maas-api-xks-ca" pod "22002d07-ffff-4f2f-b9d7-2751eae471d1" (UID: "22002d07-ffff-4f2f-b9d7-2751eae471d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:53:26.599453 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:26.599418    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22002d07-ffff-4f2f-b9d7-2751eae471d1-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:26.599453 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:26.599448    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5wh9\" (UniqueName: \"kubernetes.io/projected/22002d07-ffff-4f2f-b9d7-2751eae471d1-kube-api-access-g5wh9\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:27.216339 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:27.216306    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1690f3-9r4sv" event={"ID":"22002d07-ffff-4f2f-b9d7-2751eae471d1","Type":"ContainerDied","Data":"3ec2171757798ed5b3581797a8147e2cf1bc02d1ec933e13f35334eef1e12535"}
Nov 01 03:53:27.216513 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:27.216353    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ec2171757798ed5b3581797a8147e2cf1bc02d1ec933e13f35334eef1e12535"
Nov 01 03:53:27.216513 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:27.216353    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1690f3-9r4sv"
Nov 01 03:53:27.620634 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:27.620595    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28d34d07-bfbd-4231-869b-e7ebf0519d02" path="/var/lib/kubelet/pods/28d34d07-bfbd-4231-869b-e7ebf0519d02/volumes"
Nov 01 03:53:30.214506 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:30.214415    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:32.425395 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:32.425357    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:32.579089 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:32.579051    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:32.770680 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:32.770590    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"]
Nov 01 03:53:32.770846 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:32.770821    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb" podUID="b72d8427-c8a5-4d67-be83-57a2f54dddb4" containerName="istio-proxy" containerID="cri-o://869a7fc0c617e8e088d899f5a9ad9ec5551dbeacd8a4f27d810b9993c32857a4" gracePeriod=30
Nov 01 03:53:32.776373 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:32.776347    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:33.258583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:33.258519    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:34.655430 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:34.655396    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:35.669719 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:35.669684    4930 patch_prober.go:29] interesting pod/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:53:35.670191 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:35.669785    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb" podUID="b72d8427-c8a5-4d67-be83-57a2f54dddb4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:53:36.268490 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:36.268458    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:36.495199 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:36.495159    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:36.662258 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:36.662222    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:36.770356 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:36.770322    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c22dd8-gq8v5"]
Nov 01 03:53:36.819414 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:36.819387    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-c22dd8-gq8v5"]
Nov 01 03:53:37.203029 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:37.202989    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:37.619335 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:37.619304    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eabff457-a65e-48de-8a76-214dea2161ae" path="/var/lib/kubelet/pods/eabff457-a65e-48de-8a76-214dea2161ae/volumes"
Nov 01 03:53:38.030361 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.030333    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"
Nov 01 03:53:38.150238 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.150155    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b72d8427-c8a5-4d67-be83-57a2f54dddb4-istio-token\" (UniqueName: \"kubernetes.io/projected/b72d8427-c8a5-4d67-be83-57a2f54dddb4-istio-token\") pod \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\" (UID: \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\") "
Nov 01 03:53:38.150410 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.150277    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b72d8427-c8a5-4d67-be83-57a2f54dddb4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b72d8427-c8a5-4d67-be83-57a2f54dddb4-istio-data\") pod \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\" (UID: \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\") "
Nov 01 03:53:38.150410 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.150314    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b72d8427-c8a5-4d67-be83-57a2f54dddb4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b72d8427-c8a5-4d67-be83-57a2f54dddb4-credential-socket\") pod \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\" (UID: \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\") "
Nov 01 03:53:38.150410 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.150353    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b72d8427-c8a5-4d67-be83-57a2f54dddb4-kube-api-access-fm68f\" (UniqueName: \"kubernetes.io/projected/b72d8427-c8a5-4d67-be83-57a2f54dddb4-kube-api-access-fm68f\") pod \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\" (UID: \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\") "
Nov 01 03:53:38.150410 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.150386    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b72d8427-c8a5-4d67-be83-57a2f54dddb4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b72d8427-c8a5-4d67-be83-57a2f54dddb4-istio-envoy\") pod \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\" (UID: \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\") "
Nov 01 03:53:38.150695 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.150430    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b72d8427-c8a5-4d67-be83-57a2f54dddb4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b72d8427-c8a5-4d67-be83-57a2f54dddb4-workload-socket\") pod \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\" (UID: \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\") "
Nov 01 03:53:38.150695 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.150486    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b72d8427-c8a5-4d67-be83-57a2f54dddb4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b72d8427-c8a5-4d67-be83-57a2f54dddb4-workload-certs\") pod \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\" (UID: \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\") "
Nov 01 03:53:38.150695 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.150538    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b72d8427-c8a5-4d67-be83-57a2f54dddb4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b72d8427-c8a5-4d67-be83-57a2f54dddb4-istio-podinfo\") pod \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\" (UID: \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\") "
Nov 01 03:53:38.150695 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.150596    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b72d8427-c8a5-4d67-be83-57a2f54dddb4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b72d8427-c8a5-4d67-be83-57a2f54dddb4-istiod-ca-cert\") pod \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\" (UID: \"b72d8427-c8a5-4d67-be83-57a2f54dddb4\") "
Nov 01 03:53:38.150695 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.150625    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b72d8427-c8a5-4d67-be83-57a2f54dddb4-istio-data" pod "b72d8427-c8a5-4d67-be83-57a2f54dddb4" (UID: "b72d8427-c8a5-4d67-be83-57a2f54dddb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:38.150967 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.150755    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b72d8427-c8a5-4d67-be83-57a2f54dddb4-credential-socket" pod "b72d8427-c8a5-4d67-be83-57a2f54dddb4" (UID: "b72d8427-c8a5-4d67-be83-57a2f54dddb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:38.150967 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.150768    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b72d8427-c8a5-4d67-be83-57a2f54dddb4-workload-socket" pod "b72d8427-c8a5-4d67-be83-57a2f54dddb4" (UID: "b72d8427-c8a5-4d67-be83-57a2f54dddb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:38.151091 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.151078    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b72d8427-c8a5-4d67-be83-57a2f54dddb4-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:38.151153 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.151099    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b72d8427-c8a5-4d67-be83-57a2f54dddb4-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:38.151153 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.151115    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b72d8427-c8a5-4d67-be83-57a2f54dddb4-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:38.151260 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.151195    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b72d8427-c8a5-4d67-be83-57a2f54dddb4-workload-certs" pod "b72d8427-c8a5-4d67-be83-57a2f54dddb4" (UID: "b72d8427-c8a5-4d67-be83-57a2f54dddb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:38.151260 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.151222    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b72d8427-c8a5-4d67-be83-57a2f54dddb4-istiod-ca-cert" pod "b72d8427-c8a5-4d67-be83-57a2f54dddb4" (UID: "b72d8427-c8a5-4d67-be83-57a2f54dddb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:53:38.152691 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.152660    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b72d8427-c8a5-4d67-be83-57a2f54dddb4-kube-api-access-fm68f" pod "b72d8427-c8a5-4d67-be83-57a2f54dddb4" (UID: "b72d8427-c8a5-4d67-be83-57a2f54dddb4"). InnerVolumeSpecName "kube-api-access-fm68f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:53:38.152804 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.152771    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b72d8427-c8a5-4d67-be83-57a2f54dddb4-istio-podinfo" pod "b72d8427-c8a5-4d67-be83-57a2f54dddb4" (UID: "b72d8427-c8a5-4d67-be83-57a2f54dddb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:53:38.152960 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.152942    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b72d8427-c8a5-4d67-be83-57a2f54dddb4-istio-token" pod "b72d8427-c8a5-4d67-be83-57a2f54dddb4" (UID: "b72d8427-c8a5-4d67-be83-57a2f54dddb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:53:38.153045 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.153027    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b72d8427-c8a5-4d67-be83-57a2f54dddb4-istio-envoy" pod "b72d8427-c8a5-4d67-be83-57a2f54dddb4" (UID: "b72d8427-c8a5-4d67-be83-57a2f54dddb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:38.252314 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.252283    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b72d8427-c8a5-4d67-be83-57a2f54dddb4-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:38.252314 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.252309    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b72d8427-c8a5-4d67-be83-57a2f54dddb4-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:38.252626 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.252323    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b72d8427-c8a5-4d67-be83-57a2f54dddb4-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:38.252626 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.252342    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b72d8427-c8a5-4d67-be83-57a2f54dddb4-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:38.252626 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.252353    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm68f\" (UniqueName: \"kubernetes.io/projected/b72d8427-c8a5-4d67-be83-57a2f54dddb4-kube-api-access-fm68f\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:38.252626 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.252363    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b72d8427-c8a5-4d67-be83-57a2f54dddb4-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:38.254628 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.254589    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"]
Nov 01 03:53:38.257665 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.257644    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"]
Nov 01 03:53:38.259932 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.259907    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"]
Nov 01 03:53:38.314408 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.314379    4930 generic.go:356] "Generic (PLEG): container finished" podID="b72d8427-c8a5-4d67-be83-57a2f54dddb4" containerID="869a7fc0c617e8e088d899f5a9ad9ec5551dbeacd8a4f27d810b9993c32857a4" exitCode=0
Nov 01 03:53:38.314599 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.314435    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb" event={"ID":"b72d8427-c8a5-4d67-be83-57a2f54dddb4","Type":"ContainerDied","Data":"869a7fc0c617e8e088d899f5a9ad9ec5551dbeacd8a4f27d810b9993c32857a4"}
Nov 01 03:53:38.314599 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.314451    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"
Nov 01 03:53:38.314599 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.314489    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb" event={"ID":"b72d8427-c8a5-4d67-be83-57a2f54dddb4","Type":"ContainerDied","Data":"6801c32b51e4f8dbf1a085acfa6f7b4b166134cdcd1e0cb53e1ebb37ab454064"}
Nov 01 03:53:38.314599 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.314513    4930 scope.go:122] "RemoveContainer" containerID="869a7fc0c617e8e088d899f5a9ad9ec5551dbeacd8a4f27d810b9993c32857a4"
Nov 01 03:53:38.325363 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.325342    4930 scope.go:122] "RemoveContainer" containerID="6b1f11a10a7deb1b747125dcba38e6a0e7b6b526073041c52e6abadd79caeeac"
Nov 01 03:53:38.334800 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.334784    4930 scope.go:122] "RemoveContainer" containerID="869a7fc0c617e8e088d899f5a9ad9ec5551dbeacd8a4f27d810b9993c32857a4"
Nov 01 03:53:38.335070 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:53:38.335049    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"869a7fc0c617e8e088d899f5a9ad9ec5551dbeacd8a4f27d810b9993c32857a4\": container with ID starting with 869a7fc0c617e8e088d899f5a9ad9ec5551dbeacd8a4f27d810b9993c32857a4 not found: ID does not exist" containerID="869a7fc0c617e8e088d899f5a9ad9ec5551dbeacd8a4f27d810b9993c32857a4"
Nov 01 03:53:38.335137 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.335079    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"869a7fc0c617e8e088d899f5a9ad9ec5551dbeacd8a4f27d810b9993c32857a4"} err="failed to get container status \"869a7fc0c617e8e088d899f5a9ad9ec5551dbeacd8a4f27d810b9993c32857a4\": rpc error: code = NotFound desc = could not find container \"869a7fc0c617e8e088d899f5a9ad9ec5551dbeacd8a4f27d810b9993c32857a4\": container with ID starting with 869a7fc0c617e8e088d899f5a9ad9ec5551dbeacd8a4f27d810b9993c32857a4 not found: ID does not exist"
Nov 01 03:53:38.335137 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.335096    4930 scope.go:122] "RemoveContainer" containerID="6b1f11a10a7deb1b747125dcba38e6a0e7b6b526073041c52e6abadd79caeeac"
Nov 01 03:53:38.335338 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:53:38.335323    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b1f11a10a7deb1b747125dcba38e6a0e7b6b526073041c52e6abadd79caeeac\": container with ID starting with 6b1f11a10a7deb1b747125dcba38e6a0e7b6b526073041c52e6abadd79caeeac not found: ID does not exist" containerID="6b1f11a10a7deb1b747125dcba38e6a0e7b6b526073041c52e6abadd79caeeac"
Nov 01 03:53:38.335384 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.335342    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b1f11a10a7deb1b747125dcba38e6a0e7b6b526073041c52e6abadd79caeeac"} err="failed to get container status \"6b1f11a10a7deb1b747125dcba38e6a0e7b6b526073041c52e6abadd79caeeac\": rpc error: code = NotFound desc = could not find container \"6b1f11a10a7deb1b747125dcba38e6a0e7b6b526073041c52e6abadd79caeeac\": container with ID starting with 6b1f11a10a7deb1b747125dcba38e6a0e7b6b526073041c52e6abadd79caeeac not found: ID does not exist"
Nov 01 03:53:38.378194 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.378156    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:38.717986 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.717948    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"]
Nov 01 03:53:38.724316 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:38.724288    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-211380-openshift-default-7b97fb6879-8c2mb"]
Nov 01 03:53:39.620091 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:39.620061    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b72d8427-c8a5-4d67-be83-57a2f54dddb4" path="/var/lib/kubelet/pods/b72d8427-c8a5-4d67-be83-57a2f54dddb4/volumes"
Nov 01 03:53:40.116927 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:40.116893    4930 scope.go:122] "RemoveContainer" containerID="b86f9c9de01c897ef85827caa032d4fe67075e95b3b925d97388fc199d645f9b"
Nov 01 03:53:41.965370 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:41.965326    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:42.793077 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:42.793037    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:43.471544 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:43.471510    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:43.617578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:43.617519    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:43.799475 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:43.799390    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2"]
Nov 01 03:53:43.810027 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:43.809992    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:44.353390 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:44.353341    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2" podUID="eb6a2bc2-9530-44d6-aa35-608f182ddb04" containerName="istio-proxy" containerID="cri-o://acde80c7095a104624d04842e1217d560af7e8823b60c08afdc64c9af86a7e67" gracePeriod=30
Nov 01 03:53:44.953113 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:44.953083    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:45.098507 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:45.098476    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:45.414507 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:45.414471    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:45.613666 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:45.613633    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:46.725879 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:46.725837    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:46.871411 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:46.871324    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:49.004280 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:49.004234    4930 patch_prober.go:29] interesting pod/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 03:53:49.004711 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:49.004354    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2" podUID="eb6a2bc2-9530-44d6-aa35-608f182ddb04" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 03:53:49.122747 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:53:49.122707    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:53:49.617155 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:49.617128    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2"
Nov 01 03:53:49.786225 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:49.786133    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb6a2bc2-9530-44d6-aa35-608f182ddb04-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb6a2bc2-9530-44d6-aa35-608f182ddb04-workload-socket\") pod \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\" (UID: \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\") "
Nov 01 03:53:49.786225 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:49.786187    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb6a2bc2-9530-44d6-aa35-608f182ddb04-kube-api-access-w7mbc\" (UniqueName: \"kubernetes.io/projected/eb6a2bc2-9530-44d6-aa35-608f182ddb04-kube-api-access-w7mbc\") pod \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\" (UID: \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\") "
Nov 01 03:53:49.786468 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:49.786238    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb6a2bc2-9530-44d6-aa35-608f182ddb04-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb6a2bc2-9530-44d6-aa35-608f182ddb04-istio-podinfo\") pod \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\" (UID: \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\") "
Nov 01 03:53:49.786468 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:49.786340    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb6a2bc2-9530-44d6-aa35-608f182ddb04-istio-token\" (UniqueName: \"kubernetes.io/projected/eb6a2bc2-9530-44d6-aa35-608f182ddb04-istio-token\") pod \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\" (UID: \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\") "
Nov 01 03:53:49.786468 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:49.786392    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb6a2bc2-9530-44d6-aa35-608f182ddb04-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb6a2bc2-9530-44d6-aa35-608f182ddb04-istio-envoy\") pod \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\" (UID: \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\") "
Nov 01 03:53:49.786468 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:49.786420    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb6a2bc2-9530-44d6-aa35-608f182ddb04-workload-socket" pod "eb6a2bc2-9530-44d6-aa35-608f182ddb04" (UID: "eb6a2bc2-9530-44d6-aa35-608f182ddb04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:49.786468 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:49.786423    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb6a2bc2-9530-44d6-aa35-608f182ddb04-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb6a2bc2-9530-44d6-aa35-608f182ddb04-istio-data\") pod \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\" (UID: \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\") "
Nov 01 03:53:49.786774 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:49.786582    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb6a2bc2-9530-44d6-aa35-608f182ddb04-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb6a2bc2-9530-44d6-aa35-608f182ddb04-credential-socket\") pod \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\" (UID: \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\") "
Nov 01 03:53:49.786774 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:49.786633    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb6a2bc2-9530-44d6-aa35-608f182ddb04-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb6a2bc2-9530-44d6-aa35-608f182ddb04-istiod-ca-cert\") pod \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\" (UID: \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\") "
Nov 01 03:53:49.786774 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:49.786669    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb6a2bc2-9530-44d6-aa35-608f182ddb04-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb6a2bc2-9530-44d6-aa35-608f182ddb04-workload-certs\") pod \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\" (UID: \"eb6a2bc2-9530-44d6-aa35-608f182ddb04\") "
Nov 01 03:53:49.786948 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:49.786819    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb6a2bc2-9530-44d6-aa35-608f182ddb04-istio-data" pod "eb6a2bc2-9530-44d6-aa35-608f182ddb04" (UID: "eb6a2bc2-9530-44d6-aa35-608f182ddb04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:49.787011 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:49.786978    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb6a2bc2-9530-44d6-aa35-608f182ddb04-credential-socket" pod "eb6a2bc2-9530-44d6-aa35-608f182ddb04" (UID: "eb6a2bc2-9530-44d6-aa35-608f182ddb04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:49.787070 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:49.787030    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb6a2bc2-9530-44d6-aa35-608f182ddb04-workload-certs" pod "eb6a2bc2-9530-44d6-aa35-608f182ddb04" (UID: "eb6a2bc2-9530-44d6-aa35-608f182ddb04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:49.787213 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:49.787157    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb6a2bc2-9530-44d6-aa35-608f182ddb04-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:49.787213 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:49.787186    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb6a2bc2-9530-44d6-aa35-608f182ddb04-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:49.787213 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:49.787199    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb6a2bc2-9530-44d6-aa35-608f182ddb04-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:49.787356 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:49.787214    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb6a2bc2-9530-44d6-aa35-608f182ddb04-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:49.787356 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:49.787257    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb6a2bc2-9530-44d6-aa35-608f182ddb04-istiod-ca-cert" pod "eb6a2bc2-9530-44d6-aa35-608f182ddb04" (UID: "eb6a2bc2-9530-44d6-aa35-608f182ddb04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:53:49.789052 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:49.789024    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb6a2bc2-9530-44d6-aa35-608f182ddb04-kube-api-access-w7mbc" pod "eb6a2bc2-9530-44d6-aa35-608f182ddb04" (UID: "eb6a2bc2-9530-44d6-aa35-608f182ddb04"). InnerVolumeSpecName "kube-api-access-w7mbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:53:49.789052 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:49.789031    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb6a2bc2-9530-44d6-aa35-608f182ddb04-istio-token" pod "eb6a2bc2-9530-44d6-aa35-608f182ddb04" (UID: "eb6a2bc2-9530-44d6-aa35-608f182ddb04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:53:49.789241 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:49.789114    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb6a2bc2-9530-44d6-aa35-608f182ddb04-istio-envoy" pod "eb6a2bc2-9530-44d6-aa35-608f182ddb04" (UID: "eb6a2bc2-9530-44d6-aa35-608f182ddb04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:49.789241 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:49.789223    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb6a2bc2-9530-44d6-aa35-608f182ddb04-istio-podinfo" pod "eb6a2bc2-9530-44d6-aa35-608f182ddb04" (UID: "eb6a2bc2-9530-44d6-aa35-608f182ddb04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:53:49.888685 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:49.888656    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb6a2bc2-9530-44d6-aa35-608f182ddb04-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:49.888685 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:49.888681    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb6a2bc2-9530-44d6-aa35-608f182ddb04-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:49.888685 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:49.888691    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb6a2bc2-9530-44d6-aa35-608f182ddb04-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:49.888931 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:49.888702    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7mbc\" (UniqueName: \"kubernetes.io/projected/eb6a2bc2-9530-44d6-aa35-608f182ddb04-kube-api-access-w7mbc\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:49.888931 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:49.888722    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb6a2bc2-9530-44d6-aa35-608f182ddb04-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:50.400443 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:50.400409    4930 generic.go:356] "Generic (PLEG): container finished" podID="eb6a2bc2-9530-44d6-aa35-608f182ddb04" containerID="acde80c7095a104624d04842e1217d560af7e8823b60c08afdc64c9af86a7e67" exitCode=0
Nov 01 03:53:50.400870 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:50.400447    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2" event={"ID":"eb6a2bc2-9530-44d6-aa35-608f182ddb04","Type":"ContainerDied","Data":"acde80c7095a104624d04842e1217d560af7e8823b60c08afdc64c9af86a7e67"}
Nov 01 03:53:50.400870 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:50.400455    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2"
Nov 01 03:53:50.400870 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:50.400492    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2" event={"ID":"eb6a2bc2-9530-44d6-aa35-608f182ddb04","Type":"ContainerDied","Data":"c8063bdd7bc8d83fcda82627d1d9282af265570b62f698367aa73352044b2747"}
Nov 01 03:53:50.400870 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:50.400514    4930 scope.go:122] "RemoveContainer" containerID="acde80c7095a104624d04842e1217d560af7e8823b60c08afdc64c9af86a7e67"
Nov 01 03:53:50.411251 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:50.411231    4930 scope.go:122] "RemoveContainer" containerID="acde80c7095a104624d04842e1217d560af7e8823b60c08afdc64c9af86a7e67"
Nov 01 03:53:50.411503 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:53:50.411484    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acde80c7095a104624d04842e1217d560af7e8823b60c08afdc64c9af86a7e67\": container with ID starting with acde80c7095a104624d04842e1217d560af7e8823b60c08afdc64c9af86a7e67 not found: ID does not exist" containerID="acde80c7095a104624d04842e1217d560af7e8823b60c08afdc64c9af86a7e67"
Nov 01 03:53:50.411598 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:50.411510    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acde80c7095a104624d04842e1217d560af7e8823b60c08afdc64c9af86a7e67"} err="failed to get container status \"acde80c7095a104624d04842e1217d560af7e8823b60c08afdc64c9af86a7e67\": rpc error: code = NotFound desc = could not find container \"acde80c7095a104624d04842e1217d560af7e8823b60c08afdc64c9af86a7e67\": container with ID starting with acde80c7095a104624d04842e1217d560af7e8823b60c08afdc64c9af86a7e67 not found: ID does not exist"
Nov 01 03:53:50.617714 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:50.617681    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2"]
Nov 01 03:53:50.621804 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:50.621776    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-c22dd8-openshift-default-5879c8cc6f-nlts2"]
Nov 01 03:53:50.853707 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:50.853673    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:51.048414 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:51.048378    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-620c59b5-2pbz9"]
Nov 01 03:53:51.052970 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:51.052946    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-620c59b5-2pbz9"]
Nov 01 03:53:51.155026 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:51.154947    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3fb7a87-qbxs5"]
Nov 01 03:53:51.155363 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:51.155351    4930 cpu_manager.go:414] "containerMap: removing container" podUID="b6d688b8-b911-4a6a-a842-2b0fb7a75e83" containerName="main"
Nov 01 03:53:51.155408 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:51.155364    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6d688b8-b911-4a6a-a842-2b0fb7a75e83" containerName="main"
Nov 01 03:53:51.155408 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:51.155381    4930 cpu_manager.go:414] "containerMap: removing container" podUID="eb6a2bc2-9530-44d6-aa35-608f182ddb04" containerName="istio-proxy"
Nov 01 03:53:51.155408 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:51.155386    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb6a2bc2-9530-44d6-aa35-608f182ddb04" containerName="istio-proxy"
Nov 01 03:53:51.155408 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:51.155405    4930 cpu_manager.go:414] "containerMap: removing container" podUID="e2483a06-2f4b-4a1e-af4e-a63eb50d29bf" containerName="main"
Nov 01 03:53:51.155542 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:51.155411    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2483a06-2f4b-4a1e-af4e-a63eb50d29bf" containerName="main"
Nov 01 03:53:51.155542 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:51.155427    4930 cpu_manager.go:414] "containerMap: removing container" podUID="b72d8427-c8a5-4d67-be83-57a2f54dddb4" containerName="istio-proxy"
Nov 01 03:53:51.155542 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:51.155432    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b72d8427-c8a5-4d67-be83-57a2f54dddb4" containerName="istio-proxy"
Nov 01 03:53:51.155542 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:51.155439    4930 cpu_manager.go:414] "containerMap: removing container" podUID="b72d8427-c8a5-4d67-be83-57a2f54dddb4" containerName="istio-proxy"
Nov 01 03:53:51.155542 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:51.155443    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b72d8427-c8a5-4d67-be83-57a2f54dddb4" containerName="istio-proxy"
Nov 01 03:53:51.155542 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:51.155456    4930 cpu_manager.go:414] "containerMap: removing container" podUID="22002d07-ffff-4f2f-b9d7-2751eae471d1" containerName="revoke-keys"
Nov 01 03:53:51.155542 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:51.155460    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22002d07-ffff-4f2f-b9d7-2751eae471d1" containerName="revoke-keys"
Nov 01 03:53:51.155542 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:51.155516    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2483a06-2f4b-4a1e-af4e-a63eb50d29bf" containerName="main"
Nov 01 03:53:51.155542 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:51.155528    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="b72d8427-c8a5-4d67-be83-57a2f54dddb4" containerName="istio-proxy"
Nov 01 03:53:51.155542 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:51.155534    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="b72d8427-c8a5-4d67-be83-57a2f54dddb4" containerName="istio-proxy"
Nov 01 03:53:51.155542 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:51.155540    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="22002d07-ffff-4f2f-b9d7-2751eae471d1" containerName="revoke-keys"
Nov 01 03:53:51.155887 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:51.155547    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6d688b8-b911-4a6a-a842-2b0fb7a75e83" containerName="main"
Nov 01 03:53:51.155887 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:51.155567    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb6a2bc2-9530-44d6-aa35-608f182ddb04" containerName="istio-proxy"
Nov 01 03:53:51.161670 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:51.161646    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3fb7a87-qbxs5"
Nov 01 03:53:51.207658 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:51.200549    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8pgd\" (UniqueName: \"kubernetes.io/projected/f3b79a6d-601b-4348-8ac2-620145e0bbe1-kube-api-access-b8pgd\") pod \"maas-api-revoke-sub-c3fb7a87-qbxs5\" (UID: \"f3b79a6d-601b-4348-8ac2-620145e0bbe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3fb7a87-qbxs5"
Nov 01 03:53:51.207658 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:51.200669    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3b79a6d-601b-4348-8ac2-620145e0bbe1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3fb7a87-qbxs5\" (UID: \"f3b79a6d-601b-4348-8ac2-620145e0bbe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3fb7a87-qbxs5"
Nov 01 03:53:51.207658 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:51.200763    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3b79a6d-601b-4348-8ac2-620145e0bbe1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3fb7a87-qbxs5\" (UID: \"f3b79a6d-601b-4348-8ac2-620145e0bbe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3fb7a87-qbxs5"
Nov 01 03:53:51.215509 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:51.215467    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:53:51.215666 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:51.215470    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:53:51.223841 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:51.223816    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3fb7a87-qbxs5"]
Nov 01 03:53:51.259054 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:51.259031    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:53:51.259322 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:51.259305    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:53:51.301740 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:51.301707    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8pgd\" (UniqueName: \"kubernetes.io/projected/f3b79a6d-601b-4348-8ac2-620145e0bbe1-kube-api-access-b8pgd\") pod \"maas-api-revoke-sub-c3fb7a87-qbxs5\" (UID: \"f3b79a6d-601b-4348-8ac2-620145e0bbe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3fb7a87-qbxs5"
Nov 01 03:53:51.301884 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:51.301866    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3b79a6d-601b-4348-8ac2-620145e0bbe1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3fb7a87-qbxs5\" (UID: \"f3b79a6d-601b-4348-8ac2-620145e0bbe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3fb7a87-qbxs5"
Nov 01 03:53:51.302000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:51.301979    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3b79a6d-601b-4348-8ac2-620145e0bbe1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3fb7a87-qbxs5\" (UID: \"f3b79a6d-601b-4348-8ac2-620145e0bbe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3fb7a87-qbxs5"
Nov 01 03:53:51.302640 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:51.302617    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3b79a6d-601b-4348-8ac2-620145e0bbe1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3fb7a87-qbxs5\" (UID: \"f3b79a6d-601b-4348-8ac2-620145e0bbe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3fb7a87-qbxs5"
Nov 01 03:53:51.304176 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:51.304153    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3b79a6d-601b-4348-8ac2-620145e0bbe1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3fb7a87-qbxs5\" (UID: \"f3b79a6d-601b-4348-8ac2-620145e0bbe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3fb7a87-qbxs5"
Nov 01 03:53:51.362303 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:51.362271    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8pgd\" (UniqueName: \"kubernetes.io/projected/f3b79a6d-601b-4348-8ac2-620145e0bbe1-kube-api-access-b8pgd\") pod \"maas-api-revoke-sub-c3fb7a87-qbxs5\" (UID: \"f3b79a6d-601b-4348-8ac2-620145e0bbe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3fb7a87-qbxs5"
Nov 01 03:53:51.484508 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:51.484425    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3fb7a87-qbxs5"
Nov 01 03:53:51.619361 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:51.619326    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baa79f53-4c43-4b71-9e2a-70033aa07c73" path="/var/lib/kubelet/pods/baa79f53-4c43-4b71-9e2a-70033aa07c73/volumes"
Nov 01 03:53:51.619818 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:51.619803    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb6a2bc2-9530-44d6-aa35-608f182ddb04" path="/var/lib/kubelet/pods/eb6a2bc2-9530-44d6-aa35-608f182ddb04/volumes"
Nov 01 03:53:51.861075 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:51.861048    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3fb7a87-qbxs5"]
Nov 01 03:53:51.862992 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:53:51.862961    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3b79a6d_601b_4348_8ac2_620145e0bbe1.slice/crio-fab6145085cd36ac247b58811f8f637ca58fccc19a3e7c8695a31cecefe8f697 WatchSource:0}: Error finding container fab6145085cd36ac247b58811f8f637ca58fccc19a3e7c8695a31cecefe8f697: Status 404 returned error can't find the container with id fab6145085cd36ac247b58811f8f637ca58fccc19a3e7c8695a31cecefe8f697
Nov 01 03:53:52.047918 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:52.047882    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ca62595-4t74g"]
Nov 01 03:53:52.054018 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:52.053978    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ca62595-4t74g"]
Nov 01 03:53:52.307089 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:52.307050    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:52.415171 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:52.415131    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3fb7a87-qbxs5" event={"ID":"f3b79a6d-601b-4348-8ac2-620145e0bbe1","Type":"ContainerStarted","Data":"09e1ad8f98dfc55c8010d74ebb55875c3b6a3d4019271ffa7554fdb711ee3dc0"}
Nov 01 03:53:52.415351 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:52.415193    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3fb7a87-qbxs5" event={"ID":"f3b79a6d-601b-4348-8ac2-620145e0bbe1","Type":"ContainerStarted","Data":"fab6145085cd36ac247b58811f8f637ca58fccc19a3e7c8695a31cecefe8f697"}
Nov 01 03:53:52.531615 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:52.531517    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3fb7a87-qbxs5" podStartSLOduration=1.5315040450000001 podStartE2EDuration="1.531504045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:53:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:53:52.529991461 +0000 UTC m=+2837.052998052" watchObservedRunningTime="2026-11-01 03:53:52.531504045 +0000 UTC m=+2837.054510633"
Nov 01 03:53:53.000047 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.000010    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:53:53.274722 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.274672    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" podUID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" containerName="main" containerID="cri-o://cc9372d0920554e876a589302dd84a247bb6e71f92b186a7071aaf60da8f1986" gracePeriod=45
Nov 01 03:53:53.276771 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.276743    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps" podUID="d658772c-b072-4513-a639-d19a94c78525" containerName="main" containerID="cri-o://fbcf34847caf605aa43a2ad2d119faf20444fecc1f47218da73f3b077f32abea" gracePeriod=45
Nov 01 03:53:53.278927 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.278897    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t" podUID="13930c1e-2200-4412-b8d3-ae3e4126a851" containerName="main" containerID="cri-o://2df24b3e6888fcf107df563d3fe900e7eb77cf7c9623071339d8bff59698c040" gracePeriod=45
Nov 01 03:53:53.352473 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.352438    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t" podUID="13930c1e-2200-4412-b8d3-ae3e4126a851" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": dial tcp 10.133.0.60:8000: connect: connection refused"
Nov 01 03:53:53.384709 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.384679    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" podUID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": dial tcp 10.133.0.95:8000: connect: connection refused"
Nov 01 03:53:53.433425 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.433390    4930 generic.go:356] "Generic (PLEG): container finished" podID="13930c1e-2200-4412-b8d3-ae3e4126a851" containerID="2df24b3e6888fcf107df563d3fe900e7eb77cf7c9623071339d8bff59698c040" exitCode=0
Nov 01 03:53:53.433606 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.433519    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t" event={"ID":"13930c1e-2200-4412-b8d3-ae3e4126a851","Type":"ContainerDied","Data":"2df24b3e6888fcf107df563d3fe900e7eb77cf7c9623071339d8bff59698c040"}
Nov 01 03:53:53.435189 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.435159    4930 generic.go:356] "Generic (PLEG): container finished" podID="f3b79a6d-601b-4348-8ac2-620145e0bbe1" containerID="09e1ad8f98dfc55c8010d74ebb55875c3b6a3d4019271ffa7554fdb711ee3dc0" exitCode=0
Nov 01 03:53:53.435316 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.435285    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3fb7a87-qbxs5" event={"ID":"f3b79a6d-601b-4348-8ac2-620145e0bbe1","Type":"ContainerDied","Data":"09e1ad8f98dfc55c8010d74ebb55875c3b6a3d4019271ffa7554fdb711ee3dc0"}
Nov 01 03:53:53.437229 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.437201    4930 generic.go:356] "Generic (PLEG): container finished" podID="d658772c-b072-4513-a639-d19a94c78525" containerID="fbcf34847caf605aa43a2ad2d119faf20444fecc1f47218da73f3b077f32abea" exitCode=0
Nov 01 03:53:53.437352 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.437331    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps" event={"ID":"d658772c-b072-4513-a639-d19a94c78525","Type":"ContainerDied","Data":"fbcf34847caf605aa43a2ad2d119faf20444fecc1f47218da73f3b077f32abea"}
Nov 01 03:53:53.439728 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.439700    4930 generic.go:356] "Generic (PLEG): container finished" podID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" containerID="cc9372d0920554e876a589302dd84a247bb6e71f92b186a7071aaf60da8f1986" exitCode=0
Nov 01 03:53:53.439853 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.439795    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" event={"ID":"baf6d6ac-7399-4c2b-bf41-5423abbedef2","Type":"ContainerDied","Data":"cc9372d0920554e876a589302dd84a247bb6e71f92b186a7071aaf60da8f1986"}
Nov 01 03:53:53.560811 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.560779    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:53:53.564521 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.564495    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:53:53.589542 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.589525    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:53:53.618441 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.618414    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73f7e8a8-f82d-4176-be18-923cff53c4bd" path="/var/lib/kubelet/pods/73f7e8a8-f82d-4176-be18-923cff53c4bd/volumes"
Nov 01 03:53:53.624655 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.624633    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baf6d6ac-7399-4c2b-bf41-5423abbedef2-kube-api-access-ljxsz\" (UniqueName: \"kubernetes.io/projected/baf6d6ac-7399-4c2b-bf41-5423abbedef2-kube-api-access-ljxsz\") pod \"baf6d6ac-7399-4c2b-bf41-5423abbedef2\" (UID: \"baf6d6ac-7399-4c2b-bf41-5423abbedef2\") "
Nov 01 03:53:53.624752 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.624674    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d658772c-b072-4513-a639-d19a94c78525-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d658772c-b072-4513-a639-d19a94c78525-model-cache\") pod \"d658772c-b072-4513-a639-d19a94c78525\" (UID: \"d658772c-b072-4513-a639-d19a94c78525\") "
Nov 01 03:53:53.624752 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.624737    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baf6d6ac-7399-4c2b-bf41-5423abbedef2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf6d6ac-7399-4c2b-bf41-5423abbedef2-model-cache\") pod \"baf6d6ac-7399-4c2b-bf41-5423abbedef2\" (UID: \"baf6d6ac-7399-4c2b-bf41-5423abbedef2\") "
Nov 01 03:53:53.624880 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.624789    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baf6d6ac-7399-4c2b-bf41-5423abbedef2-home\" (UniqueName: \"kubernetes.io/empty-dir/baf6d6ac-7399-4c2b-bf41-5423abbedef2-home\") pod \"baf6d6ac-7399-4c2b-bf41-5423abbedef2\" (UID: \"baf6d6ac-7399-4c2b-bf41-5423abbedef2\") "
Nov 01 03:53:53.624880 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.624836    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13930c1e-2200-4412-b8d3-ae3e4126a851-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13930c1e-2200-4412-b8d3-ae3e4126a851-model-cache\") pod \"13930c1e-2200-4412-b8d3-ae3e4126a851\" (UID: \"13930c1e-2200-4412-b8d3-ae3e4126a851\") "
Nov 01 03:53:53.624880 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.624871    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baf6d6ac-7399-4c2b-bf41-5423abbedef2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf6d6ac-7399-4c2b-bf41-5423abbedef2-dshm\") pod \"baf6d6ac-7399-4c2b-bf41-5423abbedef2\" (UID: \"baf6d6ac-7399-4c2b-bf41-5423abbedef2\") "
Nov 01 03:53:53.625039 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.624912    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d658772c-b072-4513-a639-d19a94c78525-tls-certs\" (UniqueName: \"kubernetes.io/secret/d658772c-b072-4513-a639-d19a94c78525-tls-certs\") pod \"d658772c-b072-4513-a639-d19a94c78525\" (UID: \"d658772c-b072-4513-a639-d19a94c78525\") "
Nov 01 03:53:53.625039 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.624944    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d658772c-b072-4513-a639-d19a94c78525-model-cache" pod "d658772c-b072-4513-a639-d19a94c78525" (UID: "d658772c-b072-4513-a639-d19a94c78525"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:53.625039 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.624950    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d658772c-b072-4513-a639-d19a94c78525-home\" (UniqueName: \"kubernetes.io/empty-dir/d658772c-b072-4513-a639-d19a94c78525-home\") pod \"d658772c-b072-4513-a639-d19a94c78525\" (UID: \"d658772c-b072-4513-a639-d19a94c78525\") "
Nov 01 03:53:53.625270 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.625033    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13930c1e-2200-4412-b8d3-ae3e4126a851-home\" (UniqueName: \"kubernetes.io/empty-dir/13930c1e-2200-4412-b8d3-ae3e4126a851-home\") pod \"13930c1e-2200-4412-b8d3-ae3e4126a851\" (UID: \"13930c1e-2200-4412-b8d3-ae3e4126a851\") "
Nov 01 03:53:53.625270 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.625068    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf6d6ac-7399-4c2b-bf41-5423abbedef2-model-cache" pod "baf6d6ac-7399-4c2b-bf41-5423abbedef2" (UID: "baf6d6ac-7399-4c2b-bf41-5423abbedef2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:53.625270 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.625089    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baf6d6ac-7399-4c2b-bf41-5423abbedef2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf6d6ac-7399-4c2b-bf41-5423abbedef2-tmp-dir\") pod \"baf6d6ac-7399-4c2b-bf41-5423abbedef2\" (UID: \"baf6d6ac-7399-4c2b-bf41-5423abbedef2\") "
Nov 01 03:53:53.625270 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.625095    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13930c1e-2200-4412-b8d3-ae3e4126a851-model-cache" pod "13930c1e-2200-4412-b8d3-ae3e4126a851" (UID: "13930c1e-2200-4412-b8d3-ae3e4126a851"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:53.625270 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.625128    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/baf6d6ac-7399-4c2b-bf41-5423abbedef2-tls-certs\" (UniqueName: \"kubernetes.io/secret/baf6d6ac-7399-4c2b-bf41-5423abbedef2-tls-certs\") pod \"baf6d6ac-7399-4c2b-bf41-5423abbedef2\" (UID: \"baf6d6ac-7399-4c2b-bf41-5423abbedef2\") "
Nov 01 03:53:53.625270 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.625161    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d658772c-b072-4513-a639-d19a94c78525-home" pod "d658772c-b072-4513-a639-d19a94c78525" (UID: "d658772c-b072-4513-a639-d19a94c78525"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:53.625270 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.625178    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13930c1e-2200-4412-b8d3-ae3e4126a851-dshm\" (UniqueName: \"kubernetes.io/empty-dir/13930c1e-2200-4412-b8d3-ae3e4126a851-dshm\") pod \"13930c1e-2200-4412-b8d3-ae3e4126a851\" (UID: \"13930c1e-2200-4412-b8d3-ae3e4126a851\") "
Nov 01 03:53:53.625270 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.625216    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13930c1e-2200-4412-b8d3-ae3e4126a851-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13930c1e-2200-4412-b8d3-ae3e4126a851-tmp-dir\") pod \"13930c1e-2200-4412-b8d3-ae3e4126a851\" (UID: \"13930c1e-2200-4412-b8d3-ae3e4126a851\") "
Nov 01 03:53:53.625270 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.625243    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13930c1e-2200-4412-b8d3-ae3e4126a851-tls-certs\" (UniqueName: \"kubernetes.io/secret/13930c1e-2200-4412-b8d3-ae3e4126a851-tls-certs\") pod \"13930c1e-2200-4412-b8d3-ae3e4126a851\" (UID: \"13930c1e-2200-4412-b8d3-ae3e4126a851\") "
Nov 01 03:53:53.625270 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.625272    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d658772c-b072-4513-a639-d19a94c78525-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d658772c-b072-4513-a639-d19a94c78525-dshm\") pod \"d658772c-b072-4513-a639-d19a94c78525\" (UID: \"d658772c-b072-4513-a639-d19a94c78525\") "
Nov 01 03:53:53.626065 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.625300    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d658772c-b072-4513-a639-d19a94c78525-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d658772c-b072-4513-a639-d19a94c78525-tmp-dir\") pod \"d658772c-b072-4513-a639-d19a94c78525\" (UID: \"d658772c-b072-4513-a639-d19a94c78525\") "
Nov 01 03:53:53.626065 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.625314    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf6d6ac-7399-4c2b-bf41-5423abbedef2-home" pod "baf6d6ac-7399-4c2b-bf41-5423abbedef2" (UID: "baf6d6ac-7399-4c2b-bf41-5423abbedef2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:53.626065 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.625316    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13930c1e-2200-4412-b8d3-ae3e4126a851-home" pod "13930c1e-2200-4412-b8d3-ae3e4126a851" (UID: "13930c1e-2200-4412-b8d3-ae3e4126a851"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:53.626065 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.625332    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf6d6ac-7399-4c2b-bf41-5423abbedef2-tmp-dir" pod "baf6d6ac-7399-4c2b-bf41-5423abbedef2" (UID: "baf6d6ac-7399-4c2b-bf41-5423abbedef2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:53.626065 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.625346    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13930c1e-2200-4412-b8d3-ae3e4126a851-kube-api-access-pzmjc\" (UniqueName: \"kubernetes.io/projected/13930c1e-2200-4412-b8d3-ae3e4126a851-kube-api-access-pzmjc\") pod \"13930c1e-2200-4412-b8d3-ae3e4126a851\" (UID: \"13930c1e-2200-4412-b8d3-ae3e4126a851\") "
Nov 01 03:53:53.626065 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.625407    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d658772c-b072-4513-a639-d19a94c78525-kube-api-access-4g8l4\" (UniqueName: \"kubernetes.io/projected/d658772c-b072-4513-a639-d19a94c78525-kube-api-access-4g8l4\") pod \"d658772c-b072-4513-a639-d19a94c78525\" (UID: \"d658772c-b072-4513-a639-d19a94c78525\") "
Nov 01 03:53:53.626065 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.625510    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13930c1e-2200-4412-b8d3-ae3e4126a851-tmp-dir" pod "13930c1e-2200-4412-b8d3-ae3e4126a851" (UID: "13930c1e-2200-4412-b8d3-ae3e4126a851"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:53.626065 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.625732    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d658772c-b072-4513-a639-d19a94c78525-tmp-dir" pod "d658772c-b072-4513-a639-d19a94c78525" (UID: "d658772c-b072-4513-a639-d19a94c78525"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:53.626065 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.625757    4930 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d658772c-b072-4513-a639-d19a94c78525-model-cache\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:53.626065 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.625778    4930 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf6d6ac-7399-4c2b-bf41-5423abbedef2-model-cache\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:53.626065 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.625793    4930 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baf6d6ac-7399-4c2b-bf41-5423abbedef2-home\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:53.626065 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.625807    4930 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13930c1e-2200-4412-b8d3-ae3e4126a851-model-cache\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:53.626065 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.625820    4930 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d658772c-b072-4513-a639-d19a94c78525-home\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:53.626065 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.625834    4930 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13930c1e-2200-4412-b8d3-ae3e4126a851-home\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:53.626065 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.625846    4930 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf6d6ac-7399-4c2b-bf41-5423abbedef2-tmp-dir\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:53.626065 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.625858    4930 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13930c1e-2200-4412-b8d3-ae3e4126a851-tmp-dir\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:53.628937 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.628914    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baf6d6ac-7399-4c2b-bf41-5423abbedef2-tls-certs" pod "baf6d6ac-7399-4c2b-bf41-5423abbedef2" (UID: "baf6d6ac-7399-4c2b-bf41-5423abbedef2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:53:53.629050 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.629028    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d658772c-b072-4513-a639-d19a94c78525-tls-certs" pod "d658772c-b072-4513-a639-d19a94c78525" (UID: "d658772c-b072-4513-a639-d19a94c78525"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:53:53.629050 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.629037    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d658772c-b072-4513-a639-d19a94c78525-dshm" pod "d658772c-b072-4513-a639-d19a94c78525" (UID: "d658772c-b072-4513-a639-d19a94c78525"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:53.629244 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.629077    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13930c1e-2200-4412-b8d3-ae3e4126a851-dshm" pod "13930c1e-2200-4412-b8d3-ae3e4126a851" (UID: "13930c1e-2200-4412-b8d3-ae3e4126a851"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:53.629244 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.629167    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13930c1e-2200-4412-b8d3-ae3e4126a851-tls-certs" pod "13930c1e-2200-4412-b8d3-ae3e4126a851" (UID: "13930c1e-2200-4412-b8d3-ae3e4126a851"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:53:53.629244 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.629182    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf6d6ac-7399-4c2b-bf41-5423abbedef2-dshm" pod "baf6d6ac-7399-4c2b-bf41-5423abbedef2" (UID: "baf6d6ac-7399-4c2b-bf41-5423abbedef2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:53:53.629361 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.629275    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13930c1e-2200-4412-b8d3-ae3e4126a851-kube-api-access-pzmjc" pod "13930c1e-2200-4412-b8d3-ae3e4126a851" (UID: "13930c1e-2200-4412-b8d3-ae3e4126a851"). InnerVolumeSpecName "kube-api-access-pzmjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:53:53.629361 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.629356    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baf6d6ac-7399-4c2b-bf41-5423abbedef2-kube-api-access-ljxsz" pod "baf6d6ac-7399-4c2b-bf41-5423abbedef2" (UID: "baf6d6ac-7399-4c2b-bf41-5423abbedef2"). InnerVolumeSpecName "kube-api-access-ljxsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:53:53.629753 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.629737    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d658772c-b072-4513-a639-d19a94c78525-kube-api-access-4g8l4" pod "d658772c-b072-4513-a639-d19a94c78525" (UID: "d658772c-b072-4513-a639-d19a94c78525"). InnerVolumeSpecName "kube-api-access-4g8l4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:53:53.726580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.726529    4930 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf6d6ac-7399-4c2b-bf41-5423abbedef2-dshm\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:53.726580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.726575    4930 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d658772c-b072-4513-a639-d19a94c78525-tls-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:53.726580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.726584    4930 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf6d6ac-7399-4c2b-bf41-5423abbedef2-tls-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:53.726816 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.726594    4930 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13930c1e-2200-4412-b8d3-ae3e4126a851-dshm\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:53.726816 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.726603    4930 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13930c1e-2200-4412-b8d3-ae3e4126a851-tls-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:53.726816 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.726619    4930 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d658772c-b072-4513-a639-d19a94c78525-dshm\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:53.726816 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.726627    4930 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d658772c-b072-4513-a639-d19a94c78525-tmp-dir\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:53.726816 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.726637    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzmjc\" (UniqueName: \"kubernetes.io/projected/13930c1e-2200-4412-b8d3-ae3e4126a851-kube-api-access-pzmjc\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:53.726816 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.726646    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g8l4\" (UniqueName: \"kubernetes.io/projected/d658772c-b072-4513-a639-d19a94c78525-kube-api-access-4g8l4\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:53.726816 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:53.726655    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljxsz\" (UniqueName: \"kubernetes.io/projected/baf6d6ac-7399-4c2b-bf41-5423abbedef2-kube-api-access-ljxsz\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:54.447121 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:54.447085    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"
Nov 01 03:53:54.447121 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:54.447105    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps" event={"ID":"d658772c-b072-4513-a639-d19a94c78525","Type":"ContainerDied","Data":"d2c9a6aa6b69199b7c19744bc2ac0e2a85cc643a1947ad310ab8b99a2712182b"}
Nov 01 03:53:54.447396 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:54.447150    4930 scope.go:122] "RemoveContainer" containerID="fbcf34847caf605aa43a2ad2d119faf20444fecc1f47218da73f3b077f32abea"
Nov 01 03:53:54.448712 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:54.448689    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"
Nov 01 03:53:54.448861 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:54.448757    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5" event={"ID":"baf6d6ac-7399-4c2b-bf41-5423abbedef2","Type":"ContainerDied","Data":"76b1d42a1fa93249cbd18ac86e6bee0ac381ec71301951315257dc7cb641d076"}
Nov 01 03:53:54.450312 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:54.450287    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t" event={"ID":"13930c1e-2200-4412-b8d3-ae3e4126a851","Type":"ContainerDied","Data":"bfcef07b400ce8a43b7987fedd0b20488a34a2daa38c55f54fc4e4c2263be184"}
Nov 01 03:53:54.450445 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:54.450316    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"
Nov 01 03:53:54.459140 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:54.459124    4930 scope.go:122] "RemoveContainer" containerID="cc9372d0920554e876a589302dd84a247bb6e71f92b186a7071aaf60da8f1986"
Nov 01 03:53:54.469507 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:54.469485    4930 scope.go:122] "RemoveContainer" containerID="2df24b3e6888fcf107df563d3fe900e7eb77cf7c9623071339d8bff59698c040"
Nov 01 03:53:54.582675 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:54.582649    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3fb7a87-qbxs5"
Nov 01 03:53:54.614047 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:54.614007    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"]
Nov 01 03:53:54.616814 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:54.616791    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-19faa5/premium-w6-19faa5-kserve-7b6997745f-8wzps"]
Nov 01 03:53:54.637006 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:54.636986    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3b79a6d-601b-4348-8ac2-620145e0bbe1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3b79a6d-601b-4348-8ac2-620145e0bbe1-maas-api-xks-ca\") pod \"f3b79a6d-601b-4348-8ac2-620145e0bbe1\" (UID: \"f3b79a6d-601b-4348-8ac2-620145e0bbe1\") "
Nov 01 03:53:54.637108 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:54.637078    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3b79a6d-601b-4348-8ac2-620145e0bbe1-kube-api-access-b8pgd\" (UniqueName: \"kubernetes.io/projected/f3b79a6d-601b-4348-8ac2-620145e0bbe1-kube-api-access-b8pgd\") pod \"f3b79a6d-601b-4348-8ac2-620145e0bbe1\" (UID: \"f3b79a6d-601b-4348-8ac2-620145e0bbe1\") "
Nov 01 03:53:54.637172 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:54.637111    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3b79a6d-601b-4348-8ac2-620145e0bbe1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3b79a6d-601b-4348-8ac2-620145e0bbe1-maas-api-openshift-ca\") pod \"f3b79a6d-601b-4348-8ac2-620145e0bbe1\" (UID: \"f3b79a6d-601b-4348-8ac2-620145e0bbe1\") "
Nov 01 03:53:54.637519 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:54.637488    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3b79a6d-601b-4348-8ac2-620145e0bbe1-maas-api-openshift-ca" pod "f3b79a6d-601b-4348-8ac2-620145e0bbe1" (UID: "f3b79a6d-601b-4348-8ac2-620145e0bbe1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:53:54.639064 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:54.639042    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3b79a6d-601b-4348-8ac2-620145e0bbe1-maas-api-xks-ca" pod "f3b79a6d-601b-4348-8ac2-620145e0bbe1" (UID: "f3b79a6d-601b-4348-8ac2-620145e0bbe1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:53:54.639173 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:54.639158    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b79a6d-601b-4348-8ac2-620145e0bbe1-kube-api-access-b8pgd" pod "f3b79a6d-601b-4348-8ac2-620145e0bbe1" (UID: "f3b79a6d-601b-4348-8ac2-620145e0bbe1"). InnerVolumeSpecName "kube-api-access-b8pgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:53:54.730593 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:54.730498    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"]
Nov 01 03:53:54.734761 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:54.734735    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-19faa5/distinct-w6-19faa5-kserve-7c7b9d8f8-z74d5"]
Nov 01 03:53:54.737999 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:54.737975    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3b79a6d-601b-4348-8ac2-620145e0bbe1-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:54.738100 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:54.738017    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8pgd\" (UniqueName: \"kubernetes.io/projected/f3b79a6d-601b-4348-8ac2-620145e0bbe1-kube-api-access-b8pgd\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:54.738100 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:54.738029    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3b79a6d-601b-4348-8ac2-620145e0bbe1-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:53:54.841195 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:54.841167    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"]
Nov 01 03:53:54.845267 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:54.845242    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-19faa5/simulator-w6-19faa5-kserve-5b5796d994-khm8t"]
Nov 01 03:53:55.459901 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:55.459868    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3fb7a87-qbxs5" event={"ID":"f3b79a6d-601b-4348-8ac2-620145e0bbe1","Type":"ContainerDied","Data":"fab6145085cd36ac247b58811f8f637ca58fccc19a3e7c8695a31cecefe8f697"}
Nov 01 03:53:55.459901 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:55.459894    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3fb7a87-qbxs5"
Nov 01 03:53:55.460119 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:55.459939    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fab6145085cd36ac247b58811f8f637ca58fccc19a3e7c8695a31cecefe8f697"
Nov 01 03:53:55.618739 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:55.618704    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13930c1e-2200-4412-b8d3-ae3e4126a851" path="/var/lib/kubelet/pods/13930c1e-2200-4412-b8d3-ae3e4126a851/volumes"
Nov 01 03:53:55.619153 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:55.619133    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" path="/var/lib/kubelet/pods/baf6d6ac-7399-4c2b-bf41-5423abbedef2/volumes"
Nov 01 03:53:55.619502 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:55.619489    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d658772c-b072-4513-a639-d19a94c78525" path="/var/lib/kubelet/pods/d658772c-b072-4513-a639-d19a94c78525/volumes"
Nov 01 03:53:58.406934 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:53:58.406897    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:54:06.566382 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:06.566343    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:54:08.065225 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:08.065187    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:54:08.353393 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:08.353359    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:54:09.030780 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:09.030743    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:54:15.698770 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:15.698712    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:54:15.890272 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:15.890231    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:54:16.746050 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:16.746014    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:54:16.896896 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:16.896853    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:54:16.950131 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:16.950089    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg"]
Nov 01 03:54:16.950363 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:16.950337    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg" podUID="1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f" containerName="istio-proxy" containerID="cri-o://119476fa35f6c3aafce8b0fb2c85e01ae6bcf2fa23d5366588d75967cec3b506" gracePeriod=30
Nov 01 03:54:17.000294 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:17.000208    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7"]
Nov 01 03:54:17.000520 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:17.000477    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7" podUID="4322b03e-6c6d-435d-bf3b-9e60757a5845" containerName="istio-proxy" containerID="cri-o://7e24f94513610119af73c3a07ae06698e072024f88709e07f4fb9a0ec7545d53" gracePeriod=30
Nov 01 03:54:17.003497 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:17.003467    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8"]
Nov 01 03:54:17.003869 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:17.003835    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8" podUID="3626111a-ef96-41b1-9185-849ab00524cc" containerName="istio-proxy" containerID="cri-o://33560a8f32dd5c660dd0e3db16c1155d51b530e45403c649d4cfb246c8f17b70" gracePeriod=30
Nov 01 03:54:17.049685 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:17.049658    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb"]
Nov 01 03:54:17.049856 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:17.049836    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb" podUID="96a82d86-3895-4362-88fa-2834dac3af47" containerName="istio-proxy" containerID="cri-o://47d28e890544e3839de55e9c64b4e9e823739ffdcf565f0c4f8600060265064a" gracePeriod=30
Nov 01 03:54:17.145956 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:17.145926    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:54:17.440420 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:17.440383    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:54:19.120714 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:54:19.120671    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:54:22.268045 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.268012    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg"
Nov 01 03:54:22.315338 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.315297    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7"
Nov 01 03:54:22.341720 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.341690    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4322b03e-6c6d-435d-bf3b-9e60757a5845-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4322b03e-6c6d-435d-bf3b-9e60757a5845-istio-data\") pod \"4322b03e-6c6d-435d-bf3b-9e60757a5845\" (UID: \"4322b03e-6c6d-435d-bf3b-9e60757a5845\") "
Nov 01 03:54:22.341862 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.341739    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4322b03e-6c6d-435d-bf3b-9e60757a5845-istio-token\" (UniqueName: \"kubernetes.io/projected/4322b03e-6c6d-435d-bf3b-9e60757a5845-istio-token\") pod \"4322b03e-6c6d-435d-bf3b-9e60757a5845\" (UID: \"4322b03e-6c6d-435d-bf3b-9e60757a5845\") "
Nov 01 03:54:22.341862 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.341785    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-kube-api-access-nlgg9\" (UniqueName: \"kubernetes.io/projected/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-kube-api-access-nlgg9\") pod \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\" (UID: \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\") "
Nov 01 03:54:22.341862 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.341814    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-credential-socket\") pod \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\" (UID: \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\") "
Nov 01 03:54:22.341862 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.341846    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4322b03e-6c6d-435d-bf3b-9e60757a5845-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4322b03e-6c6d-435d-bf3b-9e60757a5845-credential-socket\") pod \"4322b03e-6c6d-435d-bf3b-9e60757a5845\" (UID: \"4322b03e-6c6d-435d-bf3b-9e60757a5845\") "
Nov 01 03:54:22.342116 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.341903    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-workload-certs\") pod \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\" (UID: \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\") "
Nov 01 03:54:22.342116 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.341939    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-istio-podinfo\") pod \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\" (UID: \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\") "
Nov 01 03:54:22.342116 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.341957    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4322b03e-6c6d-435d-bf3b-9e60757a5845-istio-data" pod "4322b03e-6c6d-435d-bf3b-9e60757a5845" (UID: "4322b03e-6c6d-435d-bf3b-9e60757a5845"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:54:22.342116 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.341976    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-workload-socket\") pod \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\" (UID: \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\") "
Nov 01 03:54:22.342116 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.342034    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4322b03e-6c6d-435d-bf3b-9e60757a5845-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4322b03e-6c6d-435d-bf3b-9e60757a5845-workload-certs\") pod \"4322b03e-6c6d-435d-bf3b-9e60757a5845\" (UID: \"4322b03e-6c6d-435d-bf3b-9e60757a5845\") "
Nov 01 03:54:22.342116 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.342069    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-istio-data\") pod \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\" (UID: \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\") "
Nov 01 03:54:22.342116 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.342103    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4322b03e-6c6d-435d-bf3b-9e60757a5845-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4322b03e-6c6d-435d-bf3b-9e60757a5845-workload-socket\") pod \"4322b03e-6c6d-435d-bf3b-9e60757a5845\" (UID: \"4322b03e-6c6d-435d-bf3b-9e60757a5845\") "
Nov 01 03:54:22.342405 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.342124    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-credential-socket" pod "1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f" (UID: "1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:54:22.342405 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.342150    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-istiod-ca-cert\") pod \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\" (UID: \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\") "
Nov 01 03:54:22.342405 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.342177    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-workload-certs" pod "1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f" (UID: "1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:54:22.342405 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.342213    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-istio-envoy\") pod \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\" (UID: \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\") "
Nov 01 03:54:22.342405 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.342250    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4322b03e-6c6d-435d-bf3b-9e60757a5845-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4322b03e-6c6d-435d-bf3b-9e60757a5845-istiod-ca-cert\") pod \"4322b03e-6c6d-435d-bf3b-9e60757a5845\" (UID: \"4322b03e-6c6d-435d-bf3b-9e60757a5845\") "
Nov 01 03:54:22.342405 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.342307    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4322b03e-6c6d-435d-bf3b-9e60757a5845-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4322b03e-6c6d-435d-bf3b-9e60757a5845-istio-podinfo\") pod \"4322b03e-6c6d-435d-bf3b-9e60757a5845\" (UID: \"4322b03e-6c6d-435d-bf3b-9e60757a5845\") "
Nov 01 03:54:22.342405 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.342335    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-istio-token\" (UniqueName: \"kubernetes.io/projected/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-istio-token\") pod \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\" (UID: \"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f\") "
Nov 01 03:54:22.342405 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.342374    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4322b03e-6c6d-435d-bf3b-9e60757a5845-kube-api-access-xbfhx\" (UniqueName: \"kubernetes.io/projected/4322b03e-6c6d-435d-bf3b-9e60757a5845-kube-api-access-xbfhx\") pod \"4322b03e-6c6d-435d-bf3b-9e60757a5845\" (UID: \"4322b03e-6c6d-435d-bf3b-9e60757a5845\") "
Nov 01 03:54:22.342405 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.342401    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4322b03e-6c6d-435d-bf3b-9e60757a5845-credential-socket" pod "4322b03e-6c6d-435d-bf3b-9e60757a5845" (UID: "4322b03e-6c6d-435d-bf3b-9e60757a5845"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:54:22.342899 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.342418    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4322b03e-6c6d-435d-bf3b-9e60757a5845-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4322b03e-6c6d-435d-bf3b-9e60757a5845-istio-envoy\") pod \"4322b03e-6c6d-435d-bf3b-9e60757a5845\" (UID: \"4322b03e-6c6d-435d-bf3b-9e60757a5845\") "
Nov 01 03:54:22.342899 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.342467    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-istiod-ca-cert" pod "1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f" (UID: "1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:54:22.342899 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.342851    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4322b03e-6c6d-435d-bf3b-9e60757a5845-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.342899 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.342875    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.342899 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.342890    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4322b03e-6c6d-435d-bf3b-9e60757a5845-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.343166 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.342904    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.343166 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.342918    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.344582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.344528    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4322b03e-6c6d-435d-bf3b-9e60757a5845-istio-token" pod "4322b03e-6c6d-435d-bf3b-9e60757a5845" (UID: "4322b03e-6c6d-435d-bf3b-9e60757a5845"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:54:22.344713 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.344550    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-kube-api-access-nlgg9" pod "1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f" (UID: "1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f"). InnerVolumeSpecName "kube-api-access-nlgg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:54:22.344787 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.344736    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-workload-socket" pod "1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f" (UID: "1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:54:22.344841 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.344825    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-istio-data" pod "1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f" (UID: "1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:54:22.345028 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.345006    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4322b03e-6c6d-435d-bf3b-9e60757a5845-workload-socket" pod "4322b03e-6c6d-435d-bf3b-9e60757a5845" (UID: "4322b03e-6c6d-435d-bf3b-9e60757a5845"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:54:22.345111 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.345006    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-istio-podinfo" pod "1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f" (UID: "1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:54:22.345111 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.345027    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-istio-token" pod "1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f" (UID: "1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:54:22.345111 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.345043    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4322b03e-6c6d-435d-bf3b-9e60757a5845-workload-certs" pod "4322b03e-6c6d-435d-bf3b-9e60757a5845" (UID: "4322b03e-6c6d-435d-bf3b-9e60757a5845"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:54:22.345250 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.345192    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4322b03e-6c6d-435d-bf3b-9e60757a5845-istio-podinfo" pod "4322b03e-6c6d-435d-bf3b-9e60757a5845" (UID: "4322b03e-6c6d-435d-bf3b-9e60757a5845"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:54:22.345299 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.345261    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8"
Nov 01 03:54:22.345402 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.345382    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4322b03e-6c6d-435d-bf3b-9e60757a5845-istiod-ca-cert" pod "4322b03e-6c6d-435d-bf3b-9e60757a5845" (UID: "4322b03e-6c6d-435d-bf3b-9e60757a5845"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:54:22.345606 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.345582    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4322b03e-6c6d-435d-bf3b-9e60757a5845-istio-envoy" pod "4322b03e-6c6d-435d-bf3b-9e60757a5845" (UID: "4322b03e-6c6d-435d-bf3b-9e60757a5845"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:54:22.346428 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.346401    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-istio-envoy" pod "1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f" (UID: "1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:54:22.346853 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.346827    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4322b03e-6c6d-435d-bf3b-9e60757a5845-kube-api-access-xbfhx" pod "4322b03e-6c6d-435d-bf3b-9e60757a5845" (UID: "4322b03e-6c6d-435d-bf3b-9e60757a5845"). InnerVolumeSpecName "kube-api-access-xbfhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:54:22.353684 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.353651    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb"
Nov 01 03:54:22.443199 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.443170    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3626111a-ef96-41b1-9185-849ab00524cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3626111a-ef96-41b1-9185-849ab00524cc-workload-socket\") pod \"3626111a-ef96-41b1-9185-849ab00524cc\" (UID: \"3626111a-ef96-41b1-9185-849ab00524cc\") "
Nov 01 03:54:22.443199 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.443202    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96a82d86-3895-4362-88fa-2834dac3af47-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96a82d86-3895-4362-88fa-2834dac3af47-istio-data\") pod \"96a82d86-3895-4362-88fa-2834dac3af47\" (UID: \"96a82d86-3895-4362-88fa-2834dac3af47\") "
Nov 01 03:54:22.443442 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.443252    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3626111a-ef96-41b1-9185-849ab00524cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3626111a-ef96-41b1-9185-849ab00524cc-istio-podinfo\") pod \"3626111a-ef96-41b1-9185-849ab00524cc\" (UID: \"3626111a-ef96-41b1-9185-849ab00524cc\") "
Nov 01 03:54:22.443442 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.443283    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96a82d86-3895-4362-88fa-2834dac3af47-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96a82d86-3895-4362-88fa-2834dac3af47-istio-envoy\") pod \"96a82d86-3895-4362-88fa-2834dac3af47\" (UID: \"96a82d86-3895-4362-88fa-2834dac3af47\") "
Nov 01 03:54:22.443442 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.443309    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3626111a-ef96-41b1-9185-849ab00524cc-istio-token\" (UniqueName: \"kubernetes.io/projected/3626111a-ef96-41b1-9185-849ab00524cc-istio-token\") pod \"3626111a-ef96-41b1-9185-849ab00524cc\" (UID: \"3626111a-ef96-41b1-9185-849ab00524cc\") "
Nov 01 03:54:22.443442 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.443341    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96a82d86-3895-4362-88fa-2834dac3af47-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96a82d86-3895-4362-88fa-2834dac3af47-istiod-ca-cert\") pod \"96a82d86-3895-4362-88fa-2834dac3af47\" (UID: \"96a82d86-3895-4362-88fa-2834dac3af47\") "
Nov 01 03:54:22.443442 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.443367    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3626111a-ef96-41b1-9185-849ab00524cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3626111a-ef96-41b1-9185-849ab00524cc-workload-certs\") pod \"3626111a-ef96-41b1-9185-849ab00524cc\" (UID: \"3626111a-ef96-41b1-9185-849ab00524cc\") "
Nov 01 03:54:22.443442 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.443408    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3626111a-ef96-41b1-9185-849ab00524cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3626111a-ef96-41b1-9185-849ab00524cc-istio-envoy\") pod \"3626111a-ef96-41b1-9185-849ab00524cc\" (UID: \"3626111a-ef96-41b1-9185-849ab00524cc\") "
Nov 01 03:54:22.444315 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.443445    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96a82d86-3895-4362-88fa-2834dac3af47-kube-api-access-q9mbv\" (UniqueName: \"kubernetes.io/projected/96a82d86-3895-4362-88fa-2834dac3af47-kube-api-access-q9mbv\") pod \"96a82d86-3895-4362-88fa-2834dac3af47\" (UID: \"96a82d86-3895-4362-88fa-2834dac3af47\") "
Nov 01 03:54:22.444315 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.443471    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96a82d86-3895-4362-88fa-2834dac3af47-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a82d86-3895-4362-88fa-2834dac3af47-credential-socket\") pod \"96a82d86-3895-4362-88fa-2834dac3af47\" (UID: \"96a82d86-3895-4362-88fa-2834dac3af47\") "
Nov 01 03:54:22.444315 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.443501    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3626111a-ef96-41b1-9185-849ab00524cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3626111a-ef96-41b1-9185-849ab00524cc-credential-socket\") pod \"3626111a-ef96-41b1-9185-849ab00524cc\" (UID: \"3626111a-ef96-41b1-9185-849ab00524cc\") "
Nov 01 03:54:22.444315 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.443534    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96a82d86-3895-4362-88fa-2834dac3af47-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96a82d86-3895-4362-88fa-2834dac3af47-istio-podinfo\") pod \"96a82d86-3895-4362-88fa-2834dac3af47\" (UID: \"96a82d86-3895-4362-88fa-2834dac3af47\") "
Nov 01 03:54:22.444315 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.443593    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96a82d86-3895-4362-88fa-2834dac3af47-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a82d86-3895-4362-88fa-2834dac3af47-workload-socket\") pod \"96a82d86-3895-4362-88fa-2834dac3af47\" (UID: \"96a82d86-3895-4362-88fa-2834dac3af47\") "
Nov 01 03:54:22.444315 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.443641    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3626111a-ef96-41b1-9185-849ab00524cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3626111a-ef96-41b1-9185-849ab00524cc-istio-data\") pod \"3626111a-ef96-41b1-9185-849ab00524cc\" (UID: \"3626111a-ef96-41b1-9185-849ab00524cc\") "
Nov 01 03:54:22.444315 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.443674    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3626111a-ef96-41b1-9185-849ab00524cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3626111a-ef96-41b1-9185-849ab00524cc-istiod-ca-cert\") pod \"3626111a-ef96-41b1-9185-849ab00524cc\" (UID: \"3626111a-ef96-41b1-9185-849ab00524cc\") "
Nov 01 03:54:22.444315 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.443715    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96a82d86-3895-4362-88fa-2834dac3af47-istio-token\" (UniqueName: \"kubernetes.io/projected/96a82d86-3895-4362-88fa-2834dac3af47-istio-token\") pod \"96a82d86-3895-4362-88fa-2834dac3af47\" (UID: \"96a82d86-3895-4362-88fa-2834dac3af47\") "
Nov 01 03:54:22.444315 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.443768    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96a82d86-3895-4362-88fa-2834dac3af47-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96a82d86-3895-4362-88fa-2834dac3af47-workload-certs\") pod \"96a82d86-3895-4362-88fa-2834dac3af47\" (UID: \"96a82d86-3895-4362-88fa-2834dac3af47\") "
Nov 01 03:54:22.444315 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.443808    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3626111a-ef96-41b1-9185-849ab00524cc-kube-api-access-4t6lt\" (UniqueName: \"kubernetes.io/projected/3626111a-ef96-41b1-9185-849ab00524cc-kube-api-access-4t6lt\") pod \"3626111a-ef96-41b1-9185-849ab00524cc\" (UID: \"3626111a-ef96-41b1-9185-849ab00524cc\") "
Nov 01 03:54:22.444315 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.443531    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a82d86-3895-4362-88fa-2834dac3af47-istio-data" pod "96a82d86-3895-4362-88fa-2834dac3af47" (UID: "96a82d86-3895-4362-88fa-2834dac3af47"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:54:22.444315 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.444200    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4322b03e-6c6d-435d-bf3b-9e60757a5845-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.444315 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.444219    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.444315 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.444236    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbfhx\" (UniqueName: \"kubernetes.io/projected/4322b03e-6c6d-435d-bf3b-9e60757a5845-kube-api-access-xbfhx\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.444315 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.444250    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4322b03e-6c6d-435d-bf3b-9e60757a5845-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.444315 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.444265    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4322b03e-6c6d-435d-bf3b-9e60757a5845-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.444315 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.444279    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlgg9\" (UniqueName: \"kubernetes.io/projected/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-kube-api-access-nlgg9\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.444315 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.444292    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.444315 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.444306    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.444315 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.444320    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96a82d86-3895-4362-88fa-2834dac3af47-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.444315 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.444334    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4322b03e-6c6d-435d-bf3b-9e60757a5845-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.445516 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.444348    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.445516 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.444361    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4322b03e-6c6d-435d-bf3b-9e60757a5845-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.445516 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.444374    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.445516 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.444388    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4322b03e-6c6d-435d-bf3b-9e60757a5845-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.445516 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.444619    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96a82d86-3895-4362-88fa-2834dac3af47-istiod-ca-cert" pod "96a82d86-3895-4362-88fa-2834dac3af47" (UID: "96a82d86-3895-4362-88fa-2834dac3af47"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:54:22.445516 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.444626    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a82d86-3895-4362-88fa-2834dac3af47-workload-socket" pod "96a82d86-3895-4362-88fa-2834dac3af47" (UID: "96a82d86-3895-4362-88fa-2834dac3af47"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:54:22.445516 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.443879    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3626111a-ef96-41b1-9185-849ab00524cc-workload-socket" pod "3626111a-ef96-41b1-9185-849ab00524cc" (UID: "3626111a-ef96-41b1-9185-849ab00524cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:54:22.445516 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.444100    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3626111a-ef96-41b1-9185-849ab00524cc-istio-data" pod "3626111a-ef96-41b1-9185-849ab00524cc" (UID: "3626111a-ef96-41b1-9185-849ab00524cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:54:22.445516 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.443723    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3626111a-ef96-41b1-9185-849ab00524cc-workload-certs" pod "3626111a-ef96-41b1-9185-849ab00524cc" (UID: "3626111a-ef96-41b1-9185-849ab00524cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:54:22.445516 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.445045    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a82d86-3895-4362-88fa-2834dac3af47-workload-certs" pod "96a82d86-3895-4362-88fa-2834dac3af47" (UID: "96a82d86-3895-4362-88fa-2834dac3af47"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:54:22.445516 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.445406    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3626111a-ef96-41b1-9185-849ab00524cc-istiod-ca-cert" pod "3626111a-ef96-41b1-9185-849ab00524cc" (UID: "3626111a-ef96-41b1-9185-849ab00524cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:54:22.446259 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.445678    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a82d86-3895-4362-88fa-2834dac3af47-credential-socket" pod "96a82d86-3895-4362-88fa-2834dac3af47" (UID: "96a82d86-3895-4362-88fa-2834dac3af47"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:54:22.446259 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.445735    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3626111a-ef96-41b1-9185-849ab00524cc-credential-socket" pod "3626111a-ef96-41b1-9185-849ab00524cc" (UID: "3626111a-ef96-41b1-9185-849ab00524cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:54:22.446935 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.446812    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3626111a-ef96-41b1-9185-849ab00524cc-istio-token" pod "3626111a-ef96-41b1-9185-849ab00524cc" (UID: "3626111a-ef96-41b1-9185-849ab00524cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:54:22.446935 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.446896    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a82d86-3895-4362-88fa-2834dac3af47-istio-envoy" pod "96a82d86-3895-4362-88fa-2834dac3af47" (UID: "96a82d86-3895-4362-88fa-2834dac3af47"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:54:22.446935 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.446909    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3626111a-ef96-41b1-9185-849ab00524cc-istio-envoy" pod "3626111a-ef96-41b1-9185-849ab00524cc" (UID: "3626111a-ef96-41b1-9185-849ab00524cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 03:54:22.447193 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.447118    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3626111a-ef96-41b1-9185-849ab00524cc-kube-api-access-4t6lt" pod "3626111a-ef96-41b1-9185-849ab00524cc" (UID: "3626111a-ef96-41b1-9185-849ab00524cc"). InnerVolumeSpecName "kube-api-access-4t6lt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:54:22.447678 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.447649    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96a82d86-3895-4362-88fa-2834dac3af47-istio-token" pod "96a82d86-3895-4362-88fa-2834dac3af47" (UID: "96a82d86-3895-4362-88fa-2834dac3af47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:54:22.447823 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.447804    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3626111a-ef96-41b1-9185-849ab00524cc-istio-podinfo" pod "3626111a-ef96-41b1-9185-849ab00524cc" (UID: "3626111a-ef96-41b1-9185-849ab00524cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:54:22.448449 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.448426    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96a82d86-3895-4362-88fa-2834dac3af47-kube-api-access-q9mbv" pod "96a82d86-3895-4362-88fa-2834dac3af47" (UID: "96a82d86-3895-4362-88fa-2834dac3af47"). InnerVolumeSpecName "kube-api-access-q9mbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:54:22.448666 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.448645    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96a82d86-3895-4362-88fa-2834dac3af47-istio-podinfo" pod "96a82d86-3895-4362-88fa-2834dac3af47" (UID: "96a82d86-3895-4362-88fa-2834dac3af47"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 03:54:22.545278 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.545196    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96a82d86-3895-4362-88fa-2834dac3af47-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.545278 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.545227    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3626111a-ef96-41b1-9185-849ab00524cc-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.545278 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.545237    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96a82d86-3895-4362-88fa-2834dac3af47-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.545278 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.545247    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3626111a-ef96-41b1-9185-849ab00524cc-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.545278 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.545255    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3626111a-ef96-41b1-9185-849ab00524cc-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.545278 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.545265    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9mbv\" (UniqueName: \"kubernetes.io/projected/96a82d86-3895-4362-88fa-2834dac3af47-kube-api-access-q9mbv\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.545278 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.545275    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a82d86-3895-4362-88fa-2834dac3af47-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.545278 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.545283    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3626111a-ef96-41b1-9185-849ab00524cc-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.545685 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.545292    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96a82d86-3895-4362-88fa-2834dac3af47-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.545685 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.545303    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a82d86-3895-4362-88fa-2834dac3af47-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.545685 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.545311    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3626111a-ef96-41b1-9185-849ab00524cc-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.545685 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.545319    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3626111a-ef96-41b1-9185-849ab00524cc-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.545685 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.545327    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96a82d86-3895-4362-88fa-2834dac3af47-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.545685 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.545336    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96a82d86-3895-4362-88fa-2834dac3af47-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.545685 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.545349    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t6lt\" (UniqueName: \"kubernetes.io/projected/3626111a-ef96-41b1-9185-849ab00524cc-kube-api-access-4t6lt\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.545685 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.545357    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3626111a-ef96-41b1-9185-849ab00524cc-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.545685 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.545365    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3626111a-ef96-41b1-9185-849ab00524cc-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:22.639935 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.639901    4930 generic.go:356] "Generic (PLEG): container finished" podID="1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f" containerID="119476fa35f6c3aafce8b0fb2c85e01ae6bcf2fa23d5366588d75967cec3b506" exitCode=0
Nov 01 03:54:22.640136 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.639947    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg"
Nov 01 03:54:22.640136 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.639982    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg" event={"ID":"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f","Type":"ContainerDied","Data":"119476fa35f6c3aafce8b0fb2c85e01ae6bcf2fa23d5366588d75967cec3b506"}
Nov 01 03:54:22.640136 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.640037    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg" event={"ID":"1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f","Type":"ContainerDied","Data":"125558f2e1542a144031b7dd605c0d0d74d6c6c385d83037c9b8cc4e379efe9d"}
Nov 01 03:54:22.640136 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.640060    4930 scope.go:122] "RemoveContainer" containerID="119476fa35f6c3aafce8b0fb2c85e01ae6bcf2fa23d5366588d75967cec3b506"
Nov 01 03:54:22.641331 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.641306    4930 generic.go:356] "Generic (PLEG): container finished" podID="4322b03e-6c6d-435d-bf3b-9e60757a5845" containerID="7e24f94513610119af73c3a07ae06698e072024f88709e07f4fb9a0ec7545d53" exitCode=0
Nov 01 03:54:22.641428 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.641337    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7" event={"ID":"4322b03e-6c6d-435d-bf3b-9e60757a5845","Type":"ContainerDied","Data":"7e24f94513610119af73c3a07ae06698e072024f88709e07f4fb9a0ec7545d53"}
Nov 01 03:54:22.641428 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.641350    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7"
Nov 01 03:54:22.641428 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.641380    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7" event={"ID":"4322b03e-6c6d-435d-bf3b-9e60757a5845","Type":"ContainerDied","Data":"6d5f17dc9e8e0da3f9955a5d1d0304278625c79c678802ba7638e63c93e4c229"}
Nov 01 03:54:22.643150 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.643125    4930 generic.go:356] "Generic (PLEG): container finished" podID="3626111a-ef96-41b1-9185-849ab00524cc" containerID="33560a8f32dd5c660dd0e3db16c1155d51b530e45403c649d4cfb246c8f17b70" exitCode=0
Nov 01 03:54:22.643286 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.643161    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8" event={"ID":"3626111a-ef96-41b1-9185-849ab00524cc","Type":"ContainerDied","Data":"33560a8f32dd5c660dd0e3db16c1155d51b530e45403c649d4cfb246c8f17b70"}
Nov 01 03:54:22.643286 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.643183    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8"
Nov 01 03:54:22.643286 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.643203    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8" event={"ID":"3626111a-ef96-41b1-9185-849ab00524cc","Type":"ContainerDied","Data":"b09c3e127a1806514a797447fa130ebb2adeaf6de20ff85bbc33b1953f0fe611"}
Nov 01 03:54:22.644891 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.644860    4930 generic.go:356] "Generic (PLEG): container finished" podID="96a82d86-3895-4362-88fa-2834dac3af47" containerID="47d28e890544e3839de55e9c64b4e9e823739ffdcf565f0c4f8600060265064a" exitCode=0
Nov 01 03:54:22.645027 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.644900    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb" event={"ID":"96a82d86-3895-4362-88fa-2834dac3af47","Type":"ContainerDied","Data":"47d28e890544e3839de55e9c64b4e9e823739ffdcf565f0c4f8600060265064a"}
Nov 01 03:54:22.645027 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.644963    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb" event={"ID":"96a82d86-3895-4362-88fa-2834dac3af47","Type":"ContainerDied","Data":"c3c62671bc212dfe22465dc901780e673256b2c5b84294d4af2ee1a4265f28a3"}
Nov 01 03:54:22.645027 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.644908    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb"
Nov 01 03:54:22.653938 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.653792    4930 scope.go:122] "RemoveContainer" containerID="119476fa35f6c3aafce8b0fb2c85e01ae6bcf2fa23d5366588d75967cec3b506"
Nov 01 03:54:22.654107 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:54:22.654084    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"119476fa35f6c3aafce8b0fb2c85e01ae6bcf2fa23d5366588d75967cec3b506\": container with ID starting with 119476fa35f6c3aafce8b0fb2c85e01ae6bcf2fa23d5366588d75967cec3b506 not found: ID does not exist" containerID="119476fa35f6c3aafce8b0fb2c85e01ae6bcf2fa23d5366588d75967cec3b506"
Nov 01 03:54:22.654161 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.654118    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"119476fa35f6c3aafce8b0fb2c85e01ae6bcf2fa23d5366588d75967cec3b506"} err="failed to get container status \"119476fa35f6c3aafce8b0fb2c85e01ae6bcf2fa23d5366588d75967cec3b506\": rpc error: code = NotFound desc = could not find container \"119476fa35f6c3aafce8b0fb2c85e01ae6bcf2fa23d5366588d75967cec3b506\": container with ID starting with 119476fa35f6c3aafce8b0fb2c85e01ae6bcf2fa23d5366588d75967cec3b506 not found: ID does not exist"
Nov 01 03:54:22.654161 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.654142    4930 scope.go:122] "RemoveContainer" containerID="7e24f94513610119af73c3a07ae06698e072024f88709e07f4fb9a0ec7545d53"
Nov 01 03:54:22.664251 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.664234    4930 scope.go:122] "RemoveContainer" containerID="7e24f94513610119af73c3a07ae06698e072024f88709e07f4fb9a0ec7545d53"
Nov 01 03:54:22.664507 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:54:22.664489    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e24f94513610119af73c3a07ae06698e072024f88709e07f4fb9a0ec7545d53\": container with ID starting with 7e24f94513610119af73c3a07ae06698e072024f88709e07f4fb9a0ec7545d53 not found: ID does not exist" containerID="7e24f94513610119af73c3a07ae06698e072024f88709e07f4fb9a0ec7545d53"
Nov 01 03:54:22.664593 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.664513    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e24f94513610119af73c3a07ae06698e072024f88709e07f4fb9a0ec7545d53"} err="failed to get container status \"7e24f94513610119af73c3a07ae06698e072024f88709e07f4fb9a0ec7545d53\": rpc error: code = NotFound desc = could not find container \"7e24f94513610119af73c3a07ae06698e072024f88709e07f4fb9a0ec7545d53\": container with ID starting with 7e24f94513610119af73c3a07ae06698e072024f88709e07f4fb9a0ec7545d53 not found: ID does not exist"
Nov 01 03:54:22.664593 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.664529    4930 scope.go:122] "RemoveContainer" containerID="33560a8f32dd5c660dd0e3db16c1155d51b530e45403c649d4cfb246c8f17b70"
Nov 01 03:54:22.673139 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.673121    4930 scope.go:122] "RemoveContainer" containerID="33560a8f32dd5c660dd0e3db16c1155d51b530e45403c649d4cfb246c8f17b70"
Nov 01 03:54:22.673394 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:54:22.673371    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33560a8f32dd5c660dd0e3db16c1155d51b530e45403c649d4cfb246c8f17b70\": container with ID starting with 33560a8f32dd5c660dd0e3db16c1155d51b530e45403c649d4cfb246c8f17b70 not found: ID does not exist" containerID="33560a8f32dd5c660dd0e3db16c1155d51b530e45403c649d4cfb246c8f17b70"
Nov 01 03:54:22.673449 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.673402    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33560a8f32dd5c660dd0e3db16c1155d51b530e45403c649d4cfb246c8f17b70"} err="failed to get container status \"33560a8f32dd5c660dd0e3db16c1155d51b530e45403c649d4cfb246c8f17b70\": rpc error: code = NotFound desc = could not find container \"33560a8f32dd5c660dd0e3db16c1155d51b530e45403c649d4cfb246c8f17b70\": container with ID starting with 33560a8f32dd5c660dd0e3db16c1155d51b530e45403c649d4cfb246c8f17b70 not found: ID does not exist"
Nov 01 03:54:22.673449 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.673420    4930 scope.go:122] "RemoveContainer" containerID="47d28e890544e3839de55e9c64b4e9e823739ffdcf565f0c4f8600060265064a"
Nov 01 03:54:22.684674 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.684646    4930 scope.go:122] "RemoveContainer" containerID="47d28e890544e3839de55e9c64b4e9e823739ffdcf565f0c4f8600060265064a"
Nov 01 03:54:22.685020 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:54:22.685001    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47d28e890544e3839de55e9c64b4e9e823739ffdcf565f0c4f8600060265064a\": container with ID starting with 47d28e890544e3839de55e9c64b4e9e823739ffdcf565f0c4f8600060265064a not found: ID does not exist" containerID="47d28e890544e3839de55e9c64b4e9e823739ffdcf565f0c4f8600060265064a"
Nov 01 03:54:22.685087 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.685028    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47d28e890544e3839de55e9c64b4e9e823739ffdcf565f0c4f8600060265064a"} err="failed to get container status \"47d28e890544e3839de55e9c64b4e9e823739ffdcf565f0c4f8600060265064a\": rpc error: code = NotFound desc = could not find container \"47d28e890544e3839de55e9c64b4e9e823739ffdcf565f0c4f8600060265064a\": container with ID starting with 47d28e890544e3839de55e9c64b4e9e823739ffdcf565f0c4f8600060265064a not found: ID does not exist"
Nov 01 03:54:22.912199 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.912165    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8"]
Nov 01 03:54:22.916314 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:22.916284    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-b97j8"]
Nov 01 03:54:23.025981 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:23.025952    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7"]
Nov 01 03:54:23.031198 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:23.031174    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cchc7"]
Nov 01 03:54:23.140935 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:23.140905    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb"]
Nov 01 03:54:23.147174 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:23.147148    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-zlfnb"]
Nov 01 03:54:23.255328 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:23.255239    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg"]
Nov 01 03:54:23.259430 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:23.259401    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cxhqg"]
Nov 01 03:54:23.618279 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:23.618246    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f" path="/var/lib/kubelet/pods/1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f/volumes"
Nov 01 03:54:23.618741 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:23.618724    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3626111a-ef96-41b1-9185-849ab00524cc" path="/var/lib/kubelet/pods/3626111a-ef96-41b1-9185-849ab00524cc/volumes"
Nov 01 03:54:23.619140 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:23.619128    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4322b03e-6c6d-435d-bf3b-9e60757a5845" path="/var/lib/kubelet/pods/4322b03e-6c6d-435d-bf3b-9e60757a5845/volumes"
Nov 01 03:54:23.619880 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:23.619864    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96a82d86-3895-4362-88fa-2834dac3af47" path="/var/lib/kubelet/pods/96a82d86-3895-4362-88fa-2834dac3af47/volumes"
Nov 01 03:54:23.788083 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:23.788047    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:54:24.681815 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:24.681784    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5d96cbcb67-ztb9c"]
Nov 01 03:54:24.682246 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:24.682148    4930 cpu_manager.go:414] "containerMap: removing container" podUID="f3b79a6d-601b-4348-8ac2-620145e0bbe1" containerName="revoke-keys"
Nov 01 03:54:24.682246 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:24.682160    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b79a6d-601b-4348-8ac2-620145e0bbe1" containerName="revoke-keys"
Nov 01 03:54:24.682246 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:24.682177    4930 cpu_manager.go:414] "containerMap: removing container" podUID="3626111a-ef96-41b1-9185-849ab00524cc" containerName="istio-proxy"
Nov 01 03:54:24.682246 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:24.682182    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3626111a-ef96-41b1-9185-849ab00524cc" containerName="istio-proxy"
Nov 01 03:54:24.682246 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:24.682196    4930 cpu_manager.go:414] "containerMap: removing container" podUID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" containerName="main"
Nov 01 03:54:24.682246 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:24.682201    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" containerName="main"
Nov 01 03:54:24.682246 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:24.682209    4930 cpu_manager.go:414] "containerMap: removing container" podUID="96a82d86-3895-4362-88fa-2834dac3af47" containerName="istio-proxy"
Nov 01 03:54:24.682246 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:24.682215    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96a82d86-3895-4362-88fa-2834dac3af47" containerName="istio-proxy"
Nov 01 03:54:24.682246 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:24.682221    4930 cpu_manager.go:414] "containerMap: removing container" podUID="1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f" containerName="istio-proxy"
Nov 01 03:54:24.682246 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:24.682225    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f" containerName="istio-proxy"
Nov 01 03:54:24.682246 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:24.682232    4930 cpu_manager.go:414] "containerMap: removing container" podUID="d658772c-b072-4513-a639-d19a94c78525" containerName="main"
Nov 01 03:54:24.682246 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:24.682236    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d658772c-b072-4513-a639-d19a94c78525" containerName="main"
Nov 01 03:54:24.682246 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:24.682247    4930 cpu_manager.go:414] "containerMap: removing container" podUID="4322b03e-6c6d-435d-bf3b-9e60757a5845" containerName="istio-proxy"
Nov 01 03:54:24.682246 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:24.682252    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4322b03e-6c6d-435d-bf3b-9e60757a5845" containerName="istio-proxy"
Nov 01 03:54:24.682756 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:24.682260    4930 cpu_manager.go:414] "containerMap: removing container" podUID="13930c1e-2200-4412-b8d3-ae3e4126a851" containerName="main"
Nov 01 03:54:24.682756 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:24.682265    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13930c1e-2200-4412-b8d3-ae3e4126a851" containerName="main"
Nov 01 03:54:24.682756 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:24.682328    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b79a6d-601b-4348-8ac2-620145e0bbe1" containerName="revoke-keys"
Nov 01 03:54:24.682756 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:24.682337    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="3626111a-ef96-41b1-9185-849ab00524cc" containerName="istio-proxy"
Nov 01 03:54:24.682756 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:24.682344    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="baf6d6ac-7399-4c2b-bf41-5423abbedef2" containerName="main"
Nov 01 03:54:24.682756 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:24.682351    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="96a82d86-3895-4362-88fa-2834dac3af47" containerName="istio-proxy"
Nov 01 03:54:24.682756 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:24.682356    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a4480eb-b9d2-4f5c-8dbc-dbcbfe88671f" containerName="istio-proxy"
Nov 01 03:54:24.682756 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:24.682361    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="d658772c-b072-4513-a639-d19a94c78525" containerName="main"
Nov 01 03:54:24.682756 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:24.682366    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="4322b03e-6c6d-435d-bf3b-9e60757a5845" containerName="istio-proxy"
Nov 01 03:54:24.682756 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:24.682371    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="13930c1e-2200-4412-b8d3-ae3e4126a851" containerName="main"
Nov 01 03:54:24.686774 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:24.686756    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d96cbcb67-ztb9c"
Nov 01 03:54:24.761375 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:24.761337    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9f3145b9-b972-4624-8c0b-187b65d4c10e-webhook-cert\") pod \"maas-controller-5d96cbcb67-ztb9c\" (UID: \"9f3145b9-b972-4624-8c0b-187b65d4c10e\") " pod="opendatahub/maas-controller-5d96cbcb67-ztb9c"
Nov 01 03:54:24.761375 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:24.761375    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jkgj\" (UniqueName: \"kubernetes.io/projected/9f3145b9-b972-4624-8c0b-187b65d4c10e-kube-api-access-4jkgj\") pod \"maas-controller-5d96cbcb67-ztb9c\" (UID: \"9f3145b9-b972-4624-8c0b-187b65d4c10e\") " pod="opendatahub/maas-controller-5d96cbcb67-ztb9c"
Nov 01 03:54:24.761635 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:24.761439    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f3145b9-b972-4624-8c0b-187b65d4c10e-metrics-certs\") pod \"maas-controller-5d96cbcb67-ztb9c\" (UID: \"9f3145b9-b972-4624-8c0b-187b65d4c10e\") " pod="opendatahub/maas-controller-5d96cbcb67-ztb9c"
Nov 01 03:54:24.794090 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:24.794056    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 01 03:54:24.794329 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:24.794075    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 01 03:54:24.794432 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:24.794082    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kdlhc\""
Nov 01 03:54:24.794432 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:24.794089    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 01 03:54:24.794432 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:24.794148    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 01 03:54:24.794432 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:24.794253    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 01 03:54:24.797411 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:24.797387    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5d96cbcb67-ztb9c"]
Nov 01 03:54:24.861920 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:24.861891    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f3145b9-b972-4624-8c0b-187b65d4c10e-metrics-certs\") pod \"maas-controller-5d96cbcb67-ztb9c\" (UID: \"9f3145b9-b972-4624-8c0b-187b65d4c10e\") " pod="opendatahub/maas-controller-5d96cbcb67-ztb9c"
Nov 01 03:54:24.862099 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:24.861993    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9f3145b9-b972-4624-8c0b-187b65d4c10e-webhook-cert\") pod \"maas-controller-5d96cbcb67-ztb9c\" (UID: \"9f3145b9-b972-4624-8c0b-187b65d4c10e\") " pod="opendatahub/maas-controller-5d96cbcb67-ztb9c"
Nov 01 03:54:24.862099 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:24.862020    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jkgj\" (UniqueName: \"kubernetes.io/projected/9f3145b9-b972-4624-8c0b-187b65d4c10e-kube-api-access-4jkgj\") pod \"maas-controller-5d96cbcb67-ztb9c\" (UID: \"9f3145b9-b972-4624-8c0b-187b65d4c10e\") " pod="opendatahub/maas-controller-5d96cbcb67-ztb9c"
Nov 01 03:54:24.864403 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:24.864376    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f3145b9-b972-4624-8c0b-187b65d4c10e-metrics-certs\") pod \"maas-controller-5d96cbcb67-ztb9c\" (UID: \"9f3145b9-b972-4624-8c0b-187b65d4c10e\") " pod="opendatahub/maas-controller-5d96cbcb67-ztb9c"
Nov 01 03:54:24.864403 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:24.864400    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9f3145b9-b972-4624-8c0b-187b65d4c10e-webhook-cert\") pod \"maas-controller-5d96cbcb67-ztb9c\" (UID: \"9f3145b9-b972-4624-8c0b-187b65d4c10e\") " pod="opendatahub/maas-controller-5d96cbcb67-ztb9c"
Nov 01 03:54:24.923038 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:24.923006    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jkgj\" (UniqueName: \"kubernetes.io/projected/9f3145b9-b972-4624-8c0b-187b65d4c10e-kube-api-access-4jkgj\") pod \"maas-controller-5d96cbcb67-ztb9c\" (UID: \"9f3145b9-b972-4624-8c0b-187b65d4c10e\") " pod="opendatahub/maas-controller-5d96cbcb67-ztb9c"
Nov 01 03:54:24.999739 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:24.999663    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d96cbcb67-ztb9c"
Nov 01 03:54:25.431257 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:25.431228    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5d96cbcb67-ztb9c"]
Nov 01 03:54:25.432039 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:54:25.432013    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f3145b9_b972_4624_8c0b_187b65d4c10e.slice/crio-5637a2b5b1d267e5c609aefe87da5eb0a5c9606749c5d720f155480591038b05 WatchSource:0}: Error finding container 5637a2b5b1d267e5c609aefe87da5eb0a5c9606749c5d720f155480591038b05: Status 404 returned error can't find the container with id 5637a2b5b1d267e5c609aefe87da5eb0a5c9606749c5d720f155480591038b05
Nov 01 03:54:25.433303 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:25.433283    4930 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 03:54:25.667551 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:25.667514    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d96cbcb67-ztb9c" event={"ID":"9f3145b9-b972-4624-8c0b-187b65d4c10e","Type":"ContainerStarted","Data":"5637a2b5b1d267e5c609aefe87da5eb0a5c9606749c5d720f155480591038b05"}
Nov 01 03:54:28.692685 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:28.692588    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d96cbcb67-ztb9c" event={"ID":"9f3145b9-b972-4624-8c0b-187b65d4c10e","Type":"ContainerStarted","Data":"cb97936d662d428f7d5f1b0aa4064d4506e3c97431e020e74ea16a1705e71fde"}
Nov 01 03:54:28.693181 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:28.692881    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5d96cbcb67-ztb9c"
Nov 01 03:54:28.813388 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:28.813323    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5d96cbcb67-ztb9c" podStartSLOduration=1.8310010989999999 podStartE2EDuration="4.813304381s" totalImagesPullingTime="2.982303282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:54:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:54:28.809988763 +0000 UTC m=+2873.332995352" watchObservedRunningTime="2026-11-01 03:54:28.813304381 +0000 UTC m=+2873.336310970"
Nov 01 03:54:39.702423 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:39.702385    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5d96cbcb67-ztb9c"
Nov 01 03:54:40.165497 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:40.165456    4930 scope.go:122] "RemoveContainer" containerID="60668cc1f4f36b471432208b7e237f66602c76edb4c74ff4ad14a81189f405b0"
Nov 01 03:54:40.174666 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:40.174642    4930 scope.go:122] "RemoveContainer" containerID="bd921bea64c3d77a46fa8973e88ace2ccb55e7588411d36ac93e9c4749c841fa"
Nov 01 03:54:40.183118 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:40.183098    4930 scope.go:122] "RemoveContainer" containerID="9ab10151b42588d9660052bfb3f5db2315b7b1e5963dfce614fa2434e58caad9"
Nov 01 03:54:40.191582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:40.191547    4930 scope.go:122] "RemoveContainer" containerID="099dcc73f336a57c40cc71a67c8580274f1490365daa79079118cba1c375caea"
Nov 01 03:54:43.142437 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:43.142392    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:54:47.206599 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.206567    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg"]
Nov 01 03:54:47.212229 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.212208    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg"
Nov 01 03:54:47.256855 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.256823    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6"]
Nov 01 03:54:47.260362 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.260345    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf"]
Nov 01 03:54:47.260484 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.260461    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6"
Nov 01 03:54:47.263768 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.263747    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf"
Nov 01 03:54:47.278485 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.278456    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s"]
Nov 01 03:54:47.282060 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.282042    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s"
Nov 01 03:54:47.303711 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.303691    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-v7nmz\""
Nov 01 03:54:47.318237 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.318208    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg"]
Nov 01 03:54:47.363285 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.363252    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s28jx"]
Nov 01 03:54:47.364540 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.364518    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12bbd677-485b-4653-8261-8a3399da12d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xtgmg\" (UID: \"12bbd677-485b-4653-8261-8a3399da12d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg"
Nov 01 03:54:47.364805 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.364549    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6trnl\" (UniqueName: \"kubernetes.io/projected/1dd1479c-0836-4b90-a7df-6f5d1a4683c6-kube-api-access-6trnl\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-wmcn6\" (UID: \"1dd1479c-0836-4b90-a7df-6f5d1a4683c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6"
Nov 01 03:54:47.368210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.365033    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4fa40a8-e13b-4c67-954e-d283853b8fe4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-d89gf\" (UID: \"d4fa40a8-e13b-4c67-954e-d283853b8fe4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf"
Nov 01 03:54:47.368210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.365096    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07489556-954b-45d9-9ebc-53b2920d4004-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cf66s\" (UID: \"07489556-954b-45d9-9ebc-53b2920d4004\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s"
Nov 01 03:54:47.368210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.365190    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4fa40a8-e13b-4c67-954e-d283853b8fe4-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-d89gf\" (UID: \"d4fa40a8-e13b-4c67-954e-d283853b8fe4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf"
Nov 01 03:54:47.368210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.365245    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07489556-954b-45d9-9ebc-53b2920d4004-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cf66s\" (UID: \"07489556-954b-45d9-9ebc-53b2920d4004\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s"
Nov 01 03:54:47.368210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.365305    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4fa40a8-e13b-4c67-954e-d283853b8fe4-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-d89gf\" (UID: \"d4fa40a8-e13b-4c67-954e-d283853b8fe4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf"
Nov 01 03:54:47.368210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.365339    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07489556-954b-45d9-9ebc-53b2920d4004-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cf66s\" (UID: \"07489556-954b-45d9-9ebc-53b2920d4004\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s"
Nov 01 03:54:47.368210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.365376    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12bbd677-485b-4653-8261-8a3399da12d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xtgmg\" (UID: \"12bbd677-485b-4653-8261-8a3399da12d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg"
Nov 01 03:54:47.368210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.365408    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dd1479c-0836-4b90-a7df-6f5d1a4683c6-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-wmcn6\" (UID: \"1dd1479c-0836-4b90-a7df-6f5d1a4683c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6"
Nov 01 03:54:47.368210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.365479    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dd1479c-0836-4b90-a7df-6f5d1a4683c6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-wmcn6\" (UID: \"1dd1479c-0836-4b90-a7df-6f5d1a4683c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6"
Nov 01 03:54:47.368210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.365549    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4fa40a8-e13b-4c67-954e-d283853b8fe4-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-d89gf\" (UID: \"d4fa40a8-e13b-4c67-954e-d283853b8fe4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf"
Nov 01 03:54:47.368210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.365617    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4fa40a8-e13b-4c67-954e-d283853b8fe4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-d89gf\" (UID: \"d4fa40a8-e13b-4c67-954e-d283853b8fe4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf"
Nov 01 03:54:47.368210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.365662    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fpnm\" (UniqueName: \"kubernetes.io/projected/07489556-954b-45d9-9ebc-53b2920d4004-kube-api-access-4fpnm\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cf66s\" (UID: \"07489556-954b-45d9-9ebc-53b2920d4004\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s"
Nov 01 03:54:47.368210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.365694    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07489556-954b-45d9-9ebc-53b2920d4004-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cf66s\" (UID: \"07489556-954b-45d9-9ebc-53b2920d4004\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s"
Nov 01 03:54:47.368210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.365729    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dd1479c-0836-4b90-a7df-6f5d1a4683c6-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-wmcn6\" (UID: \"1dd1479c-0836-4b90-a7df-6f5d1a4683c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6"
Nov 01 03:54:47.368210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.365763    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07489556-954b-45d9-9ebc-53b2920d4004-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cf66s\" (UID: \"07489556-954b-45d9-9ebc-53b2920d4004\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s"
Nov 01 03:54:47.368210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.365797    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12bbd677-485b-4653-8261-8a3399da12d5-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xtgmg\" (UID: \"12bbd677-485b-4653-8261-8a3399da12d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg"
Nov 01 03:54:47.368210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.365828    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dd1479c-0836-4b90-a7df-6f5d1a4683c6-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-wmcn6\" (UID: \"1dd1479c-0836-4b90-a7df-6f5d1a4683c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6"
Nov 01 03:54:47.368210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.365855    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07489556-954b-45d9-9ebc-53b2920d4004-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cf66s\" (UID: \"07489556-954b-45d9-9ebc-53b2920d4004\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s"
Nov 01 03:54:47.368210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.365886    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dd1479c-0836-4b90-a7df-6f5d1a4683c6-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-wmcn6\" (UID: \"1dd1479c-0836-4b90-a7df-6f5d1a4683c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6"
Nov 01 03:54:47.368210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.365923    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12bbd677-485b-4653-8261-8a3399da12d5-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xtgmg\" (UID: \"12bbd677-485b-4653-8261-8a3399da12d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg"
Nov 01 03:54:47.368210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.365955    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4fa40a8-e13b-4c67-954e-d283853b8fe4-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-d89gf\" (UID: \"d4fa40a8-e13b-4c67-954e-d283853b8fe4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf"
Nov 01 03:54:47.368210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.366002    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dd1479c-0836-4b90-a7df-6f5d1a4683c6-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-wmcn6\" (UID: \"1dd1479c-0836-4b90-a7df-6f5d1a4683c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6"
Nov 01 03:54:47.368210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.366036    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12bbd677-485b-4653-8261-8a3399da12d5-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xtgmg\" (UID: \"12bbd677-485b-4653-8261-8a3399da12d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg"
Nov 01 03:54:47.368210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.366069    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12bbd677-485b-4653-8261-8a3399da12d5-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xtgmg\" (UID: \"12bbd677-485b-4653-8261-8a3399da12d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg"
Nov 01 03:54:47.368210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.366101    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4mps\" (UniqueName: \"kubernetes.io/projected/12bbd677-485b-4653-8261-8a3399da12d5-kube-api-access-l4mps\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xtgmg\" (UID: \"12bbd677-485b-4653-8261-8a3399da12d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg"
Nov 01 03:54:47.368210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.366136    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4fa40a8-e13b-4c67-954e-d283853b8fe4-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-d89gf\" (UID: \"d4fa40a8-e13b-4c67-954e-d283853b8fe4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf"
Nov 01 03:54:47.368210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.366163    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dd1479c-0836-4b90-a7df-6f5d1a4683c6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-wmcn6\" (UID: \"1dd1479c-0836-4b90-a7df-6f5d1a4683c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6"
Nov 01 03:54:47.368210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.366209    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glbpt\" (UniqueName: \"kubernetes.io/projected/d4fa40a8-e13b-4c67-954e-d283853b8fe4-kube-api-access-glbpt\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-d89gf\" (UID: \"d4fa40a8-e13b-4c67-954e-d283853b8fe4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf"
Nov 01 03:54:47.368210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.366262    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12bbd677-485b-4653-8261-8a3399da12d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xtgmg\" (UID: \"12bbd677-485b-4653-8261-8a3399da12d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg"
Nov 01 03:54:47.368210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.366293    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12bbd677-485b-4653-8261-8a3399da12d5-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xtgmg\" (UID: \"12bbd677-485b-4653-8261-8a3399da12d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg"
Nov 01 03:54:47.368210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.366326    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07489556-954b-45d9-9ebc-53b2920d4004-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cf66s\" (UID: \"07489556-954b-45d9-9ebc-53b2920d4004\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s"
Nov 01 03:54:47.368210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.366358    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07489556-954b-45d9-9ebc-53b2920d4004-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cf66s\" (UID: \"07489556-954b-45d9-9ebc-53b2920d4004\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s"
Nov 01 03:54:47.368210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.366403    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4fa40a8-e13b-4c67-954e-d283853b8fe4-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-d89gf\" (UID: \"d4fa40a8-e13b-4c67-954e-d283853b8fe4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf"
Nov 01 03:54:47.368210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.366431    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dd1479c-0836-4b90-a7df-6f5d1a4683c6-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-wmcn6\" (UID: \"1dd1479c-0836-4b90-a7df-6f5d1a4683c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6"
Nov 01 03:54:47.368210 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.367353    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s28jx"
Nov 01 03:54:47.389905 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.389880    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf"]
Nov 01 03:54:47.393722 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.393700    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6"]
Nov 01 03:54:47.423274 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.423202    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:54:47.423274 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.423246    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:54:47.423497 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.423345    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:54:47.423497 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.423454    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:54:47.426386 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.426360    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s28jx"]
Nov 01 03:54:47.446701 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.446671    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s"]
Nov 01 03:54:47.467000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.466943    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4fa40a8-e13b-4c67-954e-d283853b8fe4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-d89gf\" (UID: \"d4fa40a8-e13b-4c67-954e-d283853b8fe4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf"
Nov 01 03:54:47.467000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.466981    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fpnm\" (UniqueName: \"kubernetes.io/projected/07489556-954b-45d9-9ebc-53b2920d4004-kube-api-access-4fpnm\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cf66s\" (UID: \"07489556-954b-45d9-9ebc-53b2920d4004\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s"
Nov 01 03:54:47.467152 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.467007    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07489556-954b-45d9-9ebc-53b2920d4004-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cf66s\" (UID: \"07489556-954b-45d9-9ebc-53b2920d4004\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s"
Nov 01 03:54:47.467152 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.467034    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dd1479c-0836-4b90-a7df-6f5d1a4683c6-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-wmcn6\" (UID: \"1dd1479c-0836-4b90-a7df-6f5d1a4683c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6"
Nov 01 03:54:47.467152 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.467060    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07489556-954b-45d9-9ebc-53b2920d4004-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cf66s\" (UID: \"07489556-954b-45d9-9ebc-53b2920d4004\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s"
Nov 01 03:54:47.467152 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.467085    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12bbd677-485b-4653-8261-8a3399da12d5-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xtgmg\" (UID: \"12bbd677-485b-4653-8261-8a3399da12d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg"
Nov 01 03:54:47.467152 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.467107    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dd1479c-0836-4b90-a7df-6f5d1a4683c6-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-wmcn6\" (UID: \"1dd1479c-0836-4b90-a7df-6f5d1a4683c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6"
Nov 01 03:54:47.467442 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.467241    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07489556-954b-45d9-9ebc-53b2920d4004-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cf66s\" (UID: \"07489556-954b-45d9-9ebc-53b2920d4004\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s"
Nov 01 03:54:47.467442 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.467272    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dd1479c-0836-4b90-a7df-6f5d1a4683c6-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-wmcn6\" (UID: \"1dd1479c-0836-4b90-a7df-6f5d1a4683c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6"
Nov 01 03:54:47.467442 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.467312    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12bbd677-485b-4653-8261-8a3399da12d5-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xtgmg\" (UID: \"12bbd677-485b-4653-8261-8a3399da12d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg"
Nov 01 03:54:47.467442 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.467336    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4fa40a8-e13b-4c67-954e-d283853b8fe4-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-d89gf\" (UID: \"d4fa40a8-e13b-4c67-954e-d283853b8fe4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf"
Nov 01 03:54:47.467442 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.467374    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2b8m\" (UniqueName: \"kubernetes.io/projected/b1bce12e-2008-43a2-991a-0e309db6b463-kube-api-access-r2b8m\") pod \"maas-api-revoke-sub-85237e45-s28jx\" (UID: \"b1bce12e-2008-43a2-991a-0e309db6b463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s28jx"
Nov 01 03:54:47.467442 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.467426    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dd1479c-0836-4b90-a7df-6f5d1a4683c6-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-wmcn6\" (UID: \"1dd1479c-0836-4b90-a7df-6f5d1a4683c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6"
Nov 01 03:54:47.467804 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.467461    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12bbd677-485b-4653-8261-8a3399da12d5-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xtgmg\" (UID: \"12bbd677-485b-4653-8261-8a3399da12d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg"
Nov 01 03:54:47.467804 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.467488    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12bbd677-485b-4653-8261-8a3399da12d5-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xtgmg\" (UID: \"12bbd677-485b-4653-8261-8a3399da12d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg"
Nov 01 03:54:47.467804 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.467527    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12bbd677-485b-4653-8261-8a3399da12d5-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xtgmg\" (UID: \"12bbd677-485b-4653-8261-8a3399da12d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg"
Nov 01 03:54:47.467804 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.467548    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4mps\" (UniqueName: \"kubernetes.io/projected/12bbd677-485b-4653-8261-8a3399da12d5-kube-api-access-l4mps\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xtgmg\" (UID: \"12bbd677-485b-4653-8261-8a3399da12d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg"
Nov 01 03:54:47.467804 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.467635    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1bce12e-2008-43a2-991a-0e309db6b463-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-s28jx\" (UID: \"b1bce12e-2008-43a2-991a-0e309db6b463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s28jx"
Nov 01 03:54:47.467804 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.467683    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4fa40a8-e13b-4c67-954e-d283853b8fe4-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-d89gf\" (UID: \"d4fa40a8-e13b-4c67-954e-d283853b8fe4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf"
Nov 01 03:54:47.467804 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.467714    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dd1479c-0836-4b90-a7df-6f5d1a4683c6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-wmcn6\" (UID: \"1dd1479c-0836-4b90-a7df-6f5d1a4683c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6"
Nov 01 03:54:47.467804 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.467757    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glbpt\" (UniqueName: \"kubernetes.io/projected/d4fa40a8-e13b-4c67-954e-d283853b8fe4-kube-api-access-glbpt\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-d89gf\" (UID: \"d4fa40a8-e13b-4c67-954e-d283853b8fe4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf"
Nov 01 03:54:47.467804 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.467775    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dd1479c-0836-4b90-a7df-6f5d1a4683c6-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-wmcn6\" (UID: \"1dd1479c-0836-4b90-a7df-6f5d1a4683c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6"
Nov 01 03:54:47.467804 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.467795    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1bce12e-2008-43a2-991a-0e309db6b463-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-s28jx\" (UID: \"b1bce12e-2008-43a2-991a-0e309db6b463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s28jx"
Nov 01 03:54:47.468355 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.468074    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12bbd677-485b-4653-8261-8a3399da12d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xtgmg\" (UID: \"12bbd677-485b-4653-8261-8a3399da12d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg"
Nov 01 03:54:47.468355 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.468108    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12bbd677-485b-4653-8261-8a3399da12d5-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xtgmg\" (UID: \"12bbd677-485b-4653-8261-8a3399da12d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg"
Nov 01 03:54:47.468355 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.468143    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07489556-954b-45d9-9ebc-53b2920d4004-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cf66s\" (UID: \"07489556-954b-45d9-9ebc-53b2920d4004\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s"
Nov 01 03:54:47.468355 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.468184    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07489556-954b-45d9-9ebc-53b2920d4004-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cf66s\" (UID: \"07489556-954b-45d9-9ebc-53b2920d4004\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s"
Nov 01 03:54:47.468355 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.468221    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4fa40a8-e13b-4c67-954e-d283853b8fe4-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-d89gf\" (UID: \"d4fa40a8-e13b-4c67-954e-d283853b8fe4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf"
Nov 01 03:54:47.468355 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.468245    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dd1479c-0836-4b90-a7df-6f5d1a4683c6-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-wmcn6\" (UID: \"1dd1479c-0836-4b90-a7df-6f5d1a4683c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6"
Nov 01 03:54:47.468355 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.468280    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4fa40a8-e13b-4c67-954e-d283853b8fe4-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-d89gf\" (UID: \"d4fa40a8-e13b-4c67-954e-d283853b8fe4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf"
Nov 01 03:54:47.468355 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.468290    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12bbd677-485b-4653-8261-8a3399da12d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xtgmg\" (UID: \"12bbd677-485b-4653-8261-8a3399da12d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg"
Nov 01 03:54:47.468355 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.468319    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6trnl\" (UniqueName: \"kubernetes.io/projected/1dd1479c-0836-4b90-a7df-6f5d1a4683c6-kube-api-access-6trnl\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-wmcn6\" (UID: \"1dd1479c-0836-4b90-a7df-6f5d1a4683c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6"
Nov 01 03:54:47.468355 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.468350    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12bbd677-485b-4653-8261-8a3399da12d5-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xtgmg\" (UID: \"12bbd677-485b-4653-8261-8a3399da12d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg"
Nov 01 03:54:47.468949 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.468360    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4fa40a8-e13b-4c67-954e-d283853b8fe4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-d89gf\" (UID: \"d4fa40a8-e13b-4c67-954e-d283853b8fe4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf"
Nov 01 03:54:47.468949 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.468541    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12bbd677-485b-4653-8261-8a3399da12d5-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xtgmg\" (UID: \"12bbd677-485b-4653-8261-8a3399da12d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg"
Nov 01 03:54:47.468949 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.468780    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07489556-954b-45d9-9ebc-53b2920d4004-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cf66s\" (UID: \"07489556-954b-45d9-9ebc-53b2920d4004\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s"
Nov 01 03:54:47.468949 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.468910    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dd1479c-0836-4b90-a7df-6f5d1a4683c6-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-wmcn6\" (UID: \"1dd1479c-0836-4b90-a7df-6f5d1a4683c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6"
Nov 01 03:54:47.469168 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.468108    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12bbd677-485b-4653-8261-8a3399da12d5-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xtgmg\" (UID: \"12bbd677-485b-4653-8261-8a3399da12d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg"
Nov 01 03:54:47.469168 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.469052    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4fa40a8-e13b-4c67-954e-d283853b8fe4-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-d89gf\" (UID: \"d4fa40a8-e13b-4c67-954e-d283853b8fe4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf"
Nov 01 03:54:47.469168 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.469148    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07489556-954b-45d9-9ebc-53b2920d4004-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cf66s\" (UID: \"07489556-954b-45d9-9ebc-53b2920d4004\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s"
Nov 01 03:54:47.470918 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.469881    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12bbd677-485b-4653-8261-8a3399da12d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xtgmg\" (UID: \"12bbd677-485b-4653-8261-8a3399da12d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg"
Nov 01 03:54:47.470918 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.470020    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dd1479c-0836-4b90-a7df-6f5d1a4683c6-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-wmcn6\" (UID: \"1dd1479c-0836-4b90-a7df-6f5d1a4683c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6"
Nov 01 03:54:47.470918 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.470141    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07489556-954b-45d9-9ebc-53b2920d4004-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cf66s\" (UID: \"07489556-954b-45d9-9ebc-53b2920d4004\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s"
Nov 01 03:54:47.470918 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.470173    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4fa40a8-e13b-4c67-954e-d283853b8fe4-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-d89gf\" (UID: \"d4fa40a8-e13b-4c67-954e-d283853b8fe4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf"
Nov 01 03:54:47.470918 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.470199    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07489556-954b-45d9-9ebc-53b2920d4004-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cf66s\" (UID: \"07489556-954b-45d9-9ebc-53b2920d4004\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s"
Nov 01 03:54:47.470918 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.470227    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4fa40a8-e13b-4c67-954e-d283853b8fe4-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-d89gf\" (UID: \"d4fa40a8-e13b-4c67-954e-d283853b8fe4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf"
Nov 01 03:54:47.470918 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.470253    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07489556-954b-45d9-9ebc-53b2920d4004-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cf66s\" (UID: \"07489556-954b-45d9-9ebc-53b2920d4004\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s"
Nov 01 03:54:47.470918 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.470284    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12bbd677-485b-4653-8261-8a3399da12d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xtgmg\" (UID: \"12bbd677-485b-4653-8261-8a3399da12d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg"
Nov 01 03:54:47.470918 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.470312    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dd1479c-0836-4b90-a7df-6f5d1a4683c6-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-wmcn6\" (UID: \"1dd1479c-0836-4b90-a7df-6f5d1a4683c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6"
Nov 01 03:54:47.470918 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.470339    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dd1479c-0836-4b90-a7df-6f5d1a4683c6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-wmcn6\" (UID: \"1dd1479c-0836-4b90-a7df-6f5d1a4683c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6"
Nov 01 03:54:47.470918 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.470377    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4fa40a8-e13b-4c67-954e-d283853b8fe4-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-d89gf\" (UID: \"d4fa40a8-e13b-4c67-954e-d283853b8fe4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf"
Nov 01 03:54:47.470918 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.470388    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07489556-954b-45d9-9ebc-53b2920d4004-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cf66s\" (UID: \"07489556-954b-45d9-9ebc-53b2920d4004\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s"
Nov 01 03:54:47.470918 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.470399    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07489556-954b-45d9-9ebc-53b2920d4004-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cf66s\" (UID: \"07489556-954b-45d9-9ebc-53b2920d4004\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s"
Nov 01 03:54:47.470918 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.470147    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dd1479c-0836-4b90-a7df-6f5d1a4683c6-istio-data\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-wmcn6\" (UID: \"1dd1479c-0836-4b90-a7df-6f5d1a4683c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6"
Nov 01 03:54:47.470918 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.470613    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4fa40a8-e13b-4c67-954e-d283853b8fe4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-d89gf\" (UID: \"d4fa40a8-e13b-4c67-954e-d283853b8fe4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf"
Nov 01 03:54:47.470918 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.470707    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4fa40a8-e13b-4c67-954e-d283853b8fe4-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-d89gf\" (UID: \"d4fa40a8-e13b-4c67-954e-d283853b8fe4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf"
Nov 01 03:54:47.470918 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.470820    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4fa40a8-e13b-4c67-954e-d283853b8fe4-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-d89gf\" (UID: \"d4fa40a8-e13b-4c67-954e-d283853b8fe4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf"
Nov 01 03:54:47.472151 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.470970    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07489556-954b-45d9-9ebc-53b2920d4004-workload-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cf66s\" (UID: \"07489556-954b-45d9-9ebc-53b2920d4004\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s"
Nov 01 03:54:47.472151 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.471025    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07489556-954b-45d9-9ebc-53b2920d4004-workload-certs\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cf66s\" (UID: \"07489556-954b-45d9-9ebc-53b2920d4004\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s"
Nov 01 03:54:47.472151 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.471231    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dd1479c-0836-4b90-a7df-6f5d1a4683c6-credential-socket\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-wmcn6\" (UID: \"1dd1479c-0836-4b90-a7df-6f5d1a4683c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6"
Nov 01 03:54:47.472151 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.471662    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dd1479c-0836-4b90-a7df-6f5d1a4683c6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-wmcn6\" (UID: \"1dd1479c-0836-4b90-a7df-6f5d1a4683c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6"
Nov 01 03:54:47.472151 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.471853    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07489556-954b-45d9-9ebc-53b2920d4004-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cf66s\" (UID: \"07489556-954b-45d9-9ebc-53b2920d4004\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s"
Nov 01 03:54:47.472444 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.472357    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dd1479c-0836-4b90-a7df-6f5d1a4683c6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-wmcn6\" (UID: \"1dd1479c-0836-4b90-a7df-6f5d1a4683c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6"
Nov 01 03:54:47.472700 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.472679    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4fa40a8-e13b-4c67-954e-d283853b8fe4-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-d89gf\" (UID: \"d4fa40a8-e13b-4c67-954e-d283853b8fe4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf"
Nov 01 03:54:47.472700 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.472692    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4fa40a8-e13b-4c67-954e-d283853b8fe4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-d89gf\" (UID: \"d4fa40a8-e13b-4c67-954e-d283853b8fe4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf"
Nov 01 03:54:47.473302 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.473281    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12bbd677-485b-4653-8261-8a3399da12d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xtgmg\" (UID: \"12bbd677-485b-4653-8261-8a3399da12d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg"
Nov 01 03:54:47.473798 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.473778    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12bbd677-485b-4653-8261-8a3399da12d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xtgmg\" (UID: \"12bbd677-485b-4653-8261-8a3399da12d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg"
Nov 01 03:54:47.531144 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.531095    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dd1479c-0836-4b90-a7df-6f5d1a4683c6-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-wmcn6\" (UID: \"1dd1479c-0836-4b90-a7df-6f5d1a4683c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6"
Nov 01 03:54:47.531383 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.531099    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07489556-954b-45d9-9ebc-53b2920d4004-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cf66s\" (UID: \"07489556-954b-45d9-9ebc-53b2920d4004\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s"
Nov 01 03:54:47.531383 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.531196    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4mps\" (UniqueName: \"kubernetes.io/projected/12bbd677-485b-4653-8261-8a3399da12d5-kube-api-access-l4mps\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xtgmg\" (UID: \"12bbd677-485b-4653-8261-8a3399da12d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg"
Nov 01 03:54:47.531508 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.531394    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fpnm\" (UniqueName: \"kubernetes.io/projected/07489556-954b-45d9-9ebc-53b2920d4004-kube-api-access-4fpnm\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-cf66s\" (UID: \"07489556-954b-45d9-9ebc-53b2920d4004\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s"
Nov 01 03:54:47.531508 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.531398    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glbpt\" (UniqueName: \"kubernetes.io/projected/d4fa40a8-e13b-4c67-954e-d283853b8fe4-kube-api-access-glbpt\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-d89gf\" (UID: \"d4fa40a8-e13b-4c67-954e-d283853b8fe4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf"
Nov 01 03:54:47.531654 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.531612    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4fa40a8-e13b-4c67-954e-d283853b8fe4-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-d89gf\" (UID: \"d4fa40a8-e13b-4c67-954e-d283853b8fe4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf"
Nov 01 03:54:47.532298 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.532277    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6trnl\" (UniqueName: \"kubernetes.io/projected/1dd1479c-0836-4b90-a7df-6f5d1a4683c6-kube-api-access-6trnl\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-wmcn6\" (UID: \"1dd1479c-0836-4b90-a7df-6f5d1a4683c6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6"
Nov 01 03:54:47.540024 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.539996    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12bbd677-485b-4653-8261-8a3399da12d5-istio-token\") pod \"maas-default-gateway-openshift-default-7c5b899d5f-xtgmg\" (UID: \"12bbd677-485b-4653-8261-8a3399da12d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg"
Nov 01 03:54:47.571534 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.571499    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2b8m\" (UniqueName: \"kubernetes.io/projected/b1bce12e-2008-43a2-991a-0e309db6b463-kube-api-access-r2b8m\") pod \"maas-api-revoke-sub-85237e45-s28jx\" (UID: \"b1bce12e-2008-43a2-991a-0e309db6b463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s28jx"
Nov 01 03:54:47.571740 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.571546    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1bce12e-2008-43a2-991a-0e309db6b463-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-s28jx\" (UID: \"b1bce12e-2008-43a2-991a-0e309db6b463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s28jx"
Nov 01 03:54:47.571740 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.571595    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1bce12e-2008-43a2-991a-0e309db6b463-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-s28jx\" (UID: \"b1bce12e-2008-43a2-991a-0e309db6b463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s28jx"
Nov 01 03:54:47.572347 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.572318    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1bce12e-2008-43a2-991a-0e309db6b463-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-s28jx\" (UID: \"b1bce12e-2008-43a2-991a-0e309db6b463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s28jx"
Nov 01 03:54:47.574120 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.574087    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1bce12e-2008-43a2-991a-0e309db6b463-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-s28jx\" (UID: \"b1bce12e-2008-43a2-991a-0e309db6b463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s28jx"
Nov 01 03:54:47.574954 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.574934    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6"
Nov 01 03:54:47.581665 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.581631    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf"
Nov 01 03:54:47.598372 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.598307    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s"
Nov 01 03:54:47.636593 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.634477    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2b8m\" (UniqueName: \"kubernetes.io/projected/b1bce12e-2008-43a2-991a-0e309db6b463-kube-api-access-r2b8m\") pod \"maas-api-revoke-sub-85237e45-s28jx\" (UID: \"b1bce12e-2008-43a2-991a-0e309db6b463\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s28jx"
Nov 01 03:54:47.693266 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.693238    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s28jx"
Nov 01 03:54:47.825601 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.825545    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg"
Nov 01 03:54:47.971140 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.971107    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6"]
Nov 01 03:54:47.972353 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:54:47.972299    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dd1479c_0836_4b90_a7df_6f5d1a4683c6.slice/crio-83ae03a999c2fbb96cb56886181435e6981a7d80fba5677dc742ca46ad591fe9 WatchSource:0}: Error finding container 83ae03a999c2fbb96cb56886181435e6981a7d80fba5677dc742ca46ad591fe9: Status 404 returned error can't find the container with id 83ae03a999c2fbb96cb56886181435e6981a7d80fba5677dc742ca46ad591fe9
Nov 01 03:54:47.974755 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.974720    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:54:47.974847 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.974800    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:54:47.974974 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:47.974873    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:54:48.020987 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:48.020966    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf"]
Nov 01 03:54:48.022053 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:54:48.022025    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07489556_954b_45d9_9ebc_53b2920d4004.slice/crio-0cfeb327cff2a46d0f8098d314712b7b7a02036bd1bfc4f18a939ec4036d14cb WatchSource:0}: Error finding container 0cfeb327cff2a46d0f8098d314712b7b7a02036bd1bfc4f18a939ec4036d14cb: Status 404 returned error can't find the container with id 0cfeb327cff2a46d0f8098d314712b7b7a02036bd1bfc4f18a939ec4036d14cb
Nov 01 03:54:48.023312 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:54:48.023276    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4fa40a8_e13b_4c67_954e_d283853b8fe4.slice/crio-1f8b48e400c683351ac8164faedc196dc76825d67e9b642e09f732d122ce64e3 WatchSource:0}: Error finding container 1f8b48e400c683351ac8164faedc196dc76825d67e9b642e09f732d122ce64e3: Status 404 returned error can't find the container with id 1f8b48e400c683351ac8164faedc196dc76825d67e9b642e09f732d122ce64e3
Nov 01 03:54:48.024386 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:48.024360    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s"]
Nov 01 03:54:48.025548 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:48.025451    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:54:48.025548 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:48.025530    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:54:48.025669 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:48.025590    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:54:48.026189 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:48.026151    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:54:48.026264 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:48.026202    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:54:48.026264 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:48.026230    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:54:48.077499 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:48.077410    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s28jx"]
Nov 01 03:54:48.078978 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:54:48.078952    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1bce12e_2008_43a2_991a_0e309db6b463.slice/crio-83f990a36bd3d30177c9059e5dcc33f3afff024150cddbca4e2cc0f1a38520b7 WatchSource:0}: Error finding container 83f990a36bd3d30177c9059e5dcc33f3afff024150cddbca4e2cc0f1a38520b7: Status 404 returned error can't find the container with id 83f990a36bd3d30177c9059e5dcc33f3afff024150cddbca4e2cc0f1a38520b7
Nov 01 03:54:48.208959 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:48.208914    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg"]
Nov 01 03:54:48.261352 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:54:48.261264    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12bbd677_485b_4653_8261_8a3399da12d5.slice/crio-ee17a48b3528a825cf50b399b2e348912f898a95857c85eed5c4af3c84380e97 WatchSource:0}: Error finding container ee17a48b3528a825cf50b399b2e348912f898a95857c85eed5c4af3c84380e97: Status 404 returned error can't find the container with id ee17a48b3528a825cf50b399b2e348912f898a95857c85eed5c4af3c84380e97
Nov 01 03:54:48.268414 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:48.268376    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:54:48.268493 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:48.268466    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:54:48.268543 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:48.268508    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 03:54:48.859577 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:48.858704    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s" event={"ID":"07489556-954b-45d9-9ebc-53b2920d4004","Type":"ContainerStarted","Data":"a82cf2ee5f1214d8471fa6becd63aa532964051ee1a393f4edbf3b41ee7b50b2"}
Nov 01 03:54:48.859577 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:48.858795    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s" event={"ID":"07489556-954b-45d9-9ebc-53b2920d4004","Type":"ContainerStarted","Data":"0cfeb327cff2a46d0f8098d314712b7b7a02036bd1bfc4f18a939ec4036d14cb"}
Nov 01 03:54:48.872024 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:48.871270    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s28jx" event={"ID":"b1bce12e-2008-43a2-991a-0e309db6b463","Type":"ContainerStarted","Data":"12b474dfb493dc3cb7bfab4e5d7bfc9167a4d58c3cbbf7d2a3aa646bab96a9e7"}
Nov 01 03:54:48.872024 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:48.871340    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s28jx" event={"ID":"b1bce12e-2008-43a2-991a-0e309db6b463","Type":"ContainerStarted","Data":"83f990a36bd3d30177c9059e5dcc33f3afff024150cddbca4e2cc0f1a38520b7"}
Nov 01 03:54:48.875205 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:48.874467    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6" event={"ID":"1dd1479c-0836-4b90-a7df-6f5d1a4683c6","Type":"ContainerStarted","Data":"8fecb34aad4352ceebced8543f5c352acce23f9c9b49cbe9f898298789342a93"}
Nov 01 03:54:48.875205 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:48.874526    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6" event={"ID":"1dd1479c-0836-4b90-a7df-6f5d1a4683c6","Type":"ContainerStarted","Data":"83ae03a999c2fbb96cb56886181435e6981a7d80fba5677dc742ca46ad591fe9"}
Nov 01 03:54:48.877658 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:48.877621    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf" event={"ID":"d4fa40a8-e13b-4c67-954e-d283853b8fe4","Type":"ContainerStarted","Data":"a80b182c6503c1d018f87b0031bb2d365660feddc3b3caec1c6ec2ee81481504"}
Nov 01 03:54:48.877846 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:48.877825    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf" event={"ID":"d4fa40a8-e13b-4c67-954e-d283853b8fe4","Type":"ContainerStarted","Data":"1f8b48e400c683351ac8164faedc196dc76825d67e9b642e09f732d122ce64e3"}
Nov 01 03:54:48.901812 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:48.901768    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg" event={"ID":"12bbd677-485b-4653-8261-8a3399da12d5","Type":"ContainerStarted","Data":"3e7fd9c1d88cbad9ad03798a17adcca163b580a16afcf98b4954eb1e55160e10"}
Nov 01 03:54:48.902078 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:48.902062    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg" event={"ID":"12bbd677-485b-4653-8261-8a3399da12d5","Type":"ContainerStarted","Data":"ee17a48b3528a825cf50b399b2e348912f898a95857c85eed5c4af3c84380e97"}
Nov 01 03:54:49.157643 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:49.152729    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s" podStartSLOduration=2.152705645 podStartE2EDuration="2.152705645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:54:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:54:49.030991974 +0000 UTC m=+2893.553998562" watchObservedRunningTime="2026-11-01 03:54:49.152705645 +0000 UTC m=+2893.675712234"
Nov 01 03:54:49.195680 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:49.194353    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg" podStartSLOduration=2.194326566 podStartE2EDuration="2.194326566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:54:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:54:49.170010439 +0000 UTC m=+2893.693017060" watchObservedRunningTime="2026-11-01 03:54:49.194326566 +0000 UTC m=+2893.717333157"
Nov 01 03:54:49.323596 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:54:49.322160    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:54:49.455827 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:49.447955    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf" podStartSLOduration=2.447935228 podStartE2EDuration="2.447935228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:54:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:54:49.32252997 +0000 UTC m=+2893.845536565" watchObservedRunningTime="2026-11-01 03:54:49.447935228 +0000 UTC m=+2893.970941818"
Nov 01 03:54:49.578304 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:49.578271    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6"
Nov 01 03:54:49.586668 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:49.586632    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf"
Nov 01 03:54:49.607872 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:49.607834    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-d89gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Nov 01 03:54:49.608175 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:49.608127    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf" podUID="d4fa40a8-e13b-4c67-954e-d283853b8fe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Nov 01 03:54:49.611661 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:49.611633    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Nov 01 03:54:49.611882 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:49.611853    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6" podUID="1dd1479c-0836-4b90-a7df-6f5d1a4683c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Nov 01 03:54:49.612039 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:49.612024    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s"
Nov 01 03:54:49.625579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:49.612591    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6" podStartSLOduration=2.612575116 podStartE2EDuration="2.612575116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:54:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:54:49.476999444 +0000 UTC m=+2894.000006033" watchObservedRunningTime="2026-11-01 03:54:49.612575116 +0000 UTC m=+2894.135581704"
Nov 01 03:54:49.625579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:49.616144    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s28jx" podStartSLOduration=2.616129252 podStartE2EDuration="2.616129252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:54:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:54:49.60982888 +0000 UTC m=+2894.132835498" watchObservedRunningTime="2026-11-01 03:54:49.616129252 +0000 UTC m=+2894.139135842"
Nov 01 03:54:49.626806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:49.626775    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-cf66s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Nov 01 03:54:49.627035 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:49.627009    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s" podUID="07489556-954b-45d9-9ebc-53b2920d4004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Nov 01 03:54:49.833639 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:49.831921    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg"
Nov 01 03:54:49.839689 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:49.839659    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Nov 01 03:54:49.839915 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:49.839888    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg" podUID="12bbd677-485b-4653-8261-8a3399da12d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Nov 01 03:54:50.029579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:50.026195    4930 generic.go:356] "Generic (PLEG): container finished" podID="b1bce12e-2008-43a2-991a-0e309db6b463" containerID="12b474dfb493dc3cb7bfab4e5d7bfc9167a4d58c3cbbf7d2a3aa646bab96a9e7" exitCode=0
Nov 01 03:54:50.029579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:50.027507    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s28jx" event={"ID":"b1bce12e-2008-43a2-991a-0e309db6b463","Type":"ContainerDied","Data":"12b474dfb493dc3cb7bfab4e5d7bfc9167a4d58c3cbbf7d2a3aa646bab96a9e7"}
Nov 01 03:54:50.577991 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:50.577947    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Nov 01 03:54:50.578668 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:50.578629    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6" podUID="1dd1479c-0836-4b90-a7df-6f5d1a4683c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Nov 01 03:54:50.582902 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:50.582876    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-d89gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Nov 01 03:54:50.583088 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:50.583067    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf" podUID="d4fa40a8-e13b-4c67-954e-d283853b8fe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Nov 01 03:54:50.599935 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:50.599906    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-cf66s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Nov 01 03:54:50.600138 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:50.600110    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s" podUID="07489556-954b-45d9-9ebc-53b2920d4004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Nov 01 03:54:50.834648 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:50.830989    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Nov 01 03:54:50.834648 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:50.831069    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg" podUID="12bbd677-485b-4653-8261-8a3399da12d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Nov 01 03:54:51.628602 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:51.614075    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Nov 01 03:54:51.628602 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:51.614156    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6" podUID="1dd1479c-0836-4b90-a7df-6f5d1a4683c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Nov 01 03:54:51.628602 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:51.616234    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-cf66s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Nov 01 03:54:51.628602 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:51.616278    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s" podUID="07489556-954b-45d9-9ebc-53b2920d4004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Nov 01 03:54:51.797637 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:51.796859    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-d89gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Nov 01 03:54:51.797637 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:51.796938    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf" podUID="d4fa40a8-e13b-4c67-954e-d283853b8fe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Nov 01 03:54:51.836578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:51.828031    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Nov 01 03:54:51.836578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:51.828091    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg" podUID="12bbd677-485b-4653-8261-8a3399da12d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Nov 01 03:54:52.047590 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:52.045764    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s28jx"
Nov 01 03:54:52.100199 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:52.100168    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1bce12e-2008-43a2-991a-0e309db6b463-kube-api-access-r2b8m\" (UniqueName: \"kubernetes.io/projected/b1bce12e-2008-43a2-991a-0e309db6b463-kube-api-access-r2b8m\") pod \"b1bce12e-2008-43a2-991a-0e309db6b463\" (UID: \"b1bce12e-2008-43a2-991a-0e309db6b463\") "
Nov 01 03:54:52.100523 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:52.100502    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1bce12e-2008-43a2-991a-0e309db6b463-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1bce12e-2008-43a2-991a-0e309db6b463-maas-api-openshift-ca\") pod \"b1bce12e-2008-43a2-991a-0e309db6b463\" (UID: \"b1bce12e-2008-43a2-991a-0e309db6b463\") "
Nov 01 03:54:52.100715 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:52.100696    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1bce12e-2008-43a2-991a-0e309db6b463-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1bce12e-2008-43a2-991a-0e309db6b463-maas-api-xks-ca\") pod \"b1bce12e-2008-43a2-991a-0e309db6b463\" (UID: \"b1bce12e-2008-43a2-991a-0e309db6b463\") "
Nov 01 03:54:52.101484 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:52.101447    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s28jx" event={"ID":"b1bce12e-2008-43a2-991a-0e309db6b463","Type":"ContainerDied","Data":"83f990a36bd3d30177c9059e5dcc33f3afff024150cddbca4e2cc0f1a38520b7"}
Nov 01 03:54:52.101677 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:52.101515    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83f990a36bd3d30177c9059e5dcc33f3afff024150cddbca4e2cc0f1a38520b7"
Nov 01 03:54:52.101677 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:52.101616    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s28jx"
Nov 01 03:54:52.105584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:52.104755    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1bce12e-2008-43a2-991a-0e309db6b463-maas-api-openshift-ca" pod "b1bce12e-2008-43a2-991a-0e309db6b463" (UID: "b1bce12e-2008-43a2-991a-0e309db6b463"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:54:52.111994 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:52.111968    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1bce12e-2008-43a2-991a-0e309db6b463-maas-api-xks-ca" pod "b1bce12e-2008-43a2-991a-0e309db6b463" (UID: "b1bce12e-2008-43a2-991a-0e309db6b463"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:54:52.122882 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:52.122840    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1bce12e-2008-43a2-991a-0e309db6b463-kube-api-access-r2b8m" pod "b1bce12e-2008-43a2-991a-0e309db6b463" (UID: "b1bce12e-2008-43a2-991a-0e309db6b463"). InnerVolumeSpecName "kube-api-access-r2b8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:54:52.416993 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:52.416872    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2b8m\" (UniqueName: \"kubernetes.io/projected/b1bce12e-2008-43a2-991a-0e309db6b463-kube-api-access-r2b8m\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:52.440581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:52.418660    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1bce12e-2008-43a2-991a-0e309db6b463-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:52.440581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:52.418712    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1bce12e-2008-43a2-991a-0e309db6b463-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:54:52.600582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:52.591070    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-d89gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Nov 01 03:54:52.600582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:52.591128    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf" podUID="d4fa40a8-e13b-4c67-954e-d283853b8fe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Nov 01 03:54:52.600582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:52.591528    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Nov 01 03:54:52.600582 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:52.591583    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6" podUID="1dd1479c-0836-4b90-a7df-6f5d1a4683c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Nov 01 03:54:52.603146 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:52.603111    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-cf66s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Nov 01 03:54:52.603366 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:52.603342    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s" podUID="07489556-954b-45d9-9ebc-53b2920d4004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Nov 01 03:54:52.849831 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:52.849572    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:54:52.849831 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:52.849707    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Nov 01 03:54:52.849831 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:52.849753    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg" podUID="12bbd677-485b-4653-8261-8a3399da12d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Nov 01 03:54:53.584586 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:53.583912    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Nov 01 03:54:53.584586 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:53.583974    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6" podUID="1dd1479c-0836-4b90-a7df-6f5d1a4683c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Nov 01 03:54:53.584847 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:53.584629    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-d89gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Nov 01 03:54:53.584847 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:53.584670    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf" podUID="d4fa40a8-e13b-4c67-954e-d283853b8fe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Nov 01 03:54:53.600586 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:53.600520    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-cf66s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Nov 01 03:54:53.600760 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:53.600596    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s" podUID="07489556-954b-45d9-9ebc-53b2920d4004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Nov 01 03:54:53.828749 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:53.828707    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Nov 01 03:54:53.829038 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:53.829007    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg" podUID="12bbd677-485b-4653-8261-8a3399da12d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Nov 01 03:54:54.581584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:54.576979    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Nov 01 03:54:54.581584 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:54.577045    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6" podUID="1dd1479c-0836-4b90-a7df-6f5d1a4683c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Nov 01 03:54:54.585583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:54.584912    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-d89gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Nov 01 03:54:54.585583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:54.584970    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf" podUID="d4fa40a8-e13b-4c67-954e-d283853b8fe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Nov 01 03:54:54.603576 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:54.601644    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-cf66s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Nov 01 03:54:54.603576 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:54.601700    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s" podUID="07489556-954b-45d9-9ebc-53b2920d4004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Nov 01 03:54:54.840641 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:54.834249    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Nov 01 03:54:54.840641 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:54.834301    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg" podUID="12bbd677-485b-4653-8261-8a3399da12d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Nov 01 03:54:55.575489 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:55.575440    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Nov 01 03:54:55.575842 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:55.575513    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6" podUID="1dd1479c-0836-4b90-a7df-6f5d1a4683c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Nov 01 03:54:55.583031 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:55.583001    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-d89gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Nov 01 03:54:55.583490 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:55.583055    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf" podUID="d4fa40a8-e13b-4c67-954e-d283853b8fe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Nov 01 03:54:55.599441 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:55.599401    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-cf66s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Nov 01 03:54:55.599627 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:55.599459    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s" podUID="07489556-954b-45d9-9ebc-53b2920d4004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Nov 01 03:54:55.826868 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:55.826776    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Nov 01 03:54:55.826868 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:55.826839    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg" podUID="12bbd677-485b-4653-8261-8a3399da12d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Nov 01 03:54:56.576057 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:56.576015    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body=
Nov 01 03:54:56.576280 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:56.576083    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6" podUID="1dd1479c-0836-4b90-a7df-6f5d1a4683c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused"
Nov 01 03:54:56.582406 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:56.582367    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-d89gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused" start-of-body=
Nov 01 03:54:56.582547 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:56.582419    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf" podUID="d4fa40a8-e13b-4c67-954e-d283853b8fe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": dial tcp 10.133.1.5:15021: connect: connection refused"
Nov 01 03:54:56.599271 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:56.599245    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-cf66s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Nov 01 03:54:56.599756 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:56.599291    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s" podUID="07489556-954b-45d9-9ebc-53b2920d4004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Nov 01 03:54:56.826712 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:56.826610    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused" start-of-body=
Nov 01 03:54:56.826712 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:56.826675    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg" podUID="12bbd677-485b-4653-8261-8a3399da12d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": dial tcp 10.133.1.3:15021: connect: connection refused"
Nov 01 03:54:58.576344 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:58.576301    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 03:54:58.577014 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:58.576972    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6" podUID="1dd1479c-0836-4b90-a7df-6f5d1a4683c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:54:58.584719 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:58.584677    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-d89gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 03:54:58.584842 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:58.584725    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf" podUID="d4fa40a8-e13b-4c67-954e-d283853b8fe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:54:58.601514 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:58.601468    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-cf66s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 03:54:58.601669 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:58.601532    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s" podUID="07489556-954b-45d9-9ebc-53b2920d4004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:54:58.828883 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:58.828780    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 03:54:58.828883 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:58.828853    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg" podUID="12bbd677-485b-4653-8261-8a3399da12d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:54:59.624665 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:59.624627    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 03:54:59.625238 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:59.625206    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6" podUID="1dd1479c-0836-4b90-a7df-6f5d1a4683c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:54:59.629682 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:59.629643    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-d89gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 03:54:59.629880 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:59.629858    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf" podUID="d4fa40a8-e13b-4c67-954e-d283853b8fe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:54:59.646696 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:59.646673    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-cf66s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 03:54:59.646864 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:59.646843    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s" podUID="07489556-954b-45d9-9ebc-53b2920d4004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:54:59.886172 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:59.886087    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:54:59.894579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:59.887704    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 03:54:59.894579 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:54:59.887784    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg" podUID="12bbd677-485b-4653-8261-8a3399da12d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:55:00.053606 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:00.053545    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5d96cbcb67-ztb9c"]
Nov 01 03:55:00.054103 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:00.054067    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5d96cbcb67-ztb9c" podUID="9f3145b9-b972-4624-8c0b-187b65d4c10e" containerName="manager" containerID="cri-o://cb97936d662d428f7d5f1b0aa4064d4506e3c97431e020e74ea16a1705e71fde" gracePeriod=30
Nov 01 03:55:00.356979 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:00.356704    4930 generic.go:356] "Generic (PLEG): container finished" podID="9f3145b9-b972-4624-8c0b-187b65d4c10e" containerID="cb97936d662d428f7d5f1b0aa4064d4506e3c97431e020e74ea16a1705e71fde" exitCode=0
Nov 01 03:55:00.356979 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:00.356837    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d96cbcb67-ztb9c" event={"ID":"9f3145b9-b972-4624-8c0b-187b65d4c10e","Type":"ContainerDied","Data":"cb97936d662d428f7d5f1b0aa4064d4506e3c97431e020e74ea16a1705e71fde"}
Nov 01 03:55:00.711999 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:00.672661    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-d89gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 03:55:00.711999 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:00.672727    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf" podUID="d4fa40a8-e13b-4c67-954e-d283853b8fe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:55:00.819989 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:00.779943    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-cf66s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 03:55:00.819989 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:00.780008    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s" podUID="07489556-954b-45d9-9ebc-53b2920d4004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:55:00.911577 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:00.907775    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 03:55:00.920670 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:00.920628    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6" podUID="1dd1479c-0836-4b90-a7df-6f5d1a4683c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:55:00.936753 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:00.936716    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 03:55:00.937056 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:00.937006    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg" podUID="12bbd677-485b-4653-8261-8a3399da12d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:55:01.077887 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:01.077600    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d96cbcb67-ztb9c"
Nov 01 03:55:01.156004 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:01.155959    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f3145b9-b972-4624-8c0b-187b65d4c10e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f3145b9-b972-4624-8c0b-187b65d4c10e-metrics-certs\") pod \"9f3145b9-b972-4624-8c0b-187b65d4c10e\" (UID: \"9f3145b9-b972-4624-8c0b-187b65d4c10e\") "
Nov 01 03:55:01.156704 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:01.156680    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f3145b9-b972-4624-8c0b-187b65d4c10e-webhook-cert\" (UniqueName: \"kubernetes.io/secret/9f3145b9-b972-4624-8c0b-187b65d4c10e-webhook-cert\") pod \"9f3145b9-b972-4624-8c0b-187b65d4c10e\" (UID: \"9f3145b9-b972-4624-8c0b-187b65d4c10e\") "
Nov 01 03:55:01.157247 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:01.157228    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f3145b9-b972-4624-8c0b-187b65d4c10e-kube-api-access-4jkgj\" (UniqueName: \"kubernetes.io/projected/9f3145b9-b972-4624-8c0b-187b65d4c10e-kube-api-access-4jkgj\") pod \"9f3145b9-b972-4624-8c0b-187b65d4c10e\" (UID: \"9f3145b9-b972-4624-8c0b-187b65d4c10e\") "
Nov 01 03:55:01.195934 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:01.195902    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f3145b9-b972-4624-8c0b-187b65d4c10e-kube-api-access-4jkgj" pod "9f3145b9-b972-4624-8c0b-187b65d4c10e" (UID: "9f3145b9-b972-4624-8c0b-187b65d4c10e"). InnerVolumeSpecName "kube-api-access-4jkgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:55:01.197494 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:01.196647    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f3145b9-b972-4624-8c0b-187b65d4c10e-metrics-certs" pod "9f3145b9-b972-4624-8c0b-187b65d4c10e" (UID: "9f3145b9-b972-4624-8c0b-187b65d4c10e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:55:01.197662 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:01.196701    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f3145b9-b972-4624-8c0b-187b65d4c10e-webhook-cert" pod "9f3145b9-b972-4624-8c0b-187b65d4c10e" (UID: "9f3145b9-b972-4624-8c0b-187b65d4c10e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:55:01.264609 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:01.263222    4930 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f3145b9-b972-4624-8c0b-187b65d4c10e-metrics-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:55:01.264926 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:01.264886    4930 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9f3145b9-b972-4624-8c0b-187b65d4c10e-webhook-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:55:01.265094 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:01.265079    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jkgj\" (UniqueName: \"kubernetes.io/projected/9f3145b9-b972-4624-8c0b-187b65d4c10e-kube-api-access-4jkgj\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:55:01.321655 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:01.305231    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s"
Nov 01 03:55:01.321655 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:01.306711    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s"
Nov 01 03:55:01.321655 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:01.307047    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6"
Nov 01 03:55:01.321655 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:01.309085    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6"
Nov 01 03:55:01.322033 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:01.321773    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s"
Nov 01 03:55:01.341303 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:01.341133    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6"
Nov 01 03:55:01.372580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:01.370529    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d96cbcb67-ztb9c"
Nov 01 03:55:01.372580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:01.370698    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d96cbcb67-ztb9c" event={"ID":"9f3145b9-b972-4624-8c0b-187b65d4c10e","Type":"ContainerDied","Data":"5637a2b5b1d267e5c609aefe87da5eb0a5c9606749c5d720f155480591038b05"}
Nov 01 03:55:01.372580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:01.370752    4930 scope.go:122] "RemoveContainer" containerID="cb97936d662d428f7d5f1b0aa4064d4506e3c97431e020e74ea16a1705e71fde"
Nov 01 03:55:01.374125 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:01.373015    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg"
Nov 01 03:55:01.374125 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:01.373702    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg"
Nov 01 03:55:01.399673 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:01.399636    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf"
Nov 01 03:55:01.400041 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:01.399987    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf"
Nov 01 03:55:01.401148 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:01.401129    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf"
Nov 01 03:55:01.424887 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:01.424866    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg"
Nov 01 03:55:02.032013 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:02.031977    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5d96cbcb67-ztb9c"]
Nov 01 03:55:02.036268 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:02.036239    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5d96cbcb67-ztb9c"]
Nov 01 03:55:03.264180 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:03.264148    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:55:03.425305 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:03.425274    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5d96cbcb67-k2pgr"]
Nov 01 03:55:03.425700 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:03.425686    4930 cpu_manager.go:414] "containerMap: removing container" podUID="9f3145b9-b972-4624-8c0b-187b65d4c10e" containerName="manager"
Nov 01 03:55:03.425770 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:03.425701    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f3145b9-b972-4624-8c0b-187b65d4c10e" containerName="manager"
Nov 01 03:55:03.425770 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:03.425727    4930 cpu_manager.go:414] "containerMap: removing container" podUID="b1bce12e-2008-43a2-991a-0e309db6b463" containerName="revoke-keys"
Nov 01 03:55:03.425770 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:03.425732    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1bce12e-2008-43a2-991a-0e309db6b463" containerName="revoke-keys"
Nov 01 03:55:03.425893 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:03.425799    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1bce12e-2008-43a2-991a-0e309db6b463" containerName="revoke-keys"
Nov 01 03:55:03.425893 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:03.425811    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f3145b9-b972-4624-8c0b-187b65d4c10e" containerName="manager"
Nov 01 03:55:03.430665 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:03.430639    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d96cbcb67-k2pgr"
Nov 01 03:55:03.489715 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:03.489685    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llgtz\" (UniqueName: \"kubernetes.io/projected/6b7143c4-cf31-4517-9d32-879238fdb77d-kube-api-access-llgtz\") pod \"maas-controller-5d96cbcb67-k2pgr\" (UID: \"6b7143c4-cf31-4517-9d32-879238fdb77d\") " pod="opendatahub/maas-controller-5d96cbcb67-k2pgr"
Nov 01 03:55:03.489977 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:03.489958    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6b7143c4-cf31-4517-9d32-879238fdb77d-webhook-cert\") pod \"maas-controller-5d96cbcb67-k2pgr\" (UID: \"6b7143c4-cf31-4517-9d32-879238fdb77d\") " pod="opendatahub/maas-controller-5d96cbcb67-k2pgr"
Nov 01 03:55:03.490284 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:03.490260    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b7143c4-cf31-4517-9d32-879238fdb77d-metrics-certs\") pod \"maas-controller-5d96cbcb67-k2pgr\" (UID: \"6b7143c4-cf31-4517-9d32-879238fdb77d\") " pod="opendatahub/maas-controller-5d96cbcb67-k2pgr"
Nov 01 03:55:03.532586 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:03.532503    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 01 03:55:03.532806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:03.532780    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 01 03:55:03.532806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:03.532825    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 01 03:55:03.533020 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:03.532883    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 01 03:55:03.533020 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:03.532882    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 01 03:55:03.533020 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:03.533002    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kdlhc\""
Nov 01 03:55:03.575982 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:03.575954    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5d96cbcb67-k2pgr"]
Nov 01 03:55:03.591680 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:03.591656    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6b7143c4-cf31-4517-9d32-879238fdb77d-webhook-cert\") pod \"maas-controller-5d96cbcb67-k2pgr\" (UID: \"6b7143c4-cf31-4517-9d32-879238fdb77d\") " pod="opendatahub/maas-controller-5d96cbcb67-k2pgr"
Nov 01 03:55:03.591796 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:03.591707    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b7143c4-cf31-4517-9d32-879238fdb77d-metrics-certs\") pod \"maas-controller-5d96cbcb67-k2pgr\" (UID: \"6b7143c4-cf31-4517-9d32-879238fdb77d\") " pod="opendatahub/maas-controller-5d96cbcb67-k2pgr"
Nov 01 03:55:03.591796 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:03.591736    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llgtz\" (UniqueName: \"kubernetes.io/projected/6b7143c4-cf31-4517-9d32-879238fdb77d-kube-api-access-llgtz\") pod \"maas-controller-5d96cbcb67-k2pgr\" (UID: \"6b7143c4-cf31-4517-9d32-879238fdb77d\") " pod="opendatahub/maas-controller-5d96cbcb67-k2pgr"
Nov 01 03:55:03.594256 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:03.594228    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b7143c4-cf31-4517-9d32-879238fdb77d-metrics-certs\") pod \"maas-controller-5d96cbcb67-k2pgr\" (UID: \"6b7143c4-cf31-4517-9d32-879238fdb77d\") " pod="opendatahub/maas-controller-5d96cbcb67-k2pgr"
Nov 01 03:55:03.594348 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:03.594261    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6b7143c4-cf31-4517-9d32-879238fdb77d-webhook-cert\") pod \"maas-controller-5d96cbcb67-k2pgr\" (UID: \"6b7143c4-cf31-4517-9d32-879238fdb77d\") " pod="opendatahub/maas-controller-5d96cbcb67-k2pgr"
Nov 01 03:55:03.620016 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:03.619993    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f3145b9-b972-4624-8c0b-187b65d4c10e" path="/var/lib/kubelet/pods/9f3145b9-b972-4624-8c0b-187b65d4c10e/volumes"
Nov 01 03:55:03.652397 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:03.652368    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llgtz\" (UniqueName: \"kubernetes.io/projected/6b7143c4-cf31-4517-9d32-879238fdb77d-kube-api-access-llgtz\") pod \"maas-controller-5d96cbcb67-k2pgr\" (UID: \"6b7143c4-cf31-4517-9d32-879238fdb77d\") " pod="opendatahub/maas-controller-5d96cbcb67-k2pgr"
Nov 01 03:55:03.743425 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:03.743404    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d96cbcb67-k2pgr"
Nov 01 03:55:04.117266 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:04.117239    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5d96cbcb67-k2pgr"]
Nov 01 03:55:04.118642 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:55:04.118608    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b7143c4_cf31_4517_9d32_879238fdb77d.slice/crio-ff0ef273d33a7788a4827487fb83f1a32456a9f24d1160905cce6057cd09cf7f WatchSource:0}: Error finding container ff0ef273d33a7788a4827487fb83f1a32456a9f24d1160905cce6057cd09cf7f: Status 404 returned error can't find the container with id ff0ef273d33a7788a4827487fb83f1a32456a9f24d1160905cce6057cd09cf7f
Nov 01 03:55:04.390069 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:04.389980    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d96cbcb67-k2pgr" event={"ID":"6b7143c4-cf31-4517-9d32-879238fdb77d","Type":"ContainerStarted","Data":"ff0ef273d33a7788a4827487fb83f1a32456a9f24d1160905cce6057cd09cf7f"}
Nov 01 03:55:05.397378 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:05.397343    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d96cbcb67-k2pgr" event={"ID":"6b7143c4-cf31-4517-9d32-879238fdb77d","Type":"ContainerStarted","Data":"71fb0d39a6f64647647c1b0cf8176b48401ee692ef47fa63d0216d91f7fb0a70"}
Nov 01 03:55:05.397824 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:05.397462    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5d96cbcb67-k2pgr"
Nov 01 03:55:05.515919 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:05.515869    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5d96cbcb67-k2pgr" podStartSLOduration=1.69792254 podStartE2EDuration="2.515855023s" totalImagesPullingTime="817.932483ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:55:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:55:05.514403453 +0000 UTC m=+2910.037410041" watchObservedRunningTime="2026-11-01 03:55:05.515855023 +0000 UTC m=+2910.038861611"
Nov 01 03:55:16.405829 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:16.405792    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5d96cbcb67-k2pgr"
Nov 01 03:55:17.128960 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:17.128919    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:55:17.273660 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:17.273614    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:55:19.121259 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:55:19.121214    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:55:33.033875 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:33.033828    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:55:40.258278 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:40.258245    4930 scope.go:122] "RemoveContainer" containerID="88749014e2c2123ad86bfd94c63964a5bf31b7b7a26c7a40f25398af5aed2a4b"
Nov 01 03:55:40.267449 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:40.267428    4930 scope.go:122] "RemoveContainer" containerID="28ab47259c162955e9d731a23eba68fa32712e828627a598dd74f727d436db72"
Nov 01 03:55:40.278115 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:40.278093    4930 scope.go:122] "RemoveContainer" containerID="064580ebbfed0c67f560363dc1b62ae4007a26445674624d962cfaf01e101732"
Nov 01 03:55:43.522506 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:43.522468    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:55:49.134425 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:55:49.134380    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:55:53.139578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:53.139516    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:55:56.783109 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:56.783082    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-h6st4"]
Nov 01 03:55:56.789369 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:56.789342    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-h6st4"
Nov 01 03:55:56.842126 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:56.842094    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:55:56.848787 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:56.848748    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ed664b9-08b4-4ef5-81dc-50a40cf43c72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-h6st4\" (UID: \"6ed664b9-08b4-4ef5-81dc-50a40cf43c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-h6st4"
Nov 01 03:55:56.848965 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:56.848891    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7qn8\" (UniqueName: \"kubernetes.io/projected/6ed664b9-08b4-4ef5-81dc-50a40cf43c72-kube-api-access-m7qn8\") pod \"maas-api-revoke-sub-c1e271a9-h6st4\" (UID: \"6ed664b9-08b4-4ef5-81dc-50a40cf43c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-h6st4"
Nov 01 03:55:56.849135 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:56.849110    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ed664b9-08b4-4ef5-81dc-50a40cf43c72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-h6st4\" (UID: \"6ed664b9-08b4-4ef5-81dc-50a40cf43c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-h6st4"
Nov 01 03:55:56.891416 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:56.891390    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:55:56.891416 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:56.891397    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:55:56.891416 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:56.891396    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:55:56.894263 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:56.894236    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-h6st4"]
Nov 01 03:55:56.950399 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:56.950367    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ed664b9-08b4-4ef5-81dc-50a40cf43c72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-h6st4\" (UID: \"6ed664b9-08b4-4ef5-81dc-50a40cf43c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-h6st4"
Nov 01 03:55:56.950532 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:56.950417    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ed664b9-08b4-4ef5-81dc-50a40cf43c72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-h6st4\" (UID: \"6ed664b9-08b4-4ef5-81dc-50a40cf43c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-h6st4"
Nov 01 03:55:56.950532 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:56.950481    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7qn8\" (UniqueName: \"kubernetes.io/projected/6ed664b9-08b4-4ef5-81dc-50a40cf43c72-kube-api-access-m7qn8\") pod \"maas-api-revoke-sub-c1e271a9-h6st4\" (UID: \"6ed664b9-08b4-4ef5-81dc-50a40cf43c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-h6st4"
Nov 01 03:55:56.951134 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:56.951111    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ed664b9-08b4-4ef5-81dc-50a40cf43c72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-h6st4\" (UID: \"6ed664b9-08b4-4ef5-81dc-50a40cf43c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-h6st4"
Nov 01 03:55:56.952952 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:56.952923    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ed664b9-08b4-4ef5-81dc-50a40cf43c72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-h6st4\" (UID: \"6ed664b9-08b4-4ef5-81dc-50a40cf43c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-h6st4"
Nov 01 03:55:57.011782 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:57.011757    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7qn8\" (UniqueName: \"kubernetes.io/projected/6ed664b9-08b4-4ef5-81dc-50a40cf43c72-kube-api-access-m7qn8\") pod \"maas-api-revoke-sub-c1e271a9-h6st4\" (UID: \"6ed664b9-08b4-4ef5-81dc-50a40cf43c72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-h6st4"
Nov 01 03:55:57.102678 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:57.102647    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-h6st4"
Nov 01 03:55:57.475762 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:57.475727    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-h6st4"]
Nov 01 03:55:57.477082 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:55:57.477052    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed664b9_08b4_4ef5_81dc_50a40cf43c72.slice/crio-9fa6b2f81836bd99b37107031313c6e57e45da1482208c981d2b8fd5002db1ed WatchSource:0}: Error finding container 9fa6b2f81836bd99b37107031313c6e57e45da1482208c981d2b8fd5002db1ed: Status 404 returned error can't find the container with id 9fa6b2f81836bd99b37107031313c6e57e45da1482208c981d2b8fd5002db1ed
Nov 01 03:55:57.758131 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:57.758051    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-h6st4" event={"ID":"6ed664b9-08b4-4ef5-81dc-50a40cf43c72","Type":"ContainerStarted","Data":"9fa6b2f81836bd99b37107031313c6e57e45da1482208c981d2b8fd5002db1ed"}
Nov 01 03:55:58.765887 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:58.765800    4930 generic.go:356] "Generic (PLEG): container finished" podID="6ed664b9-08b4-4ef5-81dc-50a40cf43c72" containerID="25d2edfee921c98ece522f5d5efdc63e1e38b9382801dcc333e95a71e2a3a423" exitCode=0
Nov 01 03:55:58.765887 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:58.765840    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-h6st4" event={"ID":"6ed664b9-08b4-4ef5-81dc-50a40cf43c72","Type":"ContainerDied","Data":"25d2edfee921c98ece522f5d5efdc63e1e38b9382801dcc333e95a71e2a3a423"}
Nov 01 03:55:59.907146 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:59.907122    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-h6st4"
Nov 01 03:55:59.977752 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:59.977715    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ed664b9-08b4-4ef5-81dc-50a40cf43c72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ed664b9-08b4-4ef5-81dc-50a40cf43c72-maas-api-xks-ca\") pod \"6ed664b9-08b4-4ef5-81dc-50a40cf43c72\" (UID: \"6ed664b9-08b4-4ef5-81dc-50a40cf43c72\") "
Nov 01 03:55:59.977871 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:59.977849    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ed664b9-08b4-4ef5-81dc-50a40cf43c72-kube-api-access-m7qn8\" (UniqueName: \"kubernetes.io/projected/6ed664b9-08b4-4ef5-81dc-50a40cf43c72-kube-api-access-m7qn8\") pod \"6ed664b9-08b4-4ef5-81dc-50a40cf43c72\" (UID: \"6ed664b9-08b4-4ef5-81dc-50a40cf43c72\") "
Nov 01 03:55:59.977920 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:59.977877    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ed664b9-08b4-4ef5-81dc-50a40cf43c72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ed664b9-08b4-4ef5-81dc-50a40cf43c72-maas-api-openshift-ca\") pod \"6ed664b9-08b4-4ef5-81dc-50a40cf43c72\" (UID: \"6ed664b9-08b4-4ef5-81dc-50a40cf43c72\") "
Nov 01 03:55:59.978286 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:59.978257    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ed664b9-08b4-4ef5-81dc-50a40cf43c72-maas-api-openshift-ca" pod "6ed664b9-08b4-4ef5-81dc-50a40cf43c72" (UID: "6ed664b9-08b4-4ef5-81dc-50a40cf43c72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:55:59.979725 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:59.979700    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ed664b9-08b4-4ef5-81dc-50a40cf43c72-maas-api-xks-ca" pod "6ed664b9-08b4-4ef5-81dc-50a40cf43c72" (UID: "6ed664b9-08b4-4ef5-81dc-50a40cf43c72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:55:59.979958 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:55:59.979934    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ed664b9-08b4-4ef5-81dc-50a40cf43c72-kube-api-access-m7qn8" pod "6ed664b9-08b4-4ef5-81dc-50a40cf43c72" (UID: "6ed664b9-08b4-4ef5-81dc-50a40cf43c72"). InnerVolumeSpecName "kube-api-access-m7qn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:56:00.079461 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:00.079371    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7qn8\" (UniqueName: \"kubernetes.io/projected/6ed664b9-08b4-4ef5-81dc-50a40cf43c72-kube-api-access-m7qn8\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:56:00.079461 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:00.079411    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ed664b9-08b4-4ef5-81dc-50a40cf43c72-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:56:00.079461 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:00.079425    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ed664b9-08b4-4ef5-81dc-50a40cf43c72-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:56:00.782141 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:00.782105    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-h6st4"
Nov 01 03:56:00.782353 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:00.782139    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-h6st4" event={"ID":"6ed664b9-08b4-4ef5-81dc-50a40cf43c72","Type":"ContainerDied","Data":"9fa6b2f81836bd99b37107031313c6e57e45da1482208c981d2b8fd5002db1ed"}
Nov 01 03:56:00.782353 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:00.782186    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fa6b2f81836bd99b37107031313c6e57e45da1482208c981d2b8fd5002db1ed"
Nov 01 03:56:15.053186 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:15.053130    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a95789f-fv6vv"]
Nov 01 03:56:15.104545 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:15.104500    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a95789f-fv6vv"]
Nov 01 03:56:15.653298 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:15.653250    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3a91e22-8c81-432a-957c-e4c301c5cb54" path="/var/lib/kubelet/pods/f3a91e22-8c81-432a-957c-e4c301c5cb54/volumes"
Nov 01 03:56:17.350708 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:17.350665    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 03:56:17.351345 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:17.351308    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6" podUID="1dd1479c-0836-4b90-a7df-6f5d1a4683c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:56:17.424578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:17.423784    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 03:56:17.424578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:17.423839    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg" podUID="12bbd677-485b-4653-8261-8a3399da12d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:56:17.447680 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:17.447648    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-d89gf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 03:56:17.447923 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:17.447893    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf" podUID="d4fa40a8-e13b-4c67-954e-d283853b8fe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 03:56:19.121579 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:56:19.121526    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:56:23.027958 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:23.027917    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:56:27.873798 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:27.873760    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vmn7p"]
Nov 01 03:56:27.874212 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:27.874192    4930 cpu_manager.go:414] "containerMap: removing container" podUID="6ed664b9-08b4-4ef5-81dc-50a40cf43c72" containerName="revoke-keys"
Nov 01 03:56:27.874212 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:27.874204    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ed664b9-08b4-4ef5-81dc-50a40cf43c72" containerName="revoke-keys"
Nov 01 03:56:27.874305 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:27.874276    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ed664b9-08b4-4ef5-81dc-50a40cf43c72" containerName="revoke-keys"
Nov 01 03:56:27.877692 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:27.877672    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vmn7p"
Nov 01 03:56:27.930147 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:27.930121    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:56:27.979358 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:27.979327    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:56:27.979498 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:27.979357    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:56:27.979498 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:27.979412    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:56:27.982187 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:27.982162    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vmn7p"]
Nov 01 03:56:27.996115 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:27.996088    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92r55\" (UniqueName: \"kubernetes.io/projected/0d115bc2-40e8-4381-afd8-465e2f1d0ef8-kube-api-access-92r55\") pod \"maas-api-revoke-sub-2e1f5b5f-vmn7p\" (UID: \"0d115bc2-40e8-4381-afd8-465e2f1d0ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vmn7p"
Nov 01 03:56:27.996253 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:27.996118    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d115bc2-40e8-4381-afd8-465e2f1d0ef8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-vmn7p\" (UID: \"0d115bc2-40e8-4381-afd8-465e2f1d0ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vmn7p"
Nov 01 03:56:27.996253 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:27.996145    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d115bc2-40e8-4381-afd8-465e2f1d0ef8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-vmn7p\" (UID: \"0d115bc2-40e8-4381-afd8-465e2f1d0ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vmn7p"
Nov 01 03:56:28.050487 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:28.050459    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26176dd3-jh22v"]
Nov 01 03:56:28.054152 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:28.054127    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26176dd3-jh22v"]
Nov 01 03:56:28.097250 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:28.097222    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92r55\" (UniqueName: \"kubernetes.io/projected/0d115bc2-40e8-4381-afd8-465e2f1d0ef8-kube-api-access-92r55\") pod \"maas-api-revoke-sub-2e1f5b5f-vmn7p\" (UID: \"0d115bc2-40e8-4381-afd8-465e2f1d0ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vmn7p"
Nov 01 03:56:28.097376 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:28.097255    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d115bc2-40e8-4381-afd8-465e2f1d0ef8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-vmn7p\" (UID: \"0d115bc2-40e8-4381-afd8-465e2f1d0ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vmn7p"
Nov 01 03:56:28.097376 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:28.097281    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d115bc2-40e8-4381-afd8-465e2f1d0ef8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-vmn7p\" (UID: \"0d115bc2-40e8-4381-afd8-465e2f1d0ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vmn7p"
Nov 01 03:56:28.097926 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:28.097907    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d115bc2-40e8-4381-afd8-465e2f1d0ef8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-vmn7p\" (UID: \"0d115bc2-40e8-4381-afd8-465e2f1d0ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vmn7p"
Nov 01 03:56:28.099691 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:28.099663    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d115bc2-40e8-4381-afd8-465e2f1d0ef8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-vmn7p\" (UID: \"0d115bc2-40e8-4381-afd8-465e2f1d0ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vmn7p"
Nov 01 03:56:28.158061 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:28.158008    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92r55\" (UniqueName: \"kubernetes.io/projected/0d115bc2-40e8-4381-afd8-465e2f1d0ef8-kube-api-access-92r55\") pod \"maas-api-revoke-sub-2e1f5b5f-vmn7p\" (UID: \"0d115bc2-40e8-4381-afd8-465e2f1d0ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vmn7p"
Nov 01 03:56:28.190546 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:28.190520    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vmn7p"
Nov 01 03:56:28.567034 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:28.567008    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vmn7p"]
Nov 01 03:56:28.568467 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:56:28.568440    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d115bc2_40e8_4381_afd8_465e2f1d0ef8.slice/crio-beaf973bcdf7b7c907edef36f9dfac0057cff5aa3536c48e588023e775e8f7ac WatchSource:0}: Error finding container beaf973bcdf7b7c907edef36f9dfac0057cff5aa3536c48e588023e775e8f7ac: Status 404 returned error can't find the container with id beaf973bcdf7b7c907edef36f9dfac0057cff5aa3536c48e588023e775e8f7ac
Nov 01 03:56:28.979805 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:28.979773    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vmn7p" event={"ID":"0d115bc2-40e8-4381-afd8-465e2f1d0ef8","Type":"ContainerStarted","Data":"d59754cce81189cfed2785aa88b550a91fda44f252db89c20bc1a597f11e3c89"}
Nov 01 03:56:28.980185 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:28.979827    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vmn7p" event={"ID":"0d115bc2-40e8-4381-afd8-465e2f1d0ef8","Type":"ContainerStarted","Data":"beaf973bcdf7b7c907edef36f9dfac0057cff5aa3536c48e588023e775e8f7ac"}
Nov 01 03:56:29.095386 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:29.095332    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vmn7p" podStartSLOduration=2.095318687 podStartE2EDuration="2.095318687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:56:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:56:29.095141851 +0000 UTC m=+2993.618148439" watchObservedRunningTime="2026-11-01 03:56:29.095318687 +0000 UTC m=+2993.618325274"
Nov 01 03:56:29.620325 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:29.620293    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbe4aee8-385b-4ba8-8a98-5de9aa154024" path="/var/lib/kubelet/pods/dbe4aee8-385b-4ba8-8a98-5de9aa154024/volumes"
Nov 01 03:56:29.987958 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:29.987924    4930 generic.go:356] "Generic (PLEG): container finished" podID="0d115bc2-40e8-4381-afd8-465e2f1d0ef8" containerID="d59754cce81189cfed2785aa88b550a91fda44f252db89c20bc1a597f11e3c89" exitCode=0
Nov 01 03:56:29.988368 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:29.988011    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vmn7p" event={"ID":"0d115bc2-40e8-4381-afd8-465e2f1d0ef8","Type":"ContainerDied","Data":"d59754cce81189cfed2785aa88b550a91fda44f252db89c20bc1a597f11e3c89"}
Nov 01 03:56:31.135747 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:31.135724    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vmn7p"
Nov 01 03:56:31.229622 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:31.229597    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d115bc2-40e8-4381-afd8-465e2f1d0ef8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d115bc2-40e8-4381-afd8-465e2f1d0ef8-maas-api-openshift-ca\") pod \"0d115bc2-40e8-4381-afd8-465e2f1d0ef8\" (UID: \"0d115bc2-40e8-4381-afd8-465e2f1d0ef8\") "
Nov 01 03:56:31.229751 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:31.229636    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d115bc2-40e8-4381-afd8-465e2f1d0ef8-kube-api-access-92r55\" (UniqueName: \"kubernetes.io/projected/0d115bc2-40e8-4381-afd8-465e2f1d0ef8-kube-api-access-92r55\") pod \"0d115bc2-40e8-4381-afd8-465e2f1d0ef8\" (UID: \"0d115bc2-40e8-4381-afd8-465e2f1d0ef8\") "
Nov 01 03:56:31.229751 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:31.229669    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d115bc2-40e8-4381-afd8-465e2f1d0ef8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d115bc2-40e8-4381-afd8-465e2f1d0ef8-maas-api-xks-ca\") pod \"0d115bc2-40e8-4381-afd8-465e2f1d0ef8\" (UID: \"0d115bc2-40e8-4381-afd8-465e2f1d0ef8\") "
Nov 01 03:56:31.229972 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:31.229949    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d115bc2-40e8-4381-afd8-465e2f1d0ef8-maas-api-openshift-ca" pod "0d115bc2-40e8-4381-afd8-465e2f1d0ef8" (UID: "0d115bc2-40e8-4381-afd8-465e2f1d0ef8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:56:31.231641 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:31.231619    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d115bc2-40e8-4381-afd8-465e2f1d0ef8-kube-api-access-92r55" pod "0d115bc2-40e8-4381-afd8-465e2f1d0ef8" (UID: "0d115bc2-40e8-4381-afd8-465e2f1d0ef8"). InnerVolumeSpecName "kube-api-access-92r55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:56:31.231806 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:31.231786    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d115bc2-40e8-4381-afd8-465e2f1d0ef8-maas-api-xks-ca" pod "0d115bc2-40e8-4381-afd8-465e2f1d0ef8" (UID: "0d115bc2-40e8-4381-afd8-465e2f1d0ef8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:56:31.330940 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:31.330877    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d115bc2-40e8-4381-afd8-465e2f1d0ef8-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:56:31.330940 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:31.330903    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d115bc2-40e8-4381-afd8-465e2f1d0ef8-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:56:31.330940 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:31.330915    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92r55\" (UniqueName: \"kubernetes.io/projected/0d115bc2-40e8-4381-afd8-465e2f1d0ef8-kube-api-access-92r55\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:56:32.004715 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:32.004617    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vmn7p"
Nov 01 03:56:32.004715 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:32.004660    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vmn7p" event={"ID":"0d115bc2-40e8-4381-afd8-465e2f1d0ef8","Type":"ContainerDied","Data":"beaf973bcdf7b7c907edef36f9dfac0057cff5aa3536c48e588023e775e8f7ac"}
Nov 01 03:56:32.004715 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:32.004712    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="beaf973bcdf7b7c907edef36f9dfac0057cff5aa3536c48e588023e775e8f7ac"
Nov 01 03:56:38.346176 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:38.346150    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jh955_f29ea61e-2a78-4c11-9566-f52497e135ff/console-operator/2.log"
Nov 01 03:56:38.390691 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:38.390668    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jh955_f29ea61e-2a78-4c11-9566-f52497e135ff/console-operator/2.log"
Nov 01 03:56:40.311904 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:40.311870    4930 scope.go:122] "RemoveContainer" containerID="9b6263b88681d17549a2032aecf1a8f66b4f902d664c7c729e3c29b3e9273d67"
Nov 01 03:56:40.321795 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:40.321771    4930 scope.go:122] "RemoveContainer" containerID="65e9cfbf234ed42b6b9ec7305dc013529e56f1a3869016104a316b576c178dff"
Nov 01 03:56:40.326988 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:40.326963    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d8bt5"]
Nov 01 03:56:40.327390 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:40.327377    4930 cpu_manager.go:414] "containerMap: removing container" podUID="0d115bc2-40e8-4381-afd8-465e2f1d0ef8" containerName="revoke-keys"
Nov 01 03:56:40.327444 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:40.327391    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d115bc2-40e8-4381-afd8-465e2f1d0ef8" containerName="revoke-keys"
Nov 01 03:56:40.327486 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:40.327475    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d115bc2-40e8-4381-afd8-465e2f1d0ef8" containerName="revoke-keys"
Nov 01 03:56:40.331980 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:40.331961    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d8bt5"
Nov 01 03:56:40.408145 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:40.408108    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d00b816-699e-4103-9a94-7b37b8d0d5e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-d8bt5\" (UID: \"7d00b816-699e-4103-9a94-7b37b8d0d5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d8bt5"
Nov 01 03:56:40.408145 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:40.408155    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvfgj\" (UniqueName: \"kubernetes.io/projected/7d00b816-699e-4103-9a94-7b37b8d0d5e4-kube-api-access-wvfgj\") pod \"maas-api-revoke-sub-ec843018-d8bt5\" (UID: \"7d00b816-699e-4103-9a94-7b37b8d0d5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d8bt5"
Nov 01 03:56:40.408371 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:40.408308    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d00b816-699e-4103-9a94-7b37b8d0d5e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-d8bt5\" (UID: \"7d00b816-699e-4103-9a94-7b37b8d0d5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d8bt5"
Nov 01 03:56:40.435429 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:40.435403    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:56:40.435640 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:40.435621    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:56:40.435700 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:40.435661    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:56:40.435783 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:40.435768    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:56:40.438390 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:40.438368    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d8bt5"]
Nov 01 03:56:40.509276 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:40.509247    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d00b816-699e-4103-9a94-7b37b8d0d5e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-d8bt5\" (UID: \"7d00b816-699e-4103-9a94-7b37b8d0d5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d8bt5"
Nov 01 03:56:40.509429 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:40.509280    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvfgj\" (UniqueName: \"kubernetes.io/projected/7d00b816-699e-4103-9a94-7b37b8d0d5e4-kube-api-access-wvfgj\") pod \"maas-api-revoke-sub-ec843018-d8bt5\" (UID: \"7d00b816-699e-4103-9a94-7b37b8d0d5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d8bt5"
Nov 01 03:56:40.509429 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:40.509391    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d00b816-699e-4103-9a94-7b37b8d0d5e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-d8bt5\" (UID: \"7d00b816-699e-4103-9a94-7b37b8d0d5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d8bt5"
Nov 01 03:56:40.510013 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:40.509995    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d00b816-699e-4103-9a94-7b37b8d0d5e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-d8bt5\" (UID: \"7d00b816-699e-4103-9a94-7b37b8d0d5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d8bt5"
Nov 01 03:56:40.511691 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:40.511674    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d00b816-699e-4103-9a94-7b37b8d0d5e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-d8bt5\" (UID: \"7d00b816-699e-4103-9a94-7b37b8d0d5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d8bt5"
Nov 01 03:56:40.569524 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:40.569467    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvfgj\" (UniqueName: \"kubernetes.io/projected/7d00b816-699e-4103-9a94-7b37b8d0d5e4-kube-api-access-wvfgj\") pod \"maas-api-revoke-sub-ec843018-d8bt5\" (UID: \"7d00b816-699e-4103-9a94-7b37b8d0d5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d8bt5"
Nov 01 03:56:40.646757 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:40.646736    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d8bt5"
Nov 01 03:56:41.022604 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:41.022568    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d8bt5"]
Nov 01 03:56:41.062946 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:41.062855    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d8bt5" event={"ID":"7d00b816-699e-4103-9a94-7b37b8d0d5e4","Type":"ContainerStarted","Data":"8c8777994c63cde7f72baecd148bcf7dfe8d737f21fd5e8324ca2c414b20ab94"}
Nov 01 03:56:42.070144 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:42.070110    4930 generic.go:356] "Generic (PLEG): container finished" podID="7d00b816-699e-4103-9a94-7b37b8d0d5e4" containerID="aad41478a96ae91d6c9017a1783dabe053b8cea19c29c21166a7d315a0b2d1ab" exitCode=0
Nov 01 03:56:42.070518 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:42.070195    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d8bt5" event={"ID":"7d00b816-699e-4103-9a94-7b37b8d0d5e4","Type":"ContainerDied","Data":"aad41478a96ae91d6c9017a1783dabe053b8cea19c29c21166a7d315a0b2d1ab"}
Nov 01 03:56:43.213437 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:43.213409    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d8bt5"
Nov 01 03:56:43.239716 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:43.239691    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d00b816-699e-4103-9a94-7b37b8d0d5e4-kube-api-access-wvfgj\" (UniqueName: \"kubernetes.io/projected/7d00b816-699e-4103-9a94-7b37b8d0d5e4-kube-api-access-wvfgj\") pod \"7d00b816-699e-4103-9a94-7b37b8d0d5e4\" (UID: \"7d00b816-699e-4103-9a94-7b37b8d0d5e4\") "
Nov 01 03:56:43.239870 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:43.239849    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d00b816-699e-4103-9a94-7b37b8d0d5e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d00b816-699e-4103-9a94-7b37b8d0d5e4-maas-api-openshift-ca\") pod \"7d00b816-699e-4103-9a94-7b37b8d0d5e4\" (UID: \"7d00b816-699e-4103-9a94-7b37b8d0d5e4\") "
Nov 01 03:56:43.239916 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:43.239895    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d00b816-699e-4103-9a94-7b37b8d0d5e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d00b816-699e-4103-9a94-7b37b8d0d5e4-maas-api-xks-ca\") pod \"7d00b816-699e-4103-9a94-7b37b8d0d5e4\" (UID: \"7d00b816-699e-4103-9a94-7b37b8d0d5e4\") "
Nov 01 03:56:43.240181 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:43.240152    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d00b816-699e-4103-9a94-7b37b8d0d5e4-maas-api-openshift-ca" pod "7d00b816-699e-4103-9a94-7b37b8d0d5e4" (UID: "7d00b816-699e-4103-9a94-7b37b8d0d5e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:56:43.240310 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:43.240283    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d00b816-699e-4103-9a94-7b37b8d0d5e4-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:56:43.241790 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:43.241770    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d00b816-699e-4103-9a94-7b37b8d0d5e4-maas-api-xks-ca" pod "7d00b816-699e-4103-9a94-7b37b8d0d5e4" (UID: "7d00b816-699e-4103-9a94-7b37b8d0d5e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:56:43.241880 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:43.241801    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d00b816-699e-4103-9a94-7b37b8d0d5e4-kube-api-access-wvfgj" pod "7d00b816-699e-4103-9a94-7b37b8d0d5e4" (UID: "7d00b816-699e-4103-9a94-7b37b8d0d5e4"). InnerVolumeSpecName "kube-api-access-wvfgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:56:43.341138 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:43.341115    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvfgj\" (UniqueName: \"kubernetes.io/projected/7d00b816-699e-4103-9a94-7b37b8d0d5e4-kube-api-access-wvfgj\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:56:43.341138 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:43.341140    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d00b816-699e-4103-9a94-7b37b8d0d5e4-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:56:43.810343 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:43.810310    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:56:44.082965 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:44.082886    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d8bt5"
Nov 01 03:56:44.082965 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:44.082929    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d8bt5" event={"ID":"7d00b816-699e-4103-9a94-7b37b8d0d5e4","Type":"ContainerDied","Data":"8c8777994c63cde7f72baecd148bcf7dfe8d737f21fd5e8324ca2c414b20ab94"}
Nov 01 03:56:44.083216 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:44.083003    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c8777994c63cde7f72baecd148bcf7dfe8d737f21fd5e8324ca2c414b20ab94"
Nov 01 03:56:49.121798 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:56:49.121711    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:56:53.531844 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:56:53.531802    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:57:10.280015 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:10.279974    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:57:10.474429 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:10.474395    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:57:13.049549 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:13.049515    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d55d5e8-lhq9q"]
Nov 01 03:57:13.098908 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:13.098876    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d55d5e8-lhq9q"]
Nov 01 03:57:13.619356 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:13.619324    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b" path="/var/lib/kubelet/pods/6bcfdfce-7c5c-4aa0-91a1-0ff72f08aa1b/volumes"
Nov 01 03:57:15.647305 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:15.647275    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jwsck"]
Nov 01 03:57:15.647732 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:15.647682    4930 cpu_manager.go:414] "containerMap: removing container" podUID="7d00b816-699e-4103-9a94-7b37b8d0d5e4" containerName="revoke-keys"
Nov 01 03:57:15.647732 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:15.647695    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d00b816-699e-4103-9a94-7b37b8d0d5e4" containerName="revoke-keys"
Nov 01 03:57:15.647820 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:15.647780    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d00b816-699e-4103-9a94-7b37b8d0d5e4" containerName="revoke-keys"
Nov 01 03:57:15.651371 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:15.651355    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jwsck"
Nov 01 03:57:15.709887 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:15.709863    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jwsck"]
Nov 01 03:57:15.755902 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:15.755876    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:57:15.756050 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:15.755939    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:57:15.756050 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:15.755997    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:57:15.756279 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:15.756223    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:57:15.810439 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:15.810409    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7589975d-4a3a-4039-8788-840ae68d295e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-jwsck\" (UID: \"7589975d-4a3a-4039-8788-840ae68d295e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jwsck"
Nov 01 03:57:15.810596 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:15.810460    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7589975d-4a3a-4039-8788-840ae68d295e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-jwsck\" (UID: \"7589975d-4a3a-4039-8788-840ae68d295e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jwsck"
Nov 01 03:57:15.810682 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:15.810655    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2msz\" (UniqueName: \"kubernetes.io/projected/7589975d-4a3a-4039-8788-840ae68d295e-kube-api-access-m2msz\") pod \"maas-api-revoke-sub-046e1343-jwsck\" (UID: \"7589975d-4a3a-4039-8788-840ae68d295e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jwsck"
Nov 01 03:57:15.911309 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:15.911237    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2msz\" (UniqueName: \"kubernetes.io/projected/7589975d-4a3a-4039-8788-840ae68d295e-kube-api-access-m2msz\") pod \"maas-api-revoke-sub-046e1343-jwsck\" (UID: \"7589975d-4a3a-4039-8788-840ae68d295e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jwsck"
Nov 01 03:57:15.911309 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:15.911295    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7589975d-4a3a-4039-8788-840ae68d295e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-jwsck\" (UID: \"7589975d-4a3a-4039-8788-840ae68d295e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jwsck"
Nov 01 03:57:15.911482 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:15.911342    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7589975d-4a3a-4039-8788-840ae68d295e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-jwsck\" (UID: \"7589975d-4a3a-4039-8788-840ae68d295e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jwsck"
Nov 01 03:57:15.912054 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:15.912024    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7589975d-4a3a-4039-8788-840ae68d295e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-jwsck\" (UID: \"7589975d-4a3a-4039-8788-840ae68d295e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jwsck"
Nov 01 03:57:15.913689 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:15.913668    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7589975d-4a3a-4039-8788-840ae68d295e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-jwsck\" (UID: \"7589975d-4a3a-4039-8788-840ae68d295e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jwsck"
Nov 01 03:57:15.972428 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:15.972406    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2msz\" (UniqueName: \"kubernetes.io/projected/7589975d-4a3a-4039-8788-840ae68d295e-kube-api-access-m2msz\") pod \"maas-api-revoke-sub-046e1343-jwsck\" (UID: \"7589975d-4a3a-4039-8788-840ae68d295e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jwsck"
Nov 01 03:57:16.264631 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:16.264515    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jwsck"
Nov 01 03:57:16.638178 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:16.638150    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jwsck"]
Nov 01 03:57:16.639778 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:57:16.639744    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7589975d_4a3a_4039_8788_840ae68d295e.slice/crio-fcce115382ad00ea44f18ae6351dad6cfabceb469713fbce5d94b0af4966a279 WatchSource:0}: Error finding container fcce115382ad00ea44f18ae6351dad6cfabceb469713fbce5d94b0af4966a279: Status 404 returned error can't find the container with id fcce115382ad00ea44f18ae6351dad6cfabceb469713fbce5d94b0af4966a279
Nov 01 03:57:17.321269 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:17.321172    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jwsck" event={"ID":"7589975d-4a3a-4039-8788-840ae68d295e","Type":"ContainerStarted","Data":"da4a11eaab9de794cc945dc6df6a005c1aff874c57fa6c854808220d07775039"}
Nov 01 03:57:17.321269 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:17.321235    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jwsck" event={"ID":"7589975d-4a3a-4039-8788-840ae68d295e","Type":"ContainerStarted","Data":"fcce115382ad00ea44f18ae6351dad6cfabceb469713fbce5d94b0af4966a279"}
Nov 01 03:57:17.442611 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:17.442540    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jwsck" podStartSLOduration=2.442524949 podStartE2EDuration="2.442524949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:57:17.440106312 +0000 UTC m=+3041.963112901" watchObservedRunningTime="2026-11-01 03:57:17.442524949 +0000 UTC m=+3041.965531569"
Nov 01 03:57:18.329326 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:18.329288    4930 generic.go:356] "Generic (PLEG): container finished" podID="7589975d-4a3a-4039-8788-840ae68d295e" containerID="da4a11eaab9de794cc945dc6df6a005c1aff874c57fa6c854808220d07775039" exitCode=0
Nov 01 03:57:18.329746 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:18.329335    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jwsck" event={"ID":"7589975d-4a3a-4039-8788-840ae68d295e","Type":"ContainerDied","Data":"da4a11eaab9de794cc945dc6df6a005c1aff874c57fa6c854808220d07775039"}
Nov 01 03:57:19.121341 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:57:19.121302    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:57:19.479257 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:19.479230    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jwsck"
Nov 01 03:57:19.645704 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:19.645631    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7589975d-4a3a-4039-8788-840ae68d295e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7589975d-4a3a-4039-8788-840ae68d295e-maas-api-openshift-ca\") pod \"7589975d-4a3a-4039-8788-840ae68d295e\" (UID: \"7589975d-4a3a-4039-8788-840ae68d295e\") "
Nov 01 03:57:19.645865 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:19.645721    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7589975d-4a3a-4039-8788-840ae68d295e-kube-api-access-m2msz\" (UniqueName: \"kubernetes.io/projected/7589975d-4a3a-4039-8788-840ae68d295e-kube-api-access-m2msz\") pod \"7589975d-4a3a-4039-8788-840ae68d295e\" (UID: \"7589975d-4a3a-4039-8788-840ae68d295e\") "
Nov 01 03:57:19.645865 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:19.645780    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7589975d-4a3a-4039-8788-840ae68d295e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7589975d-4a3a-4039-8788-840ae68d295e-maas-api-xks-ca\") pod \"7589975d-4a3a-4039-8788-840ae68d295e\" (UID: \"7589975d-4a3a-4039-8788-840ae68d295e\") "
Nov 01 03:57:19.646076 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:19.646050    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7589975d-4a3a-4039-8788-840ae68d295e-maas-api-openshift-ca" pod "7589975d-4a3a-4039-8788-840ae68d295e" (UID: "7589975d-4a3a-4039-8788-840ae68d295e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:57:19.647826 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:19.647800    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7589975d-4a3a-4039-8788-840ae68d295e-kube-api-access-m2msz" pod "7589975d-4a3a-4039-8788-840ae68d295e" (UID: "7589975d-4a3a-4039-8788-840ae68d295e"). InnerVolumeSpecName "kube-api-access-m2msz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:57:19.647908 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:19.647813    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7589975d-4a3a-4039-8788-840ae68d295e-maas-api-xks-ca" pod "7589975d-4a3a-4039-8788-840ae68d295e" (UID: "7589975d-4a3a-4039-8788-840ae68d295e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:57:19.747174 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:19.747132    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7589975d-4a3a-4039-8788-840ae68d295e-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:57:19.747174 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:19.747171    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7589975d-4a3a-4039-8788-840ae68d295e-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:57:19.747174 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:19.747182    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2msz\" (UniqueName: \"kubernetes.io/projected/7589975d-4a3a-4039-8788-840ae68d295e-kube-api-access-m2msz\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:57:20.341510 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:20.341476    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jwsck"
Nov 01 03:57:20.341510 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:20.341490    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jwsck" event={"ID":"7589975d-4a3a-4039-8788-840ae68d295e","Type":"ContainerDied","Data":"fcce115382ad00ea44f18ae6351dad6cfabceb469713fbce5d94b0af4966a279"}
Nov 01 03:57:20.341771 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:20.341544    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcce115382ad00ea44f18ae6351dad6cfabceb469713fbce5d94b0af4966a279"
Nov 01 03:57:26.446689 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:26.446648    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trpfk"]
Nov 01 03:57:26.447281 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:26.447260    4930 cpu_manager.go:414] "containerMap: removing container" podUID="7589975d-4a3a-4039-8788-840ae68d295e" containerName="revoke-keys"
Nov 01 03:57:26.447348 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:26.447283    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7589975d-4a3a-4039-8788-840ae68d295e" containerName="revoke-keys"
Nov 01 03:57:26.447468 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:26.447395    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="7589975d-4a3a-4039-8788-840ae68d295e" containerName="revoke-keys"
Nov 01 03:57:26.451131 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:26.451107    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trpfk"
Nov 01 03:57:26.508000 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:26.507963    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbwm4\" (UniqueName: \"kubernetes.io/projected/f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b-kube-api-access-mbwm4\") pod \"maas-api-revoke-sub-d3fba056-trpfk\" (UID: \"f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trpfk"
Nov 01 03:57:26.508213 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:26.508018    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-trpfk\" (UID: \"f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trpfk"
Nov 01 03:57:26.508213 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:26.508165    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-trpfk\" (UID: \"f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trpfk"
Nov 01 03:57:26.554042 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:26.554007    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:57:26.554042 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:26.554007    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:57:26.554282 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:26.554265    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:57:26.554282 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:26.554273    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:57:26.557054 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:26.557032    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trpfk"]
Nov 01 03:57:26.609104 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:26.609075    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-trpfk\" (UID: \"f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trpfk"
Nov 01 03:57:26.609246 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:26.609171    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbwm4\" (UniqueName: \"kubernetes.io/projected/f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b-kube-api-access-mbwm4\") pod \"maas-api-revoke-sub-d3fba056-trpfk\" (UID: \"f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trpfk"
Nov 01 03:57:26.609246 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:26.609207    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-trpfk\" (UID: \"f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trpfk"
Nov 01 03:57:26.609939 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:26.609918    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-trpfk\" (UID: \"f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trpfk"
Nov 01 03:57:26.611540 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:26.611523    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-trpfk\" (UID: \"f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trpfk"
Nov 01 03:57:26.669781 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:26.669758    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbwm4\" (UniqueName: \"kubernetes.io/projected/f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b-kube-api-access-mbwm4\") pod \"maas-api-revoke-sub-d3fba056-trpfk\" (UID: \"f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trpfk"
Nov 01 03:57:26.772446 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:26.772374    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trpfk"
Nov 01 03:57:27.146127 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:27.146094    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trpfk"]
Nov 01 03:57:27.147520 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:57:27.147493    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4a069dd_1cdf_4bf2_adf5_fcdb56fc0b3b.slice/crio-f4987c22055cf8ec9821e658a305a55676895023e78377c1baa75408359fb62e WatchSource:0}: Error finding container f4987c22055cf8ec9821e658a305a55676895023e78377c1baa75408359fb62e: Status 404 returned error can't find the container with id f4987c22055cf8ec9821e658a305a55676895023e78377c1baa75408359fb62e
Nov 01 03:57:27.384031 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:27.383993    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trpfk" event={"ID":"f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b","Type":"ContainerStarted","Data":"f4987c22055cf8ec9821e658a305a55676895023e78377c1baa75408359fb62e"}
Nov 01 03:57:28.392217 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:28.392133    4930 generic.go:356] "Generic (PLEG): container finished" podID="f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b" containerID="ccefd226612ec3e1520f00e71175ecea83ffabd70229168695bba41d1975870d" exitCode=0
Nov 01 03:57:28.392611 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:28.392223    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trpfk" event={"ID":"f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b","Type":"ContainerDied","Data":"ccefd226612ec3e1520f00e71175ecea83ffabd70229168695bba41d1975870d"}
Nov 01 03:57:29.614375 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:29.614066    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trpfk"
Nov 01 03:57:29.644957 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:29.644925    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b-maas-api-openshift-ca\") pod \"f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b\" (UID: \"f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b\") "
Nov 01 03:57:29.645131 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:29.645003    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b-maas-api-xks-ca\") pod \"f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b\" (UID: \"f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b\") "
Nov 01 03:57:29.645131 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:29.645117    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b-kube-api-access-mbwm4\" (UniqueName: \"kubernetes.io/projected/f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b-kube-api-access-mbwm4\") pod \"f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b\" (UID: \"f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b\") "
Nov 01 03:57:29.645366 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:29.645336    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b-maas-api-openshift-ca" pod "f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b" (UID: "f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:57:29.645590 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:29.645536    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:57:29.648187 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:29.648159    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b-maas-api-xks-ca" pod "f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b" (UID: "f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:57:29.648675 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:29.648650    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b-kube-api-access-mbwm4" pod "f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b" (UID: "f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b"). InnerVolumeSpecName "kube-api-access-mbwm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:57:29.757661 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:29.757346    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbwm4\" (UniqueName: \"kubernetes.io/projected/f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b-kube-api-access-mbwm4\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:57:29.757661 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:29.757387    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:57:30.429714 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:30.429675    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trpfk" event={"ID":"f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b","Type":"ContainerDied","Data":"f4987c22055cf8ec9821e658a305a55676895023e78377c1baa75408359fb62e"}
Nov 01 03:57:30.430010 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:30.429988    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4987c22055cf8ec9821e658a305a55676895023e78377c1baa75408359fb62e"
Nov 01 03:57:30.430207 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:30.430190    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trpfk"
Nov 01 03:57:40.356547 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:40.356515    4930 scope.go:122] "RemoveContainer" containerID="d1430e797cd12b6c5e4e695c05844e180b04a769997102e20b5753200c498da7"
Nov 01 03:57:40.366019 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:40.365998    4930 scope.go:122] "RemoveContainer" containerID="5ec224fbaa70d12831d661b0fb63398236c54040b96bb694d3c93025394a0833"
Nov 01 03:57:40.375222 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:40.375192    4930 scope.go:122] "RemoveContainer" containerID="b853dab771b161fb9f01071dee2eb76e1e015caea386367813bb36c8ba39a5dd"
Nov 01 03:57:44.179451 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:44.179414    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:57:49.120939 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:57:49.120896    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:57:51.049089 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:51.049050    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-541ff960-c986h"]
Nov 01 03:57:51.070529 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:51.070498    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-541ff960-c986h"]
Nov 01 03:57:51.618802 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:51.618769    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0068669-bf76-48d9-8cf4-fd6404431462" path="/var/lib/kubelet/pods/b0068669-bf76-48d9-8cf4-fd6404431462/volumes"
Nov 01 03:57:53.839166 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:57:53.839134    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:58:04.410689 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:04.410654    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rsnp6"]
Nov 01 03:58:04.411086 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:04.411064    4930 cpu_manager.go:414] "containerMap: removing container" podUID="f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b" containerName="revoke-keys"
Nov 01 03:58:04.411086 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:04.411075    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b" containerName="revoke-keys"
Nov 01 03:58:04.411161 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:04.411151    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b" containerName="revoke-keys"
Nov 01 03:58:04.415140 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:04.415122    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rsnp6"
Nov 01 03:58:04.519237 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:04.519214    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:58:04.519402 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:04.519346    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:58:04.519402 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:04.519372    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:58:04.519402 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:04.519389    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:58:04.522153 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:04.522133    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rsnp6"]
Nov 01 03:58:04.578734 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:04.578711    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-rsnp6\" (UID: \"55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rsnp6"
Nov 01 03:58:04.578899 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:04.578760    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wnlr\" (UniqueName: \"kubernetes.io/projected/55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd-kube-api-access-2wnlr\") pod \"maas-api-revoke-sub-ae594979-rsnp6\" (UID: \"55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rsnp6"
Nov 01 03:58:04.578899 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:04.578858    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-rsnp6\" (UID: \"55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rsnp6"
Nov 01 03:58:04.679826 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:04.679758    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-rsnp6\" (UID: \"55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rsnp6"
Nov 01 03:58:04.679826 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:04.679801    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wnlr\" (UniqueName: \"kubernetes.io/projected/55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd-kube-api-access-2wnlr\") pod \"maas-api-revoke-sub-ae594979-rsnp6\" (UID: \"55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rsnp6"
Nov 01 03:58:04.680031 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:04.679830    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-rsnp6\" (UID: \"55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rsnp6"
Nov 01 03:58:04.680448 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:04.680425    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-rsnp6\" (UID: \"55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rsnp6"
Nov 01 03:58:04.682165 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:04.682142    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-rsnp6\" (UID: \"55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rsnp6"
Nov 01 03:58:04.740750 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:04.740720    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wnlr\" (UniqueName: \"kubernetes.io/projected/55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd-kube-api-access-2wnlr\") pod \"maas-api-revoke-sub-ae594979-rsnp6\" (UID: \"55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rsnp6"
Nov 01 03:58:05.027737 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:05.027665    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rsnp6"
Nov 01 03:58:05.402992 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:05.402958    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rsnp6"]
Nov 01 03:58:05.404690 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:58:05.404662    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f00eb5_8c9e_4e42_a266_77a8a5e2f8fd.slice/crio-669e09303864f56abe6ed6335de02dae8c97ef5427c44dfd99c52074e6f4bd95 WatchSource:0}: Error finding container 669e09303864f56abe6ed6335de02dae8c97ef5427c44dfd99c52074e6f4bd95: Status 404 returned error can't find the container with id 669e09303864f56abe6ed6335de02dae8c97ef5427c44dfd99c52074e6f4bd95
Nov 01 03:58:05.703081 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:05.703056    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rsnp6" event={"ID":"55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd","Type":"ContainerStarted","Data":"bfeb771d42d44a92baa66a308928ca88cc271839100df4d07de616ca1106f547"}
Nov 01 03:58:05.703440 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:05.703106    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rsnp6" event={"ID":"55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd","Type":"ContainerStarted","Data":"669e09303864f56abe6ed6335de02dae8c97ef5427c44dfd99c52074e6f4bd95"}
Nov 01 03:58:05.820456 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:05.820401    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rsnp6" podStartSLOduration=1.820387834 podStartE2EDuration="1.820387834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:58:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:58:05.817699214 +0000 UTC m=+3090.340705802" watchObservedRunningTime="2026-11-01 03:58:05.820387834 +0000 UTC m=+3090.343394422"
Nov 01 03:58:06.710170 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:06.710138    4930 generic.go:356] "Generic (PLEG): container finished" podID="55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd" containerID="bfeb771d42d44a92baa66a308928ca88cc271839100df4d07de616ca1106f547" exitCode=0
Nov 01 03:58:06.710592 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:06.710272    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rsnp6" event={"ID":"55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd","Type":"ContainerDied","Data":"bfeb771d42d44a92baa66a308928ca88cc271839100df4d07de616ca1106f547"}
Nov 01 03:58:07.854174 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:07.854149    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rsnp6"
Nov 01 03:58:08.010422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:08.010357    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd-kube-api-access-2wnlr\" (UniqueName: \"kubernetes.io/projected/55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd-kube-api-access-2wnlr\") pod \"55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd\" (UID: \"55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd\") "
Nov 01 03:58:08.010422 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:08.010404    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd-maas-api-xks-ca\") pod \"55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd\" (UID: \"55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd\") "
Nov 01 03:58:08.010673 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:08.010507    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd-maas-api-openshift-ca\") pod \"55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd\" (UID: \"55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd\") "
Nov 01 03:58:08.010912 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:08.010884    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd-maas-api-openshift-ca" pod "55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd" (UID: "55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:58:08.012421 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:08.012395    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd-maas-api-xks-ca" pod "55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd" (UID: "55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:58:08.012530 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:08.012511    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd-kube-api-access-2wnlr" pod "55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd" (UID: "55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd"). InnerVolumeSpecName "kube-api-access-2wnlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:58:08.111359 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:08.111324    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:58:08.111359 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:08.111358    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wnlr\" (UniqueName: \"kubernetes.io/projected/55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd-kube-api-access-2wnlr\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:58:08.111359 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:08.111368    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:58:08.721935 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:08.721902    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rsnp6" event={"ID":"55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd","Type":"ContainerDied","Data":"669e09303864f56abe6ed6335de02dae8c97ef5427c44dfd99c52074e6f4bd95"}
Nov 01 03:58:08.721935 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:08.721925    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rsnp6"
Nov 01 03:58:08.721935 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:08.721949    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="669e09303864f56abe6ed6335de02dae8c97ef5427c44dfd99c52074e6f4bd95"
Nov 01 03:58:19.122298 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:58:19.122255    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:58:27.048878 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:27.048842    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f1690f3-9r4sv"]
Nov 01 03:58:27.098129 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:27.098098    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f1690f3-9r4sv"]
Nov 01 03:58:27.619612 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:27.619578    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22002d07-ffff-4f2f-b9d7-2751eae471d1" path="/var/lib/kubelet/pods/22002d07-ffff-4f2f-b9d7-2751eae471d1/volumes"
Nov 01 03:58:32.515400 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:32.515366    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6v5jl"]
Nov 01 03:58:32.515820 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:32.515807    4930 cpu_manager.go:414] "containerMap: removing container" podUID="55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd" containerName="revoke-keys"
Nov 01 03:58:32.515868 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:32.515821    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd" containerName="revoke-keys"
Nov 01 03:58:32.515912 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:32.515906    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd" containerName="revoke-keys"
Nov 01 03:58:32.519804 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:32.519787    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6v5jl"
Nov 01 03:58:32.598319 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:32.598292    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f5f13d9-80f9-40eb-a583-e622e57ee99d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-6v5jl\" (UID: \"9f5f13d9-80f9-40eb-a583-e622e57ee99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6v5jl"
Nov 01 03:58:32.598483 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:32.598343    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7whs\" (UniqueName: \"kubernetes.io/projected/9f5f13d9-80f9-40eb-a583-e622e57ee99d-kube-api-access-w7whs\") pod \"maas-api-revoke-sub-a4aef7d7-6v5jl\" (UID: \"9f5f13d9-80f9-40eb-a583-e622e57ee99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6v5jl"
Nov 01 03:58:32.598614 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:32.598551    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f5f13d9-80f9-40eb-a583-e622e57ee99d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-6v5jl\" (UID: \"9f5f13d9-80f9-40eb-a583-e622e57ee99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6v5jl"
Nov 01 03:58:32.624196 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:32.624170    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:58:32.624196 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:32.624170    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:58:32.624356 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:32.624184    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:58:32.624548 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:32.624530    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:58:32.627357 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:32.627331    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6v5jl"]
Nov 01 03:58:32.699167 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:32.699136    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f5f13d9-80f9-40eb-a583-e622e57ee99d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-6v5jl\" (UID: \"9f5f13d9-80f9-40eb-a583-e622e57ee99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6v5jl"
Nov 01 03:58:32.699303 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:32.699175    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7whs\" (UniqueName: \"kubernetes.io/projected/9f5f13d9-80f9-40eb-a583-e622e57ee99d-kube-api-access-w7whs\") pod \"maas-api-revoke-sub-a4aef7d7-6v5jl\" (UID: \"9f5f13d9-80f9-40eb-a583-e622e57ee99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6v5jl"
Nov 01 03:58:32.699403 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:32.699383    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f5f13d9-80f9-40eb-a583-e622e57ee99d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-6v5jl\" (UID: \"9f5f13d9-80f9-40eb-a583-e622e57ee99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6v5jl"
Nov 01 03:58:32.699917 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:32.699892    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f5f13d9-80f9-40eb-a583-e622e57ee99d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-6v5jl\" (UID: \"9f5f13d9-80f9-40eb-a583-e622e57ee99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6v5jl"
Nov 01 03:58:32.701847 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:32.701825    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f5f13d9-80f9-40eb-a583-e622e57ee99d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-6v5jl\" (UID: \"9f5f13d9-80f9-40eb-a583-e622e57ee99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6v5jl"
Nov 01 03:58:32.759755 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:32.759731    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7whs\" (UniqueName: \"kubernetes.io/projected/9f5f13d9-80f9-40eb-a583-e622e57ee99d-kube-api-access-w7whs\") pod \"maas-api-revoke-sub-a4aef7d7-6v5jl\" (UID: \"9f5f13d9-80f9-40eb-a583-e622e57ee99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6v5jl"
Nov 01 03:58:32.833252 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:32.833188    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6v5jl"
Nov 01 03:58:33.211158 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:33.211122    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6v5jl"]
Nov 01 03:58:33.211791 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:58:33.211759    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f5f13d9_80f9_40eb_a583_e622e57ee99d.slice/crio-d35662817292152768651424dac798b7a9bb48d432b734e2e01c7164aedd13e9 WatchSource:0}: Error finding container d35662817292152768651424dac798b7a9bb48d432b734e2e01c7164aedd13e9: Status 404 returned error can't find the container with id d35662817292152768651424dac798b7a9bb48d432b734e2e01c7164aedd13e9
Nov 01 03:58:33.918069 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:33.918029    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6v5jl" event={"ID":"9f5f13d9-80f9-40eb-a583-e622e57ee99d","Type":"ContainerStarted","Data":"2651565c673d69df216db29fc376953be126d4b1e047726fe2ccbade991b0ed7"}
Nov 01 03:58:33.918521 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:33.918080    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6v5jl" event={"ID":"9f5f13d9-80f9-40eb-a583-e622e57ee99d","Type":"ContainerStarted","Data":"d35662817292152768651424dac798b7a9bb48d432b734e2e01c7164aedd13e9"}
Nov 01 03:58:34.035920 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:34.035870    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6v5jl" podStartSLOduration=2.035854958 podStartE2EDuration="2.035854958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:58:34.033121268 +0000 UTC m=+3118.556127856" watchObservedRunningTime="2026-11-01 03:58:34.035854958 +0000 UTC m=+3118.558861546"
Nov 01 03:58:34.925073 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:34.925042    4930 generic.go:356] "Generic (PLEG): container finished" podID="9f5f13d9-80f9-40eb-a583-e622e57ee99d" containerID="2651565c673d69df216db29fc376953be126d4b1e047726fe2ccbade991b0ed7" exitCode=0
Nov 01 03:58:34.925472 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:34.925137    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6v5jl" event={"ID":"9f5f13d9-80f9-40eb-a583-e622e57ee99d","Type":"ContainerDied","Data":"2651565c673d69df216db29fc376953be126d4b1e047726fe2ccbade991b0ed7"}
Nov 01 03:58:36.073466 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:36.073434    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6v5jl"
Nov 01 03:58:36.131638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:36.131610    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f5f13d9-80f9-40eb-a583-e622e57ee99d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f5f13d9-80f9-40eb-a583-e622e57ee99d-maas-api-openshift-ca\") pod \"9f5f13d9-80f9-40eb-a583-e622e57ee99d\" (UID: \"9f5f13d9-80f9-40eb-a583-e622e57ee99d\") "
Nov 01 03:58:36.131772 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:36.131672    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f5f13d9-80f9-40eb-a583-e622e57ee99d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f5f13d9-80f9-40eb-a583-e622e57ee99d-maas-api-xks-ca\") pod \"9f5f13d9-80f9-40eb-a583-e622e57ee99d\" (UID: \"9f5f13d9-80f9-40eb-a583-e622e57ee99d\") "
Nov 01 03:58:36.131891 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:36.131871    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f5f13d9-80f9-40eb-a583-e622e57ee99d-kube-api-access-w7whs\" (UniqueName: \"kubernetes.io/projected/9f5f13d9-80f9-40eb-a583-e622e57ee99d-kube-api-access-w7whs\") pod \"9f5f13d9-80f9-40eb-a583-e622e57ee99d\" (UID: \"9f5f13d9-80f9-40eb-a583-e622e57ee99d\") "
Nov 01 03:58:36.131997 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:36.131977    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f5f13d9-80f9-40eb-a583-e622e57ee99d-maas-api-openshift-ca" pod "9f5f13d9-80f9-40eb-a583-e622e57ee99d" (UID: "9f5f13d9-80f9-40eb-a583-e622e57ee99d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:58:36.132188 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:36.132170    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f5f13d9-80f9-40eb-a583-e622e57ee99d-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:58:36.133743 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:36.133722    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f5f13d9-80f9-40eb-a583-e622e57ee99d-maas-api-xks-ca" pod "9f5f13d9-80f9-40eb-a583-e622e57ee99d" (UID: "9f5f13d9-80f9-40eb-a583-e622e57ee99d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:58:36.133841 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:36.133824    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f5f13d9-80f9-40eb-a583-e622e57ee99d-kube-api-access-w7whs" pod "9f5f13d9-80f9-40eb-a583-e622e57ee99d" (UID: "9f5f13d9-80f9-40eb-a583-e622e57ee99d"). InnerVolumeSpecName "kube-api-access-w7whs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:58:36.233237 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:36.233176    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f5f13d9-80f9-40eb-a583-e622e57ee99d-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:58:36.233237 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:36.233205    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7whs\" (UniqueName: \"kubernetes.io/projected/9f5f13d9-80f9-40eb-a583-e622e57ee99d-kube-api-access-w7whs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:58:36.938290 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:36.938255    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6v5jl" event={"ID":"9f5f13d9-80f9-40eb-a583-e622e57ee99d","Type":"ContainerDied","Data":"d35662817292152768651424dac798b7a9bb48d432b734e2e01c7164aedd13e9"}
Nov 01 03:58:36.938474 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:36.938309    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d35662817292152768651424dac798b7a9bb48d432b734e2e01c7164aedd13e9"
Nov 01 03:58:36.938474 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:36.938269    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6v5jl"
Nov 01 03:58:40.405107 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:40.405075    4930 scope.go:122] "RemoveContainer" containerID="d4918dec59c25f9647744d3c957d6892b52149381e3b4dad0d6232ad7c0fe355"
Nov 01 03:58:40.414401 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:40.414381    4930 scope.go:122] "RemoveContainer" containerID="fa688ebb814860864c8eef6e3bd4dbd58a9e7ce3fb34a1361f43b3bf3bd3deaa"
Nov 01 03:58:40.422631 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:40.422612    4930 scope.go:122] "RemoveContainer" containerID="92526190e471c1557608858d0ba19fad653500ee19005c7e48a91659dbd475e5"
Nov 01 03:58:40.430437 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:40.430416    4930 scope.go:122] "RemoveContainer" containerID="5a8de80b665c7cf0aa520f9c7d5f02f2f10bd055ad87d5e5b6b2db18f0c7338d"
Nov 01 03:58:40.438258 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:40.438239    4930 scope.go:122] "RemoveContainer" containerID="ec6a1ccf54c90759226769d7a212a2b6e80f13e040497d4f566b7cbe252d42fa"
Nov 01 03:58:44.528507 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:44.528462    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:58:49.120606 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:58:49.120552    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:58:54.294581 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:54.292067    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:58:55.057653 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:55.057613    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3fb7a87-qbxs5"]
Nov 01 03:58:55.105923 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:55.105890    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3fb7a87-qbxs5"]
Nov 01 03:58:55.619034 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:58:55.618992    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3b79a6d-601b-4348-8ac2-620145e0bbe1" path="/var/lib/kubelet/pods/f3b79a6d-601b-4348-8ac2-620145e0bbe1/volumes"
Nov 01 03:59:03.295122 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:03.295087    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-z9fd9"]
Nov 01 03:59:03.295549 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:03.295535    4930 cpu_manager.go:414] "containerMap: removing container" podUID="9f5f13d9-80f9-40eb-a583-e622e57ee99d" containerName="revoke-keys"
Nov 01 03:59:03.295623 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:03.295550    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f5f13d9-80f9-40eb-a583-e622e57ee99d" containerName="revoke-keys"
Nov 01 03:59:03.295665 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:03.295638    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f5f13d9-80f9-40eb-a583-e622e57ee99d" containerName="revoke-keys"
Nov 01 03:59:03.300053 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:03.300035    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-z9fd9"
Nov 01 03:59:03.358520 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:03.358495    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-z9fd9"]
Nov 01 03:59:03.404134 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:03.404091    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:59:03.404134 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:03.404114    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:59:03.404134 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:03.404095    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:59:03.404391 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:03.404163    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:59:03.433929 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:03.433899    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8144c0f7-1376-4854-85c4-3bdc5e03dd86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-z9fd9\" (UID: \"8144c0f7-1376-4854-85c4-3bdc5e03dd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-z9fd9"
Nov 01 03:59:03.434087 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:03.433951    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8qmf\" (UniqueName: \"kubernetes.io/projected/8144c0f7-1376-4854-85c4-3bdc5e03dd86-kube-api-access-b8qmf\") pod \"maas-api-revoke-sub-79805ca9-z9fd9\" (UID: \"8144c0f7-1376-4854-85c4-3bdc5e03dd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-z9fd9"
Nov 01 03:59:03.434146 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:03.434086    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8144c0f7-1376-4854-85c4-3bdc5e03dd86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-z9fd9\" (UID: \"8144c0f7-1376-4854-85c4-3bdc5e03dd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-z9fd9"
Nov 01 03:59:03.534810 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:03.534776    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8144c0f7-1376-4854-85c4-3bdc5e03dd86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-z9fd9\" (UID: \"8144c0f7-1376-4854-85c4-3bdc5e03dd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-z9fd9"
Nov 01 03:59:03.534981 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:03.534822    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8qmf\" (UniqueName: \"kubernetes.io/projected/8144c0f7-1376-4854-85c4-3bdc5e03dd86-kube-api-access-b8qmf\") pod \"maas-api-revoke-sub-79805ca9-z9fd9\" (UID: \"8144c0f7-1376-4854-85c4-3bdc5e03dd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-z9fd9"
Nov 01 03:59:03.534981 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:03.534858    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8144c0f7-1376-4854-85c4-3bdc5e03dd86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-z9fd9\" (UID: \"8144c0f7-1376-4854-85c4-3bdc5e03dd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-z9fd9"
Nov 01 03:59:03.535545 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:03.535522    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8144c0f7-1376-4854-85c4-3bdc5e03dd86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-z9fd9\" (UID: \"8144c0f7-1376-4854-85c4-3bdc5e03dd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-z9fd9"
Nov 01 03:59:03.537173 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:03.537154    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8144c0f7-1376-4854-85c4-3bdc5e03dd86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-z9fd9\" (UID: \"8144c0f7-1376-4854-85c4-3bdc5e03dd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-z9fd9"
Nov 01 03:59:03.595663 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:03.595640    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8qmf\" (UniqueName: \"kubernetes.io/projected/8144c0f7-1376-4854-85c4-3bdc5e03dd86-kube-api-access-b8qmf\") pod \"maas-api-revoke-sub-79805ca9-z9fd9\" (UID: \"8144c0f7-1376-4854-85c4-3bdc5e03dd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-z9fd9"
Nov 01 03:59:03.612767 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:03.612745    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-z9fd9"
Nov 01 03:59:03.987935 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:03.987907    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-z9fd9"]
Nov 01 03:59:03.988827 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:59:03.988789    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8144c0f7_1376_4854_85c4_3bdc5e03dd86.slice/crio-26158de83d5f83804eed5e6bf8f1fb613f97610480fc7f7231be51fd45c9576e WatchSource:0}: Error finding container 26158de83d5f83804eed5e6bf8f1fb613f97610480fc7f7231be51fd45c9576e: Status 404 returned error can't find the container with id 26158de83d5f83804eed5e6bf8f1fb613f97610480fc7f7231be51fd45c9576e
Nov 01 03:59:04.127255 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:04.127220    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-z9fd9" event={"ID":"8144c0f7-1376-4854-85c4-3bdc5e03dd86","Type":"ContainerStarted","Data":"26158de83d5f83804eed5e6bf8f1fb613f97610480fc7f7231be51fd45c9576e"}
Nov 01 03:59:05.134638 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:05.134602    4930 generic.go:356] "Generic (PLEG): container finished" podID="8144c0f7-1376-4854-85c4-3bdc5e03dd86" containerID="50f73c007d717a4dfc69e36db2b354ac8446ec5cf1167452fa7e4fb9036c23b3" exitCode=0
Nov 01 03:59:05.135050 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:05.134655    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-z9fd9" event={"ID":"8144c0f7-1376-4854-85c4-3bdc5e03dd86","Type":"ContainerDied","Data":"50f73c007d717a4dfc69e36db2b354ac8446ec5cf1167452fa7e4fb9036c23b3"}
Nov 01 03:59:06.289530 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:06.289501    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-z9fd9"
Nov 01 03:59:06.460311 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:06.460234    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8144c0f7-1376-4854-85c4-3bdc5e03dd86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8144c0f7-1376-4854-85c4-3bdc5e03dd86-maas-api-xks-ca\") pod \"8144c0f7-1376-4854-85c4-3bdc5e03dd86\" (UID: \"8144c0f7-1376-4854-85c4-3bdc5e03dd86\") "
Nov 01 03:59:06.460311 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:06.460282    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8144c0f7-1376-4854-85c4-3bdc5e03dd86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8144c0f7-1376-4854-85c4-3bdc5e03dd86-maas-api-openshift-ca\") pod \"8144c0f7-1376-4854-85c4-3bdc5e03dd86\" (UID: \"8144c0f7-1376-4854-85c4-3bdc5e03dd86\") "
Nov 01 03:59:06.460578 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:06.460346    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8144c0f7-1376-4854-85c4-3bdc5e03dd86-kube-api-access-b8qmf\" (UniqueName: \"kubernetes.io/projected/8144c0f7-1376-4854-85c4-3bdc5e03dd86-kube-api-access-b8qmf\") pod \"8144c0f7-1376-4854-85c4-3bdc5e03dd86\" (UID: \"8144c0f7-1376-4854-85c4-3bdc5e03dd86\") "
Nov 01 03:59:06.460743 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:06.460717    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8144c0f7-1376-4854-85c4-3bdc5e03dd86-maas-api-openshift-ca" pod "8144c0f7-1376-4854-85c4-3bdc5e03dd86" (UID: "8144c0f7-1376-4854-85c4-3bdc5e03dd86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:59:06.462349 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:06.462326    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8144c0f7-1376-4854-85c4-3bdc5e03dd86-maas-api-xks-ca" pod "8144c0f7-1376-4854-85c4-3bdc5e03dd86" (UID: "8144c0f7-1376-4854-85c4-3bdc5e03dd86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:59:06.462455 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:06.462401    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8144c0f7-1376-4854-85c4-3bdc5e03dd86-kube-api-access-b8qmf" pod "8144c0f7-1376-4854-85c4-3bdc5e03dd86" (UID: "8144c0f7-1376-4854-85c4-3bdc5e03dd86"). InnerVolumeSpecName "kube-api-access-b8qmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:59:06.561229 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:06.561205    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8qmf\" (UniqueName: \"kubernetes.io/projected/8144c0f7-1376-4854-85c4-3bdc5e03dd86-kube-api-access-b8qmf\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:59:06.561229 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:06.561231    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8144c0f7-1376-4854-85c4-3bdc5e03dd86-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:59:06.561386 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:06.561241    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8144c0f7-1376-4854-85c4-3bdc5e03dd86-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:59:07.147386 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:07.147352    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-z9fd9"
Nov 01 03:59:07.147595 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:07.147383    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-z9fd9" event={"ID":"8144c0f7-1376-4854-85c4-3bdc5e03dd86","Type":"ContainerDied","Data":"26158de83d5f83804eed5e6bf8f1fb613f97610480fc7f7231be51fd45c9576e"}
Nov 01 03:59:07.147595 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:07.147436    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26158de83d5f83804eed5e6bf8f1fb613f97610480fc7f7231be51fd45c9576e"
Nov 01 03:59:19.141634 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:59:19.141591    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:59:27.125263 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:27.125226    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rk84h"]
Nov 01 03:59:27.125789 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:27.125690    4930 cpu_manager.go:414] "containerMap: removing container" podUID="8144c0f7-1376-4854-85c4-3bdc5e03dd86" containerName="revoke-keys"
Nov 01 03:59:27.125789 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:27.125703    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8144c0f7-1376-4854-85c4-3bdc5e03dd86" containerName="revoke-keys"
Nov 01 03:59:27.125789 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:27.125787    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="8144c0f7-1376-4854-85c4-3bdc5e03dd86" containerName="revoke-keys"
Nov 01 03:59:27.129035 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:27.129013    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rk84h"
Nov 01 03:59:27.202572 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:27.202533    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eafdd6e0-4d42-4ba7-b342-df7fd1179a5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-rk84h\" (UID: \"eafdd6e0-4d42-4ba7-b342-df7fd1179a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rk84h"
Nov 01 03:59:27.202756 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:27.202639    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eafdd6e0-4d42-4ba7-b342-df7fd1179a5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-rk84h\" (UID: \"eafdd6e0-4d42-4ba7-b342-df7fd1179a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rk84h"
Nov 01 03:59:27.202756 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:27.202730    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd9l2\" (UniqueName: \"kubernetes.io/projected/eafdd6e0-4d42-4ba7-b342-df7fd1179a5e-kube-api-access-xd9l2\") pod \"maas-api-revoke-sub-0b7e8edb-rk84h\" (UID: \"eafdd6e0-4d42-4ba7-b342-df7fd1179a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rk84h"
Nov 01 03:59:27.237633 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:27.237606    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:59:27.237801 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:27.237780    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:59:27.237851 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:27.237784    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:59:27.237851 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:27.237787    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:59:27.240645 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:27.240623    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rk84h"]
Nov 01 03:59:27.303720 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:27.303695    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd9l2\" (UniqueName: \"kubernetes.io/projected/eafdd6e0-4d42-4ba7-b342-df7fd1179a5e-kube-api-access-xd9l2\") pod \"maas-api-revoke-sub-0b7e8edb-rk84h\" (UID: \"eafdd6e0-4d42-4ba7-b342-df7fd1179a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rk84h"
Nov 01 03:59:27.303859 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:27.303799    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eafdd6e0-4d42-4ba7-b342-df7fd1179a5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-rk84h\" (UID: \"eafdd6e0-4d42-4ba7-b342-df7fd1179a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rk84h"
Nov 01 03:59:27.303926 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:27.303841    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eafdd6e0-4d42-4ba7-b342-df7fd1179a5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-rk84h\" (UID: \"eafdd6e0-4d42-4ba7-b342-df7fd1179a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rk84h"
Nov 01 03:59:27.304568 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:27.304538    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eafdd6e0-4d42-4ba7-b342-df7fd1179a5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-rk84h\" (UID: \"eafdd6e0-4d42-4ba7-b342-df7fd1179a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rk84h"
Nov 01 03:59:27.306080 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:27.306064    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eafdd6e0-4d42-4ba7-b342-df7fd1179a5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-rk84h\" (UID: \"eafdd6e0-4d42-4ba7-b342-df7fd1179a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rk84h"
Nov 01 03:59:27.364077 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:27.364055    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd9l2\" (UniqueName: \"kubernetes.io/projected/eafdd6e0-4d42-4ba7-b342-df7fd1179a5e-kube-api-access-xd9l2\") pod \"maas-api-revoke-sub-0b7e8edb-rk84h\" (UID: \"eafdd6e0-4d42-4ba7-b342-df7fd1179a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rk84h"
Nov 01 03:59:27.442306 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:27.442253    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rk84h"
Nov 01 03:59:27.812233 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:27.812201    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rk84h"]
Nov 01 03:59:27.813836 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:59:27.813803    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeafdd6e0_4d42_4ba7_b342_df7fd1179a5e.slice/crio-9bdea2a546ce606c7d2669a06d48c853375084c19656b3b0c3af65dbe97be9ca WatchSource:0}: Error finding container 9bdea2a546ce606c7d2669a06d48c853375084c19656b3b0c3af65dbe97be9ca: Status 404 returned error can't find the container with id 9bdea2a546ce606c7d2669a06d48c853375084c19656b3b0c3af65dbe97be9ca
Nov 01 03:59:28.382385 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:28.382347    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rk84h" event={"ID":"eafdd6e0-4d42-4ba7-b342-df7fd1179a5e","Type":"ContainerStarted","Data":"d3c07d87e876284b71748627c25ce2e2a562d4d20b73ba69d93b5f534d2d7455"}
Nov 01 03:59:28.382842 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:28.382424    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rk84h" event={"ID":"eafdd6e0-4d42-4ba7-b342-df7fd1179a5e","Type":"ContainerStarted","Data":"9bdea2a546ce606c7d2669a06d48c853375084c19656b3b0c3af65dbe97be9ca"}
Nov 01 03:59:28.499895 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:28.499851    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rk84h" podStartSLOduration=1.499835981 podStartE2EDuration="1.499835981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 03:59:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 03:59:28.497673126 +0000 UTC m=+3173.020679717" watchObservedRunningTime="2026-11-01 03:59:28.499835981 +0000 UTC m=+3173.022842581"
Nov 01 03:59:29.389972 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:29.389938    4930 generic.go:356] "Generic (PLEG): container finished" podID="eafdd6e0-4d42-4ba7-b342-df7fd1179a5e" containerID="d3c07d87e876284b71748627c25ce2e2a562d4d20b73ba69d93b5f534d2d7455" exitCode=0
Nov 01 03:59:29.390386 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:29.390015    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rk84h" event={"ID":"eafdd6e0-4d42-4ba7-b342-df7fd1179a5e","Type":"ContainerDied","Data":"d3c07d87e876284b71748627c25ce2e2a562d4d20b73ba69d93b5f534d2d7455"}
Nov 01 03:59:30.531991 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:30.531962    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rk84h"
Nov 01 03:59:30.633409 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:30.633379    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eafdd6e0-4d42-4ba7-b342-df7fd1179a5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eafdd6e0-4d42-4ba7-b342-df7fd1179a5e-maas-api-xks-ca\") pod \"eafdd6e0-4d42-4ba7-b342-df7fd1179a5e\" (UID: \"eafdd6e0-4d42-4ba7-b342-df7fd1179a5e\") "
Nov 01 03:59:30.633583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:30.633442    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eafdd6e0-4d42-4ba7-b342-df7fd1179a5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eafdd6e0-4d42-4ba7-b342-df7fd1179a5e-maas-api-openshift-ca\") pod \"eafdd6e0-4d42-4ba7-b342-df7fd1179a5e\" (UID: \"eafdd6e0-4d42-4ba7-b342-df7fd1179a5e\") "
Nov 01 03:59:30.633583 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:30.633538    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eafdd6e0-4d42-4ba7-b342-df7fd1179a5e-kube-api-access-xd9l2\" (UniqueName: \"kubernetes.io/projected/eafdd6e0-4d42-4ba7-b342-df7fd1179a5e-kube-api-access-xd9l2\") pod \"eafdd6e0-4d42-4ba7-b342-df7fd1179a5e\" (UID: \"eafdd6e0-4d42-4ba7-b342-df7fd1179a5e\") "
Nov 01 03:59:30.633819 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:30.633799    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eafdd6e0-4d42-4ba7-b342-df7fd1179a5e-maas-api-openshift-ca" pod "eafdd6e0-4d42-4ba7-b342-df7fd1179a5e" (UID: "eafdd6e0-4d42-4ba7-b342-df7fd1179a5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:59:30.633979 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:30.633960    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eafdd6e0-4d42-4ba7-b342-df7fd1179a5e-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:59:30.635576 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:30.635540    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eafdd6e0-4d42-4ba7-b342-df7fd1179a5e-kube-api-access-xd9l2" pod "eafdd6e0-4d42-4ba7-b342-df7fd1179a5e" (UID: "eafdd6e0-4d42-4ba7-b342-df7fd1179a5e"). InnerVolumeSpecName "kube-api-access-xd9l2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:59:30.635675 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:30.635583    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eafdd6e0-4d42-4ba7-b342-df7fd1179a5e-maas-api-xks-ca" pod "eafdd6e0-4d42-4ba7-b342-df7fd1179a5e" (UID: "eafdd6e0-4d42-4ba7-b342-df7fd1179a5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:59:30.735485 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:30.735401    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd9l2\" (UniqueName: \"kubernetes.io/projected/eafdd6e0-4d42-4ba7-b342-df7fd1179a5e-kube-api-access-xd9l2\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:59:30.735485 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:30.735437    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eafdd6e0-4d42-4ba7-b342-df7fd1179a5e-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:59:31.403911 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:31.403877    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rk84h"
Nov 01 03:59:31.403911 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:31.403897    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rk84h" event={"ID":"eafdd6e0-4d42-4ba7-b342-df7fd1179a5e","Type":"ContainerDied","Data":"9bdea2a546ce606c7d2669a06d48c853375084c19656b3b0c3af65dbe97be9ca"}
Nov 01 03:59:31.404175 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:31.403959    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bdea2a546ce606c7d2669a06d48c853375084c19656b3b0c3af65dbe97be9ca"
Nov 01 03:59:38.490480 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:38.490444    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sgnfd"]
Nov 01 03:59:38.490904 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:38.490869    4930 cpu_manager.go:414] "containerMap: removing container" podUID="eafdd6e0-4d42-4ba7-b342-df7fd1179a5e" containerName="revoke-keys"
Nov 01 03:59:38.490904 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:38.490882    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eafdd6e0-4d42-4ba7-b342-df7fd1179a5e" containerName="revoke-keys"
Nov 01 03:59:38.490987 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:38.490965    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="eafdd6e0-4d42-4ba7-b342-df7fd1179a5e" containerName="revoke-keys"
Nov 01 03:59:38.494046 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:38.494025    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sgnfd"
Nov 01 03:59:38.546105 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:38.546070    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 03:59:38.595790 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:38.595755    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 03:59:38.595790 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:38.595755    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 03:59:38.595988 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:38.595771    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 03:59:38.598663 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:38.598640    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sgnfd"]
Nov 01 03:59:38.606057 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:38.606032    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8ls4\" (UniqueName: \"kubernetes.io/projected/eb0b8932-7fb3-4867-919b-4d6b422a8877-kube-api-access-t8ls4\") pod \"maas-api-revoke-sub-3d4156f1-sgnfd\" (UID: \"eb0b8932-7fb3-4867-919b-4d6b422a8877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sgnfd"
Nov 01 03:59:38.606185 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:38.606084    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb0b8932-7fb3-4867-919b-4d6b422a8877-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-sgnfd\" (UID: \"eb0b8932-7fb3-4867-919b-4d6b422a8877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sgnfd"
Nov 01 03:59:38.606274 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:38.606256    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb0b8932-7fb3-4867-919b-4d6b422a8877-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-sgnfd\" (UID: \"eb0b8932-7fb3-4867-919b-4d6b422a8877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sgnfd"
Nov 01 03:59:38.707507 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:38.707470    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb0b8932-7fb3-4867-919b-4d6b422a8877-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-sgnfd\" (UID: \"eb0b8932-7fb3-4867-919b-4d6b422a8877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sgnfd"
Nov 01 03:59:38.707698 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:38.707522    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8ls4\" (UniqueName: \"kubernetes.io/projected/eb0b8932-7fb3-4867-919b-4d6b422a8877-kube-api-access-t8ls4\") pod \"maas-api-revoke-sub-3d4156f1-sgnfd\" (UID: \"eb0b8932-7fb3-4867-919b-4d6b422a8877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sgnfd"
Nov 01 03:59:38.707698 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:38.707589    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb0b8932-7fb3-4867-919b-4d6b422a8877-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-sgnfd\" (UID: \"eb0b8932-7fb3-4867-919b-4d6b422a8877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sgnfd"
Nov 01 03:59:38.708262 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:38.708240    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb0b8932-7fb3-4867-919b-4d6b422a8877-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-sgnfd\" (UID: \"eb0b8932-7fb3-4867-919b-4d6b422a8877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sgnfd"
Nov 01 03:59:38.709922 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:38.709898    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb0b8932-7fb3-4867-919b-4d6b422a8877-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-sgnfd\" (UID: \"eb0b8932-7fb3-4867-919b-4d6b422a8877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sgnfd"
Nov 01 03:59:38.768450 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:38.768398    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8ls4\" (UniqueName: \"kubernetes.io/projected/eb0b8932-7fb3-4867-919b-4d6b422a8877-kube-api-access-t8ls4\") pod \"maas-api-revoke-sub-3d4156f1-sgnfd\" (UID: \"eb0b8932-7fb3-4867-919b-4d6b422a8877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sgnfd"
Nov 01 03:59:38.808812 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:38.808788    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sgnfd"
Nov 01 03:59:39.183262 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:39.183234    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sgnfd"]
Nov 01 03:59:39.184646 ip-10-0-141-122 kubenswrapper[4930]: W1101 03:59:39.184616    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb0b8932_7fb3_4867_919b_4d6b422a8877.slice/crio-65746f5ca16197f4ad2a6e3c8bbcd467e9b0c765c9825a91c0f3280da92e278f WatchSource:0}: Error finding container 65746f5ca16197f4ad2a6e3c8bbcd467e9b0c765c9825a91c0f3280da92e278f: Status 404 returned error can't find the container with id 65746f5ca16197f4ad2a6e3c8bbcd467e9b0c765c9825a91c0f3280da92e278f
Nov 01 03:59:39.451429 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:39.451344    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sgnfd" event={"ID":"eb0b8932-7fb3-4867-919b-4d6b422a8877","Type":"ContainerStarted","Data":"65746f5ca16197f4ad2a6e3c8bbcd467e9b0c765c9825a91c0f3280da92e278f"}
Nov 01 03:59:40.462575 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:40.461694    4930 generic.go:356] "Generic (PLEG): container finished" podID="eb0b8932-7fb3-4867-919b-4d6b422a8877" containerID="40a8c0bdeaf311bfb618414fbb11cd32b16bc86af375e51ef17174f0ccd68b89" exitCode=0
Nov 01 03:59:40.462575 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:40.461915    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sgnfd" event={"ID":"eb0b8932-7fb3-4867-919b-4d6b422a8877","Type":"ContainerDied","Data":"40a8c0bdeaf311bfb618414fbb11cd32b16bc86af375e51ef17174f0ccd68b89"}
Nov 01 03:59:40.498646 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:40.498410    4930 scope.go:122] "RemoveContainer" containerID="179c9fa30c155cffa31bc0240c1a0dbffdc44a505a2911082ce33e4a4f650a5c"
Nov 01 03:59:40.547835 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:40.544406    4930 scope.go:122] "RemoveContainer" containerID="369b41775f0bb02752a5a9ec10fc5de9f01b28fe0271d74fba154f45b5e73f41"
Nov 01 03:59:40.591580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:40.580989    4930 scope.go:122] "RemoveContainer" containerID="3932c3aada2c330e20152084459fac969d882e2fb3085fddc5f06696dc60f87a"
Nov 01 03:59:40.679852 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:40.679808    4930 scope.go:122] "RemoveContainer" containerID="f3e1ac6f52840531f0d254fe28108cc09991d35a678836245ed993fb65c7998f"
Nov 01 03:59:42.376580 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:42.370783    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sgnfd"
Nov 01 03:59:42.452380 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:42.452344    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb0b8932-7fb3-4867-919b-4d6b422a8877-kube-api-access-t8ls4\" (UniqueName: \"kubernetes.io/projected/eb0b8932-7fb3-4867-919b-4d6b422a8877-kube-api-access-t8ls4\") pod \"eb0b8932-7fb3-4867-919b-4d6b422a8877\" (UID: \"eb0b8932-7fb3-4867-919b-4d6b422a8877\") "
Nov 01 03:59:42.452622 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:42.452405    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb0b8932-7fb3-4867-919b-4d6b422a8877-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb0b8932-7fb3-4867-919b-4d6b422a8877-maas-api-openshift-ca\") pod \"eb0b8932-7fb3-4867-919b-4d6b422a8877\" (UID: \"eb0b8932-7fb3-4867-919b-4d6b422a8877\") "
Nov 01 03:59:42.452622 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:42.452443    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb0b8932-7fb3-4867-919b-4d6b422a8877-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb0b8932-7fb3-4867-919b-4d6b422a8877-maas-api-xks-ca\") pod \"eb0b8932-7fb3-4867-919b-4d6b422a8877\" (UID: \"eb0b8932-7fb3-4867-919b-4d6b422a8877\") "
Nov 01 03:59:42.453113 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:42.453083    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb0b8932-7fb3-4867-919b-4d6b422a8877-maas-api-openshift-ca" pod "eb0b8932-7fb3-4867-919b-4d6b422a8877" (UID: "eb0b8932-7fb3-4867-919b-4d6b422a8877"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 03:59:42.455163 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:42.455134    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb0b8932-7fb3-4867-919b-4d6b422a8877-kube-api-access-t8ls4" pod "eb0b8932-7fb3-4867-919b-4d6b422a8877" (UID: "eb0b8932-7fb3-4867-919b-4d6b422a8877"). InnerVolumeSpecName "kube-api-access-t8ls4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 03:59:42.455391 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:42.455361    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb0b8932-7fb3-4867-919b-4d6b422a8877-maas-api-xks-ca" pod "eb0b8932-7fb3-4867-919b-4d6b422a8877" (UID: "eb0b8932-7fb3-4867-919b-4d6b422a8877"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 03:59:42.494738 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:42.488942    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sgnfd"
Nov 01 03:59:42.494738 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:42.489024    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sgnfd" event={"ID":"eb0b8932-7fb3-4867-919b-4d6b422a8877","Type":"ContainerDied","Data":"65746f5ca16197f4ad2a6e3c8bbcd467e9b0c765c9825a91c0f3280da92e278f"}
Nov 01 03:59:42.494738 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:42.489083    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65746f5ca16197f4ad2a6e3c8bbcd467e9b0c765c9825a91c0f3280da92e278f"
Nov 01 03:59:42.553367 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:42.553291    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8ls4\" (UniqueName: \"kubernetes.io/projected/eb0b8932-7fb3-4867-919b-4d6b422a8877-kube-api-access-t8ls4\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:59:42.553367 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:42.553320    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb0b8932-7fb3-4867-919b-4d6b422a8877-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:59:42.553367 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:42.553331    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb0b8932-7fb3-4867-919b-4d6b422a8877-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 03:59:44.901908 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:44.901875    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 03:59:49.120781 ip-10-0-141-122 kubenswrapper[4930]: E1101 03:59:49.120696    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 03:59:53.050298 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:53.050260    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s28jx"]
Nov 01 03:59:53.100254 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:53.100225    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-s28jx"]
Nov 01 03:59:53.624791 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:53.624747    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1bce12e-2008-43a2-991a-0e309db6b463" path="/var/lib/kubelet/pods/b1bce12e-2008-43a2-991a-0e309db6b463/volumes"
Nov 01 03:59:54.690354 ip-10-0-141-122 kubenswrapper[4930]: I1101 03:59:54.690313    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:00:00.165239 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:00.165207    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891760-864wd"]
Nov 01 04:00:00.165750 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:00.165647    4930 cpu_manager.go:414] "containerMap: removing container" podUID="eb0b8932-7fb3-4867-919b-4d6b422a8877" containerName="revoke-keys"
Nov 01 04:00:00.165750 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:00.165661    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb0b8932-7fb3-4867-919b-4d6b422a8877" containerName="revoke-keys"
Nov 01 04:00:00.165750 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:00.165738    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb0b8932-7fb3-4867-919b-4d6b422a8877" containerName="revoke-keys"
Nov 01 04:00:00.170132 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:00.170109    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891760-864wd"
Nov 01 04:00:00.224414 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:00.224387    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 04:00:00.224666 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:00.224631    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 04:00:00.271817 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:00.271782    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpj5k\" (UniqueName: \"kubernetes.io/projected/f1607529-6ef0-4f47-bf3e-2dca9b592afe-kube-api-access-fpj5k\") pod \"maas-api-key-cleanup-29891760-864wd\" (UID: \"f1607529-6ef0-4f47-bf3e-2dca9b592afe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891760-864wd"
Nov 01 04:00:00.271972 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:00.271910    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1607529-6ef0-4f47-bf3e-2dca9b592afe-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891760-864wd\" (UID: \"f1607529-6ef0-4f47-bf3e-2dca9b592afe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891760-864wd"
Nov 01 04:00:00.274090 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:00.274070    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 04:00:00.276997 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:00.276979    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891760-864wd"]
Nov 01 04:00:00.372855 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:00.372824    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1607529-6ef0-4f47-bf3e-2dca9b592afe-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891760-864wd\" (UID: \"f1607529-6ef0-4f47-bf3e-2dca9b592afe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891760-864wd"
Nov 01 04:00:00.373061 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:00.372896    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpj5k\" (UniqueName: \"kubernetes.io/projected/f1607529-6ef0-4f47-bf3e-2dca9b592afe-kube-api-access-fpj5k\") pod \"maas-api-key-cleanup-29891760-864wd\" (UID: \"f1607529-6ef0-4f47-bf3e-2dca9b592afe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891760-864wd"
Nov 01 04:00:00.373541 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:00.373512    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1607529-6ef0-4f47-bf3e-2dca9b592afe-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891760-864wd\" (UID: \"f1607529-6ef0-4f47-bf3e-2dca9b592afe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891760-864wd"
Nov 01 04:00:00.433985 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:00.433918    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpj5k\" (UniqueName: \"kubernetes.io/projected/f1607529-6ef0-4f47-bf3e-2dca9b592afe-kube-api-access-fpj5k\") pod \"maas-api-key-cleanup-29891760-864wd\" (UID: \"f1607529-6ef0-4f47-bf3e-2dca9b592afe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891760-864wd"
Nov 01 04:00:00.482953 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:00.482924    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891760-864wd"
Nov 01 04:00:00.857475 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:00.857447    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891760-864wd"]
Nov 01 04:00:00.858794 ip-10-0-141-122 kubenswrapper[4930]: W1101 04:00:00.858768    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1607529_6ef0_4f47_bf3e_2dca9b592afe.slice/crio-59260fb04b4323de4f0890850ae22e34410204f985d7fed21f7461f3bd7e9ab7 WatchSource:0}: Error finding container 59260fb04b4323de4f0890850ae22e34410204f985d7fed21f7461f3bd7e9ab7: Status 404 returned error can't find the container with id 59260fb04b4323de4f0890850ae22e34410204f985d7fed21f7461f3bd7e9ab7
Nov 01 04:00:01.636022 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:01.635980    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891760-864wd" event={"ID":"f1607529-6ef0-4f47-bf3e-2dca9b592afe","Type":"ContainerStarted","Data":"37dbde70aa424caa7f7727099fd7a0f76212cf3fc6a7a8ca2cf3cfa2919792f4"}
Nov 01 04:00:01.636455 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:01.636041    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891760-864wd" event={"ID":"f1607529-6ef0-4f47-bf3e-2dca9b592afe","Type":"ContainerStarted","Data":"59260fb04b4323de4f0890850ae22e34410204f985d7fed21f7461f3bd7e9ab7"}
Nov 01 04:00:01.752030 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:01.751973    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891760-864wd" podStartSLOduration=1.751958806 podStartE2EDuration="1.751958806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:00:01.750656562 +0000 UTC m=+3206.273663150" watchObservedRunningTime="2026-11-01 04:00:01.751958806 +0000 UTC m=+3206.274965394"
Nov 01 04:00:02.644326 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:02.644291    4930 generic.go:356] "Generic (PLEG): container finished" podID="f1607529-6ef0-4f47-bf3e-2dca9b592afe" containerID="37dbde70aa424caa7f7727099fd7a0f76212cf3fc6a7a8ca2cf3cfa2919792f4" exitCode=60
Nov 01 04:00:02.644772 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:02.644343    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891760-864wd" event={"ID":"f1607529-6ef0-4f47-bf3e-2dca9b592afe","Type":"ContainerDied","Data":"37dbde70aa424caa7f7727099fd7a0f76212cf3fc6a7a8ca2cf3cfa2919792f4"}
Nov 01 04:00:02.644772 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:02.644587    4930 scope.go:122] "RemoveContainer" containerID="37dbde70aa424caa7f7727099fd7a0f76212cf3fc6a7a8ca2cf3cfa2919792f4"
Nov 01 04:00:03.652813 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:03.652765    4930 generic.go:356] "Generic (PLEG): container finished" podID="f1607529-6ef0-4f47-bf3e-2dca9b592afe" containerID="2ebcf83f6f76da423839199dfd264c96a8066405db6f4aff2334fd64e057fcbc" exitCode=60
Nov 01 04:00:03.653301 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:03.652842    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891760-864wd" event={"ID":"f1607529-6ef0-4f47-bf3e-2dca9b592afe","Type":"ContainerDied","Data":"2ebcf83f6f76da423839199dfd264c96a8066405db6f4aff2334fd64e057fcbc"}
Nov 01 04:00:03.653301 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:03.652908    4930 scope.go:122] "RemoveContainer" containerID="37dbde70aa424caa7f7727099fd7a0f76212cf3fc6a7a8ca2cf3cfa2919792f4"
Nov 01 04:00:03.653301 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:03.653108    4930 scope.go:122] "RemoveContainer" containerID="2ebcf83f6f76da423839199dfd264c96a8066405db6f4aff2334fd64e057fcbc"
Nov 01 04:00:03.653485 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:00:03.653359    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891760-864wd_odh-ai-gateway-infra(f1607529-6ef0-4f47-bf3e-2dca9b592afe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891760-864wd" podUID="f1607529-6ef0-4f47-bf3e-2dca9b592afe"
Nov 01 04:00:04.660646 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:04.660614    4930 scope.go:122] "RemoveContainer" containerID="2ebcf83f6f76da423839199dfd264c96a8066405db6f4aff2334fd64e057fcbc"
Nov 01 04:00:04.661107 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:00:04.660874    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891760-864wd_odh-ai-gateway-infra(f1607529-6ef0-4f47-bf3e-2dca9b592afe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891760-864wd" podUID="f1607529-6ef0-4f47-bf3e-2dca9b592afe"
Nov 01 04:00:12.215408 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:12.215372    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sgnfd"]
Nov 01 04:00:12.263953 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:12.263917    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sgnfd"]
Nov 01 04:00:13.619965 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:13.619934    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb0b8932-7fb3-4867-919b-4d6b422a8877" path="/var/lib/kubelet/pods/eb0b8932-7fb3-4867-919b-4d6b422a8877/volumes"
Nov 01 04:00:18.612885 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:18.612855    4930 scope.go:122] "RemoveContainer" containerID="2ebcf83f6f76da423839199dfd264c96a8066405db6f4aff2334fd64e057fcbc"
Nov 01 04:00:19.120008 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:00:19.119932    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 04:00:19.748042 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:19.747965    4930 generic.go:356] "Generic (PLEG): container finished" podID="f1607529-6ef0-4f47-bf3e-2dca9b592afe" containerID="005ce6e5d5d53ff9ca47dd2445a7fca1e92c08829a1c034b038a4e019567a9a8" exitCode=60
Nov 01 04:00:19.748449 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:19.748037    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891760-864wd" event={"ID":"f1607529-6ef0-4f47-bf3e-2dca9b592afe","Type":"ContainerDied","Data":"005ce6e5d5d53ff9ca47dd2445a7fca1e92c08829a1c034b038a4e019567a9a8"}
Nov 01 04:00:19.748449 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:19.748090    4930 scope.go:122] "RemoveContainer" containerID="2ebcf83f6f76da423839199dfd264c96a8066405db6f4aff2334fd64e057fcbc"
Nov 01 04:00:19.748449 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:19.748418    4930 scope.go:122] "RemoveContainer" containerID="005ce6e5d5d53ff9ca47dd2445a7fca1e92c08829a1c034b038a4e019567a9a8"
Nov 01 04:00:19.748740 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:00:19.748714    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891760-864wd_odh-ai-gateway-infra(f1607529-6ef0-4f47-bf3e-2dca9b592afe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891760-864wd" podUID="f1607529-6ef0-4f47-bf3e-2dca9b592afe"
Nov 01 04:00:20.888871 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:20.888834    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891760-864wd"]
Nov 01 04:00:21.040381 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:21.040354    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891760-864wd"
Nov 01 04:00:21.075693 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:21.075666    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1607529-6ef0-4f47-bf3e-2dca9b592afe-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1607529-6ef0-4f47-bf3e-2dca9b592afe-maas-api-service-ca\") pod \"f1607529-6ef0-4f47-bf3e-2dca9b592afe\" (UID: \"f1607529-6ef0-4f47-bf3e-2dca9b592afe\") "
Nov 01 04:00:21.075814 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:21.075763    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1607529-6ef0-4f47-bf3e-2dca9b592afe-kube-api-access-fpj5k\" (UniqueName: \"kubernetes.io/projected/f1607529-6ef0-4f47-bf3e-2dca9b592afe-kube-api-access-fpj5k\") pod \"f1607529-6ef0-4f47-bf3e-2dca9b592afe\" (UID: \"f1607529-6ef0-4f47-bf3e-2dca9b592afe\") "
Nov 01 04:00:21.075996 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:21.075975    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1607529-6ef0-4f47-bf3e-2dca9b592afe-maas-api-service-ca" pod "f1607529-6ef0-4f47-bf3e-2dca9b592afe" (UID: "f1607529-6ef0-4f47-bf3e-2dca9b592afe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:00:21.077758 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:21.077735    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1607529-6ef0-4f47-bf3e-2dca9b592afe-kube-api-access-fpj5k" pod "f1607529-6ef0-4f47-bf3e-2dca9b592afe" (UID: "f1607529-6ef0-4f47-bf3e-2dca9b592afe"). InnerVolumeSpecName "kube-api-access-fpj5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:00:21.183639 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:21.177412    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpj5k\" (UniqueName: \"kubernetes.io/projected/f1607529-6ef0-4f47-bf3e-2dca9b592afe-kube-api-access-fpj5k\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:00:21.183639 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:21.177454    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1607529-6ef0-4f47-bf3e-2dca9b592afe-maas-api-service-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:00:21.762730 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:21.762701    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891760-864wd"
Nov 01 04:00:21.762907 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:21.762744    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891760-864wd" event={"ID":"f1607529-6ef0-4f47-bf3e-2dca9b592afe","Type":"ContainerDied","Data":"59260fb04b4323de4f0890850ae22e34410204f985d7fed21f7461f3bd7e9ab7"}
Nov 01 04:00:21.762907 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:21.762807    4930 scope.go:122] "RemoveContainer" containerID="005ce6e5d5d53ff9ca47dd2445a7fca1e92c08829a1c034b038a4e019567a9a8"
Nov 01 04:00:21.929746 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:21.929662    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891760-864wd"]
Nov 01 04:00:21.936090 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:21.936058    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891760-864wd"]
Nov 01 04:00:23.619663 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:23.619626    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1607529-6ef0-4f47-bf3e-2dca9b592afe" path="/var/lib/kubelet/pods/f1607529-6ef0-4f47-bf3e-2dca9b592afe/volumes"
Nov 01 04:00:40.852452 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:40.852422    4930 scope.go:122] "RemoveContainer" containerID="09e1ad8f98dfc55c8010d74ebb55875c3b6a3d4019271ffa7554fdb711ee3dc0"
Nov 01 04:00:47.095668 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:47.095627    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:00:49.121421 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:00:49.121383    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 04:00:55.025101 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:55.025059    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:00:59.693342 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:59.693302    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nlssn"]
Nov 01 04:00:59.693935 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:59.693912    4930 cpu_manager.go:414] "containerMap: removing container" podUID="f1607529-6ef0-4f47-bf3e-2dca9b592afe" containerName="cleanup"
Nov 01 04:00:59.694031 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:59.693937    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1607529-6ef0-4f47-bf3e-2dca9b592afe" containerName="cleanup"
Nov 01 04:00:59.694031 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:59.693950    4930 cpu_manager.go:414] "containerMap: removing container" podUID="f1607529-6ef0-4f47-bf3e-2dca9b592afe" containerName="cleanup"
Nov 01 04:00:59.694031 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:59.693957    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1607529-6ef0-4f47-bf3e-2dca9b592afe" containerName="cleanup"
Nov 01 04:00:59.694202 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:59.694117    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1607529-6ef0-4f47-bf3e-2dca9b592afe" containerName="cleanup"
Nov 01 04:00:59.694202 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:59.694135    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1607529-6ef0-4f47-bf3e-2dca9b592afe" containerName="cleanup"
Nov 01 04:00:59.694202 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:59.694144    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1607529-6ef0-4f47-bf3e-2dca9b592afe" containerName="cleanup"
Nov 01 04:00:59.698933 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:59.698909    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nlssn"
Nov 01 04:00:59.708911 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:59.708888    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/702790ea-5079-4431-ba0c-674bf255859a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-nlssn\" (UID: \"702790ea-5079-4431-ba0c-674bf255859a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nlssn"
Nov 01 04:00:59.709025 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:59.708926    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/702790ea-5079-4431-ba0c-674bf255859a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-nlssn\" (UID: \"702790ea-5079-4431-ba0c-674bf255859a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nlssn"
Nov 01 04:00:59.709084 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:59.709051    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9fmf\" (UniqueName: \"kubernetes.io/projected/702790ea-5079-4431-ba0c-674bf255859a-kube-api-access-j9fmf\") pod \"maas-api-revoke-sub-e2fc9c5b-nlssn\" (UID: \"702790ea-5079-4431-ba0c-674bf255859a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nlssn"
Nov 01 04:00:59.802469 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:59.802427    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:00:59.802469 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:59.802446    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 04:00:59.802716 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:59.802429    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 04:00:59.802716 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:59.802515    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 04:00:59.805644 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:59.805624    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nlssn"]
Nov 01 04:00:59.810023 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:59.809998    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/702790ea-5079-4431-ba0c-674bf255859a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-nlssn\" (UID: \"702790ea-5079-4431-ba0c-674bf255859a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nlssn"
Nov 01 04:00:59.810148 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:59.810033    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/702790ea-5079-4431-ba0c-674bf255859a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-nlssn\" (UID: \"702790ea-5079-4431-ba0c-674bf255859a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nlssn"
Nov 01 04:00:59.810148 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:59.810070    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9fmf\" (UniqueName: \"kubernetes.io/projected/702790ea-5079-4431-ba0c-674bf255859a-kube-api-access-j9fmf\") pod \"maas-api-revoke-sub-e2fc9c5b-nlssn\" (UID: \"702790ea-5079-4431-ba0c-674bf255859a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nlssn"
Nov 01 04:00:59.810664 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:59.810640    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/702790ea-5079-4431-ba0c-674bf255859a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-nlssn\" (UID: \"702790ea-5079-4431-ba0c-674bf255859a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nlssn"
Nov 01 04:00:59.812347 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:59.812325    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/702790ea-5079-4431-ba0c-674bf255859a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-nlssn\" (UID: \"702790ea-5079-4431-ba0c-674bf255859a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nlssn"
Nov 01 04:00:59.871210 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:00:59.871190    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9fmf\" (UniqueName: \"kubernetes.io/projected/702790ea-5079-4431-ba0c-674bf255859a-kube-api-access-j9fmf\") pod \"maas-api-revoke-sub-e2fc9c5b-nlssn\" (UID: \"702790ea-5079-4431-ba0c-674bf255859a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nlssn"
Nov 01 04:01:00.012481 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:00.012403    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nlssn"
Nov 01 04:01:00.385536 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:00.385511    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nlssn"]
Nov 01 04:01:00.386462 ip-10-0-141-122 kubenswrapper[4930]: W1101 04:01:00.386433    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod702790ea_5079_4431_ba0c_674bf255859a.slice/crio-105cf1d354ef466e23d5741860ca37276b5525e4d2d707f3b7ce2be400f27dc0 WatchSource:0}: Error finding container 105cf1d354ef466e23d5741860ca37276b5525e4d2d707f3b7ce2be400f27dc0: Status 404 returned error can't find the container with id 105cf1d354ef466e23d5741860ca37276b5525e4d2d707f3b7ce2be400f27dc0
Nov 01 04:01:01.016346 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:01.016253    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nlssn" event={"ID":"702790ea-5079-4431-ba0c-674bf255859a","Type":"ContainerStarted","Data":"bdb37430a7a465b7c3e70658b908c107ff1d75e864260633e066ff347ca67909"}
Nov 01 04:01:01.016346 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:01.016312    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nlssn" event={"ID":"702790ea-5079-4431-ba0c-674bf255859a","Type":"ContainerStarted","Data":"105cf1d354ef466e23d5741860ca37276b5525e4d2d707f3b7ce2be400f27dc0"}
Nov 01 04:01:01.047649 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:01.047618    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-h6st4"]
Nov 01 04:01:01.097204 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:01.097174    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-h6st4"]
Nov 01 04:01:01.158294 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:01.158229    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nlssn" podStartSLOduration=2.158210051 podStartE2EDuration="2.158210051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:00:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:01:01.156863131 +0000 UTC m=+3265.679869720" watchObservedRunningTime="2026-11-01 04:01:01.158210051 +0000 UTC m=+3265.681216640"
Nov 01 04:01:01.619937 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:01.619895    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ed664b9-08b4-4ef5-81dc-50a40cf43c72" path="/var/lib/kubelet/pods/6ed664b9-08b4-4ef5-81dc-50a40cf43c72/volumes"
Nov 01 04:01:02.023681 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:02.023593    4930 generic.go:356] "Generic (PLEG): container finished" podID="702790ea-5079-4431-ba0c-674bf255859a" containerID="bdb37430a7a465b7c3e70658b908c107ff1d75e864260633e066ff347ca67909" exitCode=0
Nov 01 04:01:02.023681 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:02.023632    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nlssn" event={"ID":"702790ea-5079-4431-ba0c-674bf255859a","Type":"ContainerDied","Data":"bdb37430a7a465b7c3e70658b908c107ff1d75e864260633e066ff347ca67909"}
Nov 01 04:01:03.155404 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:03.155372    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nlssn"
Nov 01 04:01:03.249193 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:03.249162    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/702790ea-5079-4431-ba0c-674bf255859a-kube-api-access-j9fmf\" (UniqueName: \"kubernetes.io/projected/702790ea-5079-4431-ba0c-674bf255859a-kube-api-access-j9fmf\") pod \"702790ea-5079-4431-ba0c-674bf255859a\" (UID: \"702790ea-5079-4431-ba0c-674bf255859a\") "
Nov 01 04:01:03.249362 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:03.249259    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/702790ea-5079-4431-ba0c-674bf255859a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/702790ea-5079-4431-ba0c-674bf255859a-maas-api-xks-ca\") pod \"702790ea-5079-4431-ba0c-674bf255859a\" (UID: \"702790ea-5079-4431-ba0c-674bf255859a\") "
Nov 01 04:01:03.249362 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:03.249296    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/702790ea-5079-4431-ba0c-674bf255859a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/702790ea-5079-4431-ba0c-674bf255859a-maas-api-openshift-ca\") pod \"702790ea-5079-4431-ba0c-674bf255859a\" (UID: \"702790ea-5079-4431-ba0c-674bf255859a\") "
Nov 01 04:01:03.249739 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:03.249711    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/702790ea-5079-4431-ba0c-674bf255859a-maas-api-openshift-ca" pod "702790ea-5079-4431-ba0c-674bf255859a" (UID: "702790ea-5079-4431-ba0c-674bf255859a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:01:03.251355 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:03.251332    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/702790ea-5079-4431-ba0c-674bf255859a-maas-api-xks-ca" pod "702790ea-5079-4431-ba0c-674bf255859a" (UID: "702790ea-5079-4431-ba0c-674bf255859a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:01:03.251502 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:03.251481    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/702790ea-5079-4431-ba0c-674bf255859a-kube-api-access-j9fmf" pod "702790ea-5079-4431-ba0c-674bf255859a" (UID: "702790ea-5079-4431-ba0c-674bf255859a"). InnerVolumeSpecName "kube-api-access-j9fmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:01:03.350311 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:03.350266    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/702790ea-5079-4431-ba0c-674bf255859a-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:01:03.350311 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:03.350311    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/702790ea-5079-4431-ba0c-674bf255859a-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:01:03.350311 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:03.350322    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9fmf\" (UniqueName: \"kubernetes.io/projected/702790ea-5079-4431-ba0c-674bf255859a-kube-api-access-j9fmf\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:01:04.037278 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:04.037199    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nlssn"
Nov 01 04:01:04.037278 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:04.037235    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nlssn" event={"ID":"702790ea-5079-4431-ba0c-674bf255859a","Type":"ContainerDied","Data":"105cf1d354ef466e23d5741860ca37276b5525e4d2d707f3b7ce2be400f27dc0"}
Nov 01 04:01:04.037467 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:04.037286    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="105cf1d354ef466e23d5741860ca37276b5525e4d2d707f3b7ce2be400f27dc0"
Nov 01 04:01:10.529458 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:10.529420    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dw724"]
Nov 01 04:01:10.529977 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:10.529865    4930 cpu_manager.go:414] "containerMap: removing container" podUID="702790ea-5079-4431-ba0c-674bf255859a" containerName="revoke-keys"
Nov 01 04:01:10.529977 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:10.529879    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="702790ea-5079-4431-ba0c-674bf255859a" containerName="revoke-keys"
Nov 01 04:01:10.529977 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:10.529909    4930 cpu_manager.go:414] "containerMap: removing container" podUID="f1607529-6ef0-4f47-bf3e-2dca9b592afe" containerName="cleanup"
Nov 01 04:01:10.529977 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:10.529914    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1607529-6ef0-4f47-bf3e-2dca9b592afe" containerName="cleanup"
Nov 01 04:01:10.530201 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:10.529986    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="702790ea-5079-4431-ba0c-674bf255859a" containerName="revoke-keys"
Nov 01 04:01:10.533271 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:10.533245    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dw724"
Nov 01 04:01:10.585334 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:10.585303    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:01:10.621162 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:10.621129    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrcr5\" (UniqueName: \"kubernetes.io/projected/47773233-705b-48a6-b893-315fe79f75df-kube-api-access-rrcr5\") pod \"maas-api-revoke-sub-5a29efb3-dw724\" (UID: \"47773233-705b-48a6-b893-315fe79f75df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dw724"
Nov 01 04:01:10.621309 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:10.621270    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47773233-705b-48a6-b893-315fe79f75df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-dw724\" (UID: \"47773233-705b-48a6-b893-315fe79f75df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dw724"
Nov 01 04:01:10.621358 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:10.621319    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47773233-705b-48a6-b893-315fe79f75df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-dw724\" (UID: \"47773233-705b-48a6-b893-315fe79f75df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dw724"
Nov 01 04:01:10.634974 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:10.634946    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 04:01:10.635105 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:10.635034    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 04:01:10.635105 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:10.635067    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 04:01:10.638006 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:10.637983    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dw724"]
Nov 01 04:01:10.722600 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:10.722545    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47773233-705b-48a6-b893-315fe79f75df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-dw724\" (UID: \"47773233-705b-48a6-b893-315fe79f75df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dw724"
Nov 01 04:01:10.722750 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:10.722622    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47773233-705b-48a6-b893-315fe79f75df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-dw724\" (UID: \"47773233-705b-48a6-b893-315fe79f75df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dw724"
Nov 01 04:01:10.722854 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:10.722828    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrcr5\" (UniqueName: \"kubernetes.io/projected/47773233-705b-48a6-b893-315fe79f75df-kube-api-access-rrcr5\") pod \"maas-api-revoke-sub-5a29efb3-dw724\" (UID: \"47773233-705b-48a6-b893-315fe79f75df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dw724"
Nov 01 04:01:10.723326 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:10.723309    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47773233-705b-48a6-b893-315fe79f75df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-dw724\" (UID: \"47773233-705b-48a6-b893-315fe79f75df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dw724"
Nov 01 04:01:10.725000 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:10.724979    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47773233-705b-48a6-b893-315fe79f75df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-dw724\" (UID: \"47773233-705b-48a6-b893-315fe79f75df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dw724"
Nov 01 04:01:10.783524 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:10.783462    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrcr5\" (UniqueName: \"kubernetes.io/projected/47773233-705b-48a6-b893-315fe79f75df-kube-api-access-rrcr5\") pod \"maas-api-revoke-sub-5a29efb3-dw724\" (UID: \"47773233-705b-48a6-b893-315fe79f75df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dw724"
Nov 01 04:01:10.847438 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:10.847413    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dw724"
Nov 01 04:01:11.222237 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:11.222207    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dw724"]
Nov 01 04:01:11.223057 ip-10-0-141-122 kubenswrapper[4930]: W1101 04:01:11.223035    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47773233_705b_48a6_b893_315fe79f75df.slice/crio-8fcfa7e21f288f0f69a54777bcd2b39b2d70400e08db24b6d25252f0e4e1fdcc WatchSource:0}: Error finding container 8fcfa7e21f288f0f69a54777bcd2b39b2d70400e08db24b6d25252f0e4e1fdcc: Status 404 returned error can't find the container with id 8fcfa7e21f288f0f69a54777bcd2b39b2d70400e08db24b6d25252f0e4e1fdcc
Nov 01 04:01:12.084383 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:12.084355    4930 generic.go:356] "Generic (PLEG): container finished" podID="47773233-705b-48a6-b893-315fe79f75df" containerID="8c9a7071e67dc3715e51f78581a4716f1de5ffe41cd8a4facfc6c370bc746cda" exitCode=0
Nov 01 04:01:12.084787 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:12.084444    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dw724" event={"ID":"47773233-705b-48a6-b893-315fe79f75df","Type":"ContainerDied","Data":"8c9a7071e67dc3715e51f78581a4716f1de5ffe41cd8a4facfc6c370bc746cda"}
Nov 01 04:01:12.084787 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:12.084509    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dw724" event={"ID":"47773233-705b-48a6-b893-315fe79f75df","Type":"ContainerStarted","Data":"8fcfa7e21f288f0f69a54777bcd2b39b2d70400e08db24b6d25252f0e4e1fdcc"}
Nov 01 04:01:13.293497 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:13.293463    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dw724"
Nov 01 04:01:13.356401 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:13.356322    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47773233-705b-48a6-b893-315fe79f75df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47773233-705b-48a6-b893-315fe79f75df-maas-api-openshift-ca\") pod \"47773233-705b-48a6-b893-315fe79f75df\" (UID: \"47773233-705b-48a6-b893-315fe79f75df\") "
Nov 01 04:01:13.356401 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:13.356388    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47773233-705b-48a6-b893-315fe79f75df-kube-api-access-rrcr5\" (UniqueName: \"kubernetes.io/projected/47773233-705b-48a6-b893-315fe79f75df-kube-api-access-rrcr5\") pod \"47773233-705b-48a6-b893-315fe79f75df\" (UID: \"47773233-705b-48a6-b893-315fe79f75df\") "
Nov 01 04:01:13.356691 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:13.356431    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47773233-705b-48a6-b893-315fe79f75df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47773233-705b-48a6-b893-315fe79f75df-maas-api-xks-ca\") pod \"47773233-705b-48a6-b893-315fe79f75df\" (UID: \"47773233-705b-48a6-b893-315fe79f75df\") "
Nov 01 04:01:13.357228 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:13.357202    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47773233-705b-48a6-b893-315fe79f75df-maas-api-openshift-ca" pod "47773233-705b-48a6-b893-315fe79f75df" (UID: "47773233-705b-48a6-b893-315fe79f75df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:01:13.359352 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:13.359312    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47773233-705b-48a6-b893-315fe79f75df-kube-api-access-rrcr5" pod "47773233-705b-48a6-b893-315fe79f75df" (UID: "47773233-705b-48a6-b893-315fe79f75df"). InnerVolumeSpecName "kube-api-access-rrcr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:01:13.359466 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:13.359372    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47773233-705b-48a6-b893-315fe79f75df-maas-api-xks-ca" pod "47773233-705b-48a6-b893-315fe79f75df" (UID: "47773233-705b-48a6-b893-315fe79f75df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:01:13.458022 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:13.457918    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrcr5\" (UniqueName: \"kubernetes.io/projected/47773233-705b-48a6-b893-315fe79f75df-kube-api-access-rrcr5\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:01:13.458022 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:13.457954    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47773233-705b-48a6-b893-315fe79f75df-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:01:13.458022 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:13.457970    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47773233-705b-48a6-b893-315fe79f75df-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:01:14.158242 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:14.158195    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dw724" event={"ID":"47773233-705b-48a6-b893-315fe79f75df","Type":"ContainerDied","Data":"8fcfa7e21f288f0f69a54777bcd2b39b2d70400e08db24b6d25252f0e4e1fdcc"}
Nov 01 04:01:14.162607 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:14.158543    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fcfa7e21f288f0f69a54777bcd2b39b2d70400e08db24b6d25252f0e4e1fdcc"
Nov 01 04:01:14.162897 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:14.162880    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dw724"
Nov 01 04:01:19.120639 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:01:19.120542    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 04:01:27.450902 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:27.450862    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ll4dv"]
Nov 01 04:01:27.451628 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:27.451604    4930 cpu_manager.go:414] "containerMap: removing container" podUID="47773233-705b-48a6-b893-315fe79f75df" containerName="revoke-keys"
Nov 01 04:01:27.451689 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:27.451631    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47773233-705b-48a6-b893-315fe79f75df" containerName="revoke-keys"
Nov 01 04:01:27.451796 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:27.451783    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="47773233-705b-48a6-b893-315fe79f75df" containerName="revoke-keys"
Nov 01 04:01:27.457276 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:27.456139    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ll4dv"
Nov 01 04:01:27.509267 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:27.509238    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 04:01:27.558784 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:27.558751    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:01:27.558784 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:27.558761    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 04:01:27.559030 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:27.558783    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 04:01:27.559903 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:27.559873    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417b98ad-5bb3-4be6-b51b-25569c5e4f90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ll4dv\" (UID: \"417b98ad-5bb3-4be6-b51b-25569c5e4f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ll4dv"
Nov 01 04:01:27.560041 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:27.559911    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417b98ad-5bb3-4be6-b51b-25569c5e4f90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ll4dv\" (UID: \"417b98ad-5bb3-4be6-b51b-25569c5e4f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ll4dv"
Nov 01 04:01:27.560104 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:27.560052    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twdld\" (UniqueName: \"kubernetes.io/projected/417b98ad-5bb3-4be6-b51b-25569c5e4f90-kube-api-access-twdld\") pod \"maas-api-revoke-sub-3d4156f1-ll4dv\" (UID: \"417b98ad-5bb3-4be6-b51b-25569c5e4f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ll4dv"
Nov 01 04:01:27.561885 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:27.561857    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ll4dv"]
Nov 01 04:01:27.661646 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:27.661607    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417b98ad-5bb3-4be6-b51b-25569c5e4f90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ll4dv\" (UID: \"417b98ad-5bb3-4be6-b51b-25569c5e4f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ll4dv"
Nov 01 04:01:27.661824 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:27.661656    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417b98ad-5bb3-4be6-b51b-25569c5e4f90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ll4dv\" (UID: \"417b98ad-5bb3-4be6-b51b-25569c5e4f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ll4dv"
Nov 01 04:01:27.661824 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:27.661716    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twdld\" (UniqueName: \"kubernetes.io/projected/417b98ad-5bb3-4be6-b51b-25569c5e4f90-kube-api-access-twdld\") pod \"maas-api-revoke-sub-3d4156f1-ll4dv\" (UID: \"417b98ad-5bb3-4be6-b51b-25569c5e4f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ll4dv"
Nov 01 04:01:27.662385 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:27.662361    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417b98ad-5bb3-4be6-b51b-25569c5e4f90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ll4dv\" (UID: \"417b98ad-5bb3-4be6-b51b-25569c5e4f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ll4dv"
Nov 01 04:01:27.664272 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:27.664238    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417b98ad-5bb3-4be6-b51b-25569c5e4f90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ll4dv\" (UID: \"417b98ad-5bb3-4be6-b51b-25569c5e4f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ll4dv"
Nov 01 04:01:27.723104 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:27.723038    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twdld\" (UniqueName: \"kubernetes.io/projected/417b98ad-5bb3-4be6-b51b-25569c5e4f90-kube-api-access-twdld\") pod \"maas-api-revoke-sub-3d4156f1-ll4dv\" (UID: \"417b98ad-5bb3-4be6-b51b-25569c5e4f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ll4dv"
Nov 01 04:01:27.772537 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:27.772507    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ll4dv"
Nov 01 04:01:28.150698 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:28.150668    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ll4dv"]
Nov 01 04:01:28.152427 ip-10-0-141-122 kubenswrapper[4930]: W1101 04:01:28.152393    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod417b98ad_5bb3_4be6_b51b_25569c5e4f90.slice/crio-c528da87e263061c401b2a62c475ea14eb3ee7b72de8cfff1e16fef1b4dd0e97 WatchSource:0}: Error finding container c528da87e263061c401b2a62c475ea14eb3ee7b72de8cfff1e16fef1b4dd0e97: Status 404 returned error can't find the container with id c528da87e263061c401b2a62c475ea14eb3ee7b72de8cfff1e16fef1b4dd0e97
Nov 01 04:01:28.255314 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:28.255282    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ll4dv" event={"ID":"417b98ad-5bb3-4be6-b51b-25569c5e4f90","Type":"ContainerStarted","Data":"c528da87e263061c401b2a62c475ea14eb3ee7b72de8cfff1e16fef1b4dd0e97"}
Nov 01 04:01:29.263003 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:29.262904    4930 generic.go:356] "Generic (PLEG): container finished" podID="417b98ad-5bb3-4be6-b51b-25569c5e4f90" containerID="f6a22f209212e5e8613fe5bf34ebe67cf0c179162ab0974877597877c0b92bde" exitCode=0
Nov 01 04:01:29.263003 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:29.262945    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ll4dv" event={"ID":"417b98ad-5bb3-4be6-b51b-25569c5e4f90","Type":"ContainerDied","Data":"f6a22f209212e5e8613fe5bf34ebe67cf0c179162ab0974877597877c0b92bde"}
Nov 01 04:01:30.404229 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:30.404207    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ll4dv"
Nov 01 04:01:30.488503 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:30.488475    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/417b98ad-5bb3-4be6-b51b-25569c5e4f90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417b98ad-5bb3-4be6-b51b-25569c5e4f90-maas-api-openshift-ca\") pod \"417b98ad-5bb3-4be6-b51b-25569c5e4f90\" (UID: \"417b98ad-5bb3-4be6-b51b-25569c5e4f90\") "
Nov 01 04:01:30.488656 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:30.488537    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/417b98ad-5bb3-4be6-b51b-25569c5e4f90-kube-api-access-twdld\" (UniqueName: \"kubernetes.io/projected/417b98ad-5bb3-4be6-b51b-25569c5e4f90-kube-api-access-twdld\") pod \"417b98ad-5bb3-4be6-b51b-25569c5e4f90\" (UID: \"417b98ad-5bb3-4be6-b51b-25569c5e4f90\") "
Nov 01 04:01:30.488656 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:30.488619    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/417b98ad-5bb3-4be6-b51b-25569c5e4f90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417b98ad-5bb3-4be6-b51b-25569c5e4f90-maas-api-xks-ca\") pod \"417b98ad-5bb3-4be6-b51b-25569c5e4f90\" (UID: \"417b98ad-5bb3-4be6-b51b-25569c5e4f90\") "
Nov 01 04:01:30.488897 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:30.488873    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/417b98ad-5bb3-4be6-b51b-25569c5e4f90-maas-api-openshift-ca" pod "417b98ad-5bb3-4be6-b51b-25569c5e4f90" (UID: "417b98ad-5bb3-4be6-b51b-25569c5e4f90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:01:30.490639 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:30.490616    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/417b98ad-5bb3-4be6-b51b-25569c5e4f90-kube-api-access-twdld" pod "417b98ad-5bb3-4be6-b51b-25569c5e4f90" (UID: "417b98ad-5bb3-4be6-b51b-25569c5e4f90"). InnerVolumeSpecName "kube-api-access-twdld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:01:30.490709 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:30.490656    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/417b98ad-5bb3-4be6-b51b-25569c5e4f90-maas-api-xks-ca" pod "417b98ad-5bb3-4be6-b51b-25569c5e4f90" (UID: "417b98ad-5bb3-4be6-b51b-25569c5e4f90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:01:30.589968 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:30.589943    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417b98ad-5bb3-4be6-b51b-25569c5e4f90-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:01:30.589968 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:30.589968    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417b98ad-5bb3-4be6-b51b-25569c5e4f90-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:01:30.590165 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:30.589979    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twdld\" (UniqueName: \"kubernetes.io/projected/417b98ad-5bb3-4be6-b51b-25569c5e4f90-kube-api-access-twdld\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:01:31.276827 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:31.276794    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ll4dv"
Nov 01 04:01:31.277048 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:31.276830    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ll4dv" event={"ID":"417b98ad-5bb3-4be6-b51b-25569c5e4f90","Type":"ContainerDied","Data":"c528da87e263061c401b2a62c475ea14eb3ee7b72de8cfff1e16fef1b4dd0e97"}
Nov 01 04:01:31.277048 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:31.276886    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c528da87e263061c401b2a62c475ea14eb3ee7b72de8cfff1e16fef1b4dd0e97"
Nov 01 04:01:32.048468 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:32.048429    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vmn7p"]
Nov 01 04:01:32.097629 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:32.097598    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vmn7p"]
Nov 01 04:01:33.619353 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:33.619321    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d115bc2-40e8-4381-afd8-465e2f1d0ef8" path="/var/lib/kubelet/pods/0d115bc2-40e8-4381-afd8-465e2f1d0ef8/volumes"
Nov 01 04:01:38.384013 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:38.383983    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jh955_f29ea61e-2a78-4c11-9566-f52497e135ff/console-operator/2.log"
Nov 01 04:01:38.431742 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:38.431716    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jh955_f29ea61e-2a78-4c11-9566-f52497e135ff/console-operator/2.log"
Nov 01 04:01:40.856041 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:40.856002    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:01:40.886957 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:40.886927    4930 scope.go:122] "RemoveContainer" containerID="12b474dfb493dc3cb7bfab4e5d7bfc9167a4d58c3cbbf7d2a3aa646bab96a9e7"
Nov 01 04:01:44.064814 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:44.064777    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d8bt5"]
Nov 01 04:01:44.121596 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:44.121541    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-d8bt5"]
Nov 01 04:01:45.525311 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:45.525275    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:01:45.619833 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:45.619795    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d00b816-699e-4103-9a94-7b37b8d0d5e4" path="/var/lib/kubelet/pods/7d00b816-699e-4103-9a94-7b37b8d0d5e4/volumes"
Nov 01 04:01:49.121359 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:01:49.121321    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 04:01:55.428173 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:55.428141    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:01:59.253173 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:59.253136    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:01:59.865383 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:59.865350    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"]
Nov 01 04:01:59.865817 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:59.865797    4930 cpu_manager.go:414] "containerMap: removing container" podUID="417b98ad-5bb3-4be6-b51b-25569c5e4f90" containerName="revoke-keys"
Nov 01 04:01:59.865817 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:59.865816    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="417b98ad-5bb3-4be6-b51b-25569c5e4f90" containerName="revoke-keys"
Nov 01 04:01:59.866038 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:59.865905    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="417b98ad-5bb3-4be6-b51b-25569c5e4f90" containerName="revoke-keys"
Nov 01 04:01:59.870675 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:59.870655    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"
Nov 01 04:01:59.925844 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:59.925807    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-feacb8-openshift-default-dockercfg-fqqsj\""
Nov 01 04:01:59.940760 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:59.940724    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a0ed6d-e285-4062-9baf-fd36504d1f43-istiod-ca-cert\") pod \"e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s\" (UID: \"01a0ed6d-e285-4062-9baf-fd36504d1f43\") " pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"
Nov 01 04:01:59.940760 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:59.940766    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a0ed6d-e285-4062-9baf-fd36504d1f43-workload-socket\") pod \"e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s\" (UID: \"01a0ed6d-e285-4062-9baf-fd36504d1f43\") " pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"
Nov 01 04:01:59.941048 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:59.940799    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a0ed6d-e285-4062-9baf-fd36504d1f43-istio-envoy\") pod \"e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s\" (UID: \"01a0ed6d-e285-4062-9baf-fd36504d1f43\") " pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"
Nov 01 04:01:59.941048 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:59.940816    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a0ed6d-e285-4062-9baf-fd36504d1f43-workload-certs\") pod \"e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s\" (UID: \"01a0ed6d-e285-4062-9baf-fd36504d1f43\") " pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"
Nov 01 04:01:59.941048 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:59.940838    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a0ed6d-e285-4062-9baf-fd36504d1f43-istio-podinfo\") pod \"e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s\" (UID: \"01a0ed6d-e285-4062-9baf-fd36504d1f43\") " pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"
Nov 01 04:01:59.941048 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:59.940878    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a0ed6d-e285-4062-9baf-fd36504d1f43-credential-socket\") pod \"e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s\" (UID: \"01a0ed6d-e285-4062-9baf-fd36504d1f43\") " pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"
Nov 01 04:01:59.941048 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:59.940899    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a0ed6d-e285-4062-9baf-fd36504d1f43-istio-data\") pod \"e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s\" (UID: \"01a0ed6d-e285-4062-9baf-fd36504d1f43\") " pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"
Nov 01 04:01:59.941048 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:59.940934    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01a0ed6d-e285-4062-9baf-fd36504d1f43-istio-token\") pod \"e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s\" (UID: \"01a0ed6d-e285-4062-9baf-fd36504d1f43\") " pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"
Nov 01 04:01:59.941048 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:59.940975    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xktm8\" (UniqueName: \"kubernetes.io/projected/01a0ed6d-e285-4062-9baf-fd36504d1f43-kube-api-access-xktm8\") pod \"e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s\" (UID: \"01a0ed6d-e285-4062-9baf-fd36504d1f43\") " pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"
Nov 01 04:01:59.957358 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:59.957318    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:01:59.979031 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:01:59.979002    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"]
Nov 01 04:02:00.042547 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:00.042504    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a0ed6d-e285-4062-9baf-fd36504d1f43-credential-socket\") pod \"e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s\" (UID: \"01a0ed6d-e285-4062-9baf-fd36504d1f43\") " pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"
Nov 01 04:02:00.042765 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:00.042575    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a0ed6d-e285-4062-9baf-fd36504d1f43-istio-data\") pod \"e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s\" (UID: \"01a0ed6d-e285-4062-9baf-fd36504d1f43\") " pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"
Nov 01 04:02:00.042765 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:00.042630    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01a0ed6d-e285-4062-9baf-fd36504d1f43-istio-token\") pod \"e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s\" (UID: \"01a0ed6d-e285-4062-9baf-fd36504d1f43\") " pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"
Nov 01 04:02:00.042765 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:00.042692    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xktm8\" (UniqueName: \"kubernetes.io/projected/01a0ed6d-e285-4062-9baf-fd36504d1f43-kube-api-access-xktm8\") pod \"e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s\" (UID: \"01a0ed6d-e285-4062-9baf-fd36504d1f43\") " pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"
Nov 01 04:02:00.042974 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:00.042785    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a0ed6d-e285-4062-9baf-fd36504d1f43-istiod-ca-cert\") pod \"e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s\" (UID: \"01a0ed6d-e285-4062-9baf-fd36504d1f43\") " pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"
Nov 01 04:02:00.042974 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:00.042817    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a0ed6d-e285-4062-9baf-fd36504d1f43-workload-socket\") pod \"e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s\" (UID: \"01a0ed6d-e285-4062-9baf-fd36504d1f43\") " pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"
Nov 01 04:02:00.042974 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:00.042857    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a0ed6d-e285-4062-9baf-fd36504d1f43-istio-envoy\") pod \"e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s\" (UID: \"01a0ed6d-e285-4062-9baf-fd36504d1f43\") " pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"
Nov 01 04:02:00.042974 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:00.042885    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a0ed6d-e285-4062-9baf-fd36504d1f43-workload-certs\") pod \"e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s\" (UID: \"01a0ed6d-e285-4062-9baf-fd36504d1f43\") " pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"
Nov 01 04:02:00.042974 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:00.042916    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a0ed6d-e285-4062-9baf-fd36504d1f43-istio-podinfo\") pod \"e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s\" (UID: \"01a0ed6d-e285-4062-9baf-fd36504d1f43\") " pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"
Nov 01 04:02:00.043287 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:00.042995    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a0ed6d-e285-4062-9baf-fd36504d1f43-credential-socket\") pod \"e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s\" (UID: \"01a0ed6d-e285-4062-9baf-fd36504d1f43\") " pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"
Nov 01 04:02:00.043287 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:00.043215    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a0ed6d-e285-4062-9baf-fd36504d1f43-istio-data\") pod \"e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s\" (UID: \"01a0ed6d-e285-4062-9baf-fd36504d1f43\") " pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"
Nov 01 04:02:00.043287 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:00.043265    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a0ed6d-e285-4062-9baf-fd36504d1f43-workload-socket\") pod \"e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s\" (UID: \"01a0ed6d-e285-4062-9baf-fd36504d1f43\") " pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"
Nov 01 04:02:00.043454 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:00.043300    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a0ed6d-e285-4062-9baf-fd36504d1f43-workload-certs\") pod \"e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s\" (UID: \"01a0ed6d-e285-4062-9baf-fd36504d1f43\") " pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"
Nov 01 04:02:00.043648 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:00.043626    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a0ed6d-e285-4062-9baf-fd36504d1f43-istiod-ca-cert\") pod \"e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s\" (UID: \"01a0ed6d-e285-4062-9baf-fd36504d1f43\") " pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"
Nov 01 04:02:00.045312 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:00.045285    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a0ed6d-e285-4062-9baf-fd36504d1f43-istio-envoy\") pod \"e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s\" (UID: \"01a0ed6d-e285-4062-9baf-fd36504d1f43\") " pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"
Nov 01 04:02:00.045527 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:00.045506    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a0ed6d-e285-4062-9baf-fd36504d1f43-istio-podinfo\") pod \"e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s\" (UID: \"01a0ed6d-e285-4062-9baf-fd36504d1f43\") " pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"
Nov 01 04:02:00.108202 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:00.108077    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xktm8\" (UniqueName: \"kubernetes.io/projected/01a0ed6d-e285-4062-9baf-fd36504d1f43-kube-api-access-xktm8\") pod \"e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s\" (UID: \"01a0ed6d-e285-4062-9baf-fd36504d1f43\") " pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"
Nov 01 04:02:00.108962 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:00.108795    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01a0ed6d-e285-4062-9baf-fd36504d1f43-istio-token\") pod \"e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s\" (UID: \"01a0ed6d-e285-4062-9baf-fd36504d1f43\") " pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"
Nov 01 04:02:00.165121 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:00.165024    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:02:00.183898 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:00.183868    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"
Nov 01 04:02:00.623781 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:00.623745    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"]
Nov 01 04:02:00.624158 ip-10-0-141-122 kubenswrapper[4930]: W1101 04:02:00.623749    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a0ed6d_e285_4062_9baf_fd36504d1f43.slice/crio-7213385fd846f2a7830b9371968803405cbba87c3532a7cc0b3e3849b44c87ca WatchSource:0}: Error finding container 7213385fd846f2a7830b9371968803405cbba87c3532a7cc0b3e3849b44c87ca: Status 404 returned error can't find the container with id 7213385fd846f2a7830b9371968803405cbba87c3532a7cc0b3e3849b44c87ca
Nov 01 04:02:00.626403 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:00.626370    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:02:00.626499 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:00.626434    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:02:00.626499 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:00.626462    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:02:01.476081 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:01.476039    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s" event={"ID":"01a0ed6d-e285-4062-9baf-fd36504d1f43","Type":"ContainerStarted","Data":"1f8f619174d0de1e49caa98c29003e62baf35cc032637515ce3b6c16cfb41b04"}
Nov 01 04:02:01.476275 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:01.476105    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s" event={"ID":"01a0ed6d-e285-4062-9baf-fd36504d1f43","Type":"ContainerStarted","Data":"7213385fd846f2a7830b9371968803405cbba87c3532a7cc0b3e3849b44c87ca"}
Nov 01 04:02:01.599207 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:01.599150    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s" podStartSLOduration=2.599133597 podStartE2EDuration="2.599133597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:01:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:02:01.595932752 +0000 UTC m=+3326.118939353" watchObservedRunningTime="2026-11-01 04:02:01.599133597 +0000 UTC m=+3326.122140182"
Nov 01 04:02:02.184201 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:02.184161    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"
Nov 01 04:02:02.189640 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:02.189610    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"
Nov 01 04:02:02.482518 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:02.482446    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"
Nov 01 04:02:02.483356 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:02.483335    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"
Nov 01 04:02:04.517828 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:04.517792    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:02:04.712998 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:04.712964    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:02:04.861731 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:04.861701    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:02:07.784384 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:07.784348    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:02:07.980876 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:07.980830    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:02:08.203815 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:08.203779    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-feacb8-fd9f6th7dt"]
Nov 01 04:02:08.206832 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:08.206812    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-feacb8-fd9f6th7dt"
Nov 01 04:02:08.316609 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:08.312896    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-0e7a475e-dockercfg-hqvlm\""
Nov 01 04:02:08.316609 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:08.313052    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-feacb8\""
Nov 01 04:02:08.318420 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:08.318325    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-feacb8-fd9f6th7dt"]
Nov 01 04:02:08.337643 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:08.337598    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6z69\" (UniqueName: \"kubernetes.io/projected/b722df12-81b7-46a9-8fe3-0047132bf1be-kube-api-access-m6z69\") pod \"payload-pre-processing-e2e-worker-single-main-feacb8-fd9f6th7dt\" (UID: \"b722df12-81b7-46a9-8fe3-0047132bf1be\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-feacb8-fd9f6th7dt"
Nov 01 04:02:08.337825 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:08.337769    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b722df12-81b7-46a9-8fe3-0047132bf1be-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-feacb8-fd9f6th7dt\" (UID: \"b722df12-81b7-46a9-8fe3-0047132bf1be\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-feacb8-fd9f6th7dt"
Nov 01 04:02:08.365242 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:08.365211    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-feacb8-75f48cddcfzthf"]
Nov 01 04:02:08.368681 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:08.368661    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-feacb8-75f48cddcfzthf"
Nov 01 04:02:08.420633 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:08.420601    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-feacb8-75f48cddcfzthf"]
Nov 01 04:02:08.438534 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:08.438507    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6z69\" (UniqueName: \"kubernetes.io/projected/b722df12-81b7-46a9-8fe3-0047132bf1be-kube-api-access-m6z69\") pod \"payload-pre-processing-e2e-worker-single-main-feacb8-fd9f6th7dt\" (UID: \"b722df12-81b7-46a9-8fe3-0047132bf1be\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-feacb8-fd9f6th7dt"
Nov 01 04:02:08.438686 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:08.438588    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbmfk\" (UniqueName: \"kubernetes.io/projected/a4eaeaed-f66c-4719-adf2-759a555e07e6-kube-api-access-vbmfk\") pod \"payload-processing-e2e-worker-single-main-feacb8-75f48cddcfzthf\" (UID: \"a4eaeaed-f66c-4719-adf2-759a555e07e6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-feacb8-75f48cddcfzthf"
Nov 01 04:02:08.438686 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:08.438625    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4eaeaed-f66c-4719-adf2-759a555e07e6-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-feacb8-75f48cddcfzthf\" (UID: \"a4eaeaed-f66c-4719-adf2-759a555e07e6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-feacb8-75f48cddcfzthf"
Nov 01 04:02:08.438841 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:08.438816    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b722df12-81b7-46a9-8fe3-0047132bf1be-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-feacb8-fd9f6th7dt\" (UID: \"b722df12-81b7-46a9-8fe3-0047132bf1be\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-feacb8-fd9f6th7dt"
Nov 01 04:02:08.439518 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:08.439501    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b722df12-81b7-46a9-8fe3-0047132bf1be-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-feacb8-fd9f6th7dt\" (UID: \"b722df12-81b7-46a9-8fe3-0047132bf1be\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-feacb8-fd9f6th7dt"
Nov 01 04:02:08.499374 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:08.499312    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6z69\" (UniqueName: \"kubernetes.io/projected/b722df12-81b7-46a9-8fe3-0047132bf1be-kube-api-access-m6z69\") pod \"payload-pre-processing-e2e-worker-single-main-feacb8-fd9f6th7dt\" (UID: \"b722df12-81b7-46a9-8fe3-0047132bf1be\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-feacb8-fd9f6th7dt"
Nov 01 04:02:08.520392 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:08.520367    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-feacb8-fd9f6th7dt"
Nov 01 04:02:08.539425 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:08.539399    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbmfk\" (UniqueName: \"kubernetes.io/projected/a4eaeaed-f66c-4719-adf2-759a555e07e6-kube-api-access-vbmfk\") pod \"payload-processing-e2e-worker-single-main-feacb8-75f48cddcfzthf\" (UID: \"a4eaeaed-f66c-4719-adf2-759a555e07e6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-feacb8-75f48cddcfzthf"
Nov 01 04:02:08.539543 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:08.539453    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4eaeaed-f66c-4719-adf2-759a555e07e6-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-feacb8-75f48cddcfzthf\" (UID: \"a4eaeaed-f66c-4719-adf2-759a555e07e6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-feacb8-75f48cddcfzthf"
Nov 01 04:02:08.540371 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:08.540346    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4eaeaed-f66c-4719-adf2-759a555e07e6-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-feacb8-75f48cddcfzthf\" (UID: \"a4eaeaed-f66c-4719-adf2-759a555e07e6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-feacb8-75f48cddcfzthf"
Nov 01 04:02:08.601772 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:08.601723    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbmfk\" (UniqueName: \"kubernetes.io/projected/a4eaeaed-f66c-4719-adf2-759a555e07e6-kube-api-access-vbmfk\") pod \"payload-processing-e2e-worker-single-main-feacb8-75f48cddcfzthf\" (UID: \"a4eaeaed-f66c-4719-adf2-759a555e07e6\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-feacb8-75f48cddcfzthf"
Nov 01 04:02:08.682117 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:08.682088    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-feacb8-75f48cddcfzthf"
Nov 01 04:02:08.894757 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:08.894732    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-feacb8-fd9f6th7dt"]
Nov 01 04:02:08.896367 ip-10-0-141-122 kubenswrapper[4930]: W1101 04:02:08.896335    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb722df12_81b7_46a9_8fe3_0047132bf1be.slice/crio-e5dc074f5ab528c0b31a0878d679963b09513f406543134de4f17b40091d43a6 WatchSource:0}: Error finding container e5dc074f5ab528c0b31a0878d679963b09513f406543134de4f17b40091d43a6: Status 404 returned error can't find the container with id e5dc074f5ab528c0b31a0878d679963b09513f406543134de4f17b40091d43a6
Nov 01 04:02:09.065378 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:09.065351    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-feacb8-75f48cddcfzthf"]
Nov 01 04:02:09.066548 ip-10-0-141-122 kubenswrapper[4930]: W1101 04:02:09.066522    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4eaeaed_f66c_4719_adf2_759a555e07e6.slice/crio-d71754c8dcb6f4a4d7b76ea2933843325449095779e1f0f15aec6e376e5ef2d6 WatchSource:0}: Error finding container d71754c8dcb6f4a4d7b76ea2933843325449095779e1f0f15aec6e376e5ef2d6: Status 404 returned error can't find the container with id d71754c8dcb6f4a4d7b76ea2933843325449095779e1f0f15aec6e376e5ef2d6
Nov 01 04:02:09.527535 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:09.527498    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-feacb8-fd9f6th7dt" event={"ID":"b722df12-81b7-46a9-8fe3-0047132bf1be","Type":"ContainerStarted","Data":"e6b735e207078be8432e538c73466557512b7742e0d6ccd0e67e6ebcb55ba8df"}
Nov 01 04:02:09.527780 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:09.527582    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-feacb8-fd9f6th7dt" event={"ID":"b722df12-81b7-46a9-8fe3-0047132bf1be","Type":"ContainerStarted","Data":"e5dc074f5ab528c0b31a0878d679963b09513f406543134de4f17b40091d43a6"}
Nov 01 04:02:09.527780 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:09.527735    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-feacb8-fd9f6th7dt"
Nov 01 04:02:09.529197 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:09.529165    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-feacb8-75f48cddcfzthf" event={"ID":"a4eaeaed-f66c-4719-adf2-759a555e07e6","Type":"ContainerStarted","Data":"7bd6c7764533cc62d49dafc1b9708152bee682921dc841e6326b7f61420c59fb"}
Nov 01 04:02:09.529341 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:09.529214    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-feacb8-75f48cddcfzthf" event={"ID":"a4eaeaed-f66c-4719-adf2-759a555e07e6","Type":"ContainerStarted","Data":"d71754c8dcb6f4a4d7b76ea2933843325449095779e1f0f15aec6e376e5ef2d6"}
Nov 01 04:02:09.529341 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:09.529258    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-feacb8-75f48cddcfzthf"
Nov 01 04:02:09.642977 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:09.642923    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-feacb8-fd9f6th7dt" podStartSLOduration=1.6429084029999999 podStartE2EDuration="1.642908403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:02:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:02:09.642412229 +0000 UTC m=+3334.165418818" watchObservedRunningTime="2026-11-01 04:02:09.642908403 +0000 UTC m=+3334.165914988"
Nov 01 04:02:09.759306 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:09.759252    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-feacb8-75f48cddcfzthf" podStartSLOduration=1.759234498 podStartE2EDuration="1.759234498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:02:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:02:09.757763993 +0000 UTC m=+3334.280770581" watchObservedRunningTime="2026-11-01 04:02:09.759234498 +0000 UTC m=+3334.282241087"
Nov 01 04:02:10.726678 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:10.726644    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:02:10.918287 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:10.918235    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:02:11.070094 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:11.070007    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:02:15.757432 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:15.757389    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:02:19.122684 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:02:19.122647    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 04:02:20.048234 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:20.048199    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jwsck"]
Nov 01 04:02:20.097429 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:20.097392    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jwsck"]
Nov 01 04:02:20.539222 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:20.539193    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-feacb8-fd9f6th7dt"
Nov 01 04:02:20.539664 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:20.539266    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-feacb8-75f48cddcfzthf"
Nov 01 04:02:20.603772 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:20.603725    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:02:21.618936 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:21.618902    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7589975d-4a3a-4039-8788-840ae68d295e" path="/var/lib/kubelet/pods/7589975d-4a3a-4039-8788-840ae68d295e/volumes"
Nov 01 04:02:25.491093 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:25.491047    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:02:28.654393 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:28.654308    4930 generic.go:356] "Generic (PLEG): container finished" podID="01a0ed6d-e285-4062-9baf-fd36504d1f43" containerID="1f8f619174d0de1e49caa98c29003e62baf35cc032637515ce3b6c16cfb41b04" exitCode=0
Nov 01 04:02:28.654393 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:28.654355    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s" event={"ID":"01a0ed6d-e285-4062-9baf-fd36504d1f43","Type":"ContainerDied","Data":"1f8f619174d0de1e49caa98c29003e62baf35cc032637515ce3b6c16cfb41b04"}
Nov 01 04:02:28.654878 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:28.654751    4930 scope.go:122] "RemoveContainer" containerID="1f8f619174d0de1e49caa98c29003e62baf35cc032637515ce3b6c16cfb41b04"
Nov 01 04:02:28.656000 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:28.655968    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:02:28.656110 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:28.656076    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:02:28.656178 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:28.656124    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:02:29.410167 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:29.410128    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:02:29.688722 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:29.688626    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s" event={"ID":"01a0ed6d-e285-4062-9baf-fd36504d1f43","Type":"ContainerStarted","Data":"fcbd152aef85a055256333b1c4fd2de393770e04054e9d23efb2dd665475f6a0"}
Nov 01 04:02:29.955330 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:29.955241    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:02:30.054063 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:30.054025    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trpfk"]
Nov 01 04:02:30.087246 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:30.087202    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trpfk"]
Nov 01 04:02:30.111134 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:30.111101    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:02:30.188536 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:30.188001    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"
Nov 01 04:02:30.188536 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:30.188163    4930 patch_prober.go:29] interesting pod/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body=
Nov 01 04:02:30.188536 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:30.188247    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s" podUID="01a0ed6d-e285-4062-9baf-fd36504d1f43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused"
Nov 01 04:02:30.279638 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:30.271051    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:02:30.709717 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:30.709679    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:02:31.184315 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:31.184279    4930 patch_prober.go:29] interesting pod/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body=
Nov 01 04:02:31.184543 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:31.184341    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s" podUID="01a0ed6d-e285-4062-9baf-fd36504d1f43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused"
Nov 01 04:02:31.620336 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:31.620304    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b" path="/var/lib/kubelet/pods/f4a069dd-1cdf-4bf2-adf5-fcdb56fc0b3b/volumes"
Nov 01 04:02:32.185206 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:32.185173    4930 patch_prober.go:29] interesting pod/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body=
Nov 01 04:02:32.185674 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:32.185228    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s" podUID="01a0ed6d-e285-4062-9baf-fd36504d1f43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused"
Nov 01 04:02:32.483641 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:32.482655    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"
Nov 01 04:02:32.828348 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:32.828259    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:02:33.189583 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:33.184978    4930 patch_prober.go:29] interesting pod/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body=
Nov 01 04:02:33.189583 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:33.185061    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s" podUID="01a0ed6d-e285-4062-9baf-fd36504d1f43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused"
Nov 01 04:02:34.184995 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:34.184957    4930 patch_prober.go:29] interesting pod/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused" start-of-body=
Nov 01 04:02:34.185198 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:34.185031    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s" podUID="01a0ed6d-e285-4062-9baf-fd36504d1f43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.23:15021/healthz/ready\": dial tcp 10.133.1.23:15021: connect: connection refused"
Nov 01 04:02:35.188381 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:35.188348    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"
Nov 01 04:02:35.189405 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:35.189383    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"
Nov 01 04:02:35.585425 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:35.585394    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:02:40.421186 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:40.421152    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:02:40.911477 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:40.911442    4930 scope.go:122] "RemoveContainer" containerID="25d2edfee921c98ece522f5d5efdc63e1e38b9382801dcc333e95a71e2a3a423"
Nov 01 04:02:40.920679 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:40.920653    4930 scope.go:122] "RemoveContainer" containerID="d59754cce81189cfed2785aa88b550a91fda44f252db89c20bc1a597f11e3c89"
Nov 01 04:02:45.274476 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:45.274429    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:02:45.897929 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:45.897861    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:02:49.121351 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:02:49.121271    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 04:02:50.088626 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:50.088577    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:02:54.885052 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:54.885017    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:02:55.717126 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:55.717091    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:02:59.783339 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:02:59.783292    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:03:04.594404 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:04.594365    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:03:08.048336 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:08.048301    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rsnp6"]
Nov 01 04:03:08.097769 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:08.097735    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rsnp6"]
Nov 01 04:03:09.552351 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:09.552306    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:03:09.620107 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:09.620075    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd" path="/var/lib/kubelet/pods/55f00eb5-8c9e-4e42-a266-77a8a5e2f8fd/volumes"
Nov 01 04:03:14.280849 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:14.280816    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:03:19.121138 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:03:19.121099    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 04:03:19.186147 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:19.186112    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:03:24.003288 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:24.003255    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:03:28.872699 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:28.872664    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:03:29.453661 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:29.453610    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:03:29.985826 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:29.985782    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-feacb8-qz857"]
Nov 01 04:03:29.990380 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:29.990358    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-feacb8-qz857"
Nov 01 04:03:30.048832 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:30.048793    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-feacb8-qz857"]
Nov 01 04:03:30.080592 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:30.080535    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4aaf866-5c93-48e1-b893-8bac00fd35ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-feacb8-qz857\" (UID: \"d4aaf866-5c93-48e1-b893-8bac00fd35ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-feacb8-qz857"
Nov 01 04:03:30.080747 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:30.080613    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9s8m\" (UniqueName: \"kubernetes.io/projected/d4aaf866-5c93-48e1-b893-8bac00fd35ab-kube-api-access-p9s8m\") pod \"maas-api-revoke-keys-e2e-worker-single-main-feacb8-qz857\" (UID: \"d4aaf866-5c93-48e1-b893-8bac00fd35ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-feacb8-qz857"
Nov 01 04:03:30.080747 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:30.080639    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4aaf866-5c93-48e1-b893-8bac00fd35ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-feacb8-qz857\" (UID: \"d4aaf866-5c93-48e1-b893-8bac00fd35ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-feacb8-qz857"
Nov 01 04:03:30.094681 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:30.094654    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:03:30.094804 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:30.094762    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 04:03:30.094877 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:30.094821    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 04:03:30.094877 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:30.094827    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 04:03:30.181978 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:30.181951    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4aaf866-5c93-48e1-b893-8bac00fd35ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-feacb8-qz857\" (UID: \"d4aaf866-5c93-48e1-b893-8bac00fd35ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-feacb8-qz857"
Nov 01 04:03:30.182146 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:30.181993    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9s8m\" (UniqueName: \"kubernetes.io/projected/d4aaf866-5c93-48e1-b893-8bac00fd35ab-kube-api-access-p9s8m\") pod \"maas-api-revoke-keys-e2e-worker-single-main-feacb8-qz857\" (UID: \"d4aaf866-5c93-48e1-b893-8bac00fd35ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-feacb8-qz857"
Nov 01 04:03:30.182146 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:30.182015    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4aaf866-5c93-48e1-b893-8bac00fd35ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-feacb8-qz857\" (UID: \"d4aaf866-5c93-48e1-b893-8bac00fd35ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-feacb8-qz857"
Nov 01 04:03:30.182743 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:30.182719    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4aaf866-5c93-48e1-b893-8bac00fd35ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-feacb8-qz857\" (UID: \"d4aaf866-5c93-48e1-b893-8bac00fd35ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-feacb8-qz857"
Nov 01 04:03:30.184378 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:30.184347    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4aaf866-5c93-48e1-b893-8bac00fd35ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-feacb8-qz857\" (UID: \"d4aaf866-5c93-48e1-b893-8bac00fd35ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-feacb8-qz857"
Nov 01 04:03:30.243474 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:30.243418    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9s8m\" (UniqueName: \"kubernetes.io/projected/d4aaf866-5c93-48e1-b893-8bac00fd35ab-kube-api-access-p9s8m\") pod \"maas-api-revoke-keys-e2e-worker-single-main-feacb8-qz857\" (UID: \"d4aaf866-5c93-48e1-b893-8bac00fd35ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-feacb8-qz857"
Nov 01 04:03:30.303411 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:30.303385    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-feacb8-qz857"
Nov 01 04:03:30.676279 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:30.676251    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-feacb8-qz857"]
Nov 01 04:03:30.678129 ip-10-0-141-122 kubenswrapper[4930]: W1101 04:03:30.678103    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4aaf866_5c93_48e1_b893_8bac00fd35ab.slice/crio-d1e200837ae12b328413957599612a1954d7e8a67cfa90bc8d7e4a1b5ffc9fb5 WatchSource:0}: Error finding container d1e200837ae12b328413957599612a1954d7e8a67cfa90bc8d7e4a1b5ffc9fb5: Status 404 returned error can't find the container with id d1e200837ae12b328413957599612a1954d7e8a67cfa90bc8d7e4a1b5ffc9fb5
Nov 01 04:03:31.069082 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:31.069040    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-feacb8-qz857" event={"ID":"d4aaf866-5c93-48e1-b893-8bac00fd35ab","Type":"ContainerStarted","Data":"f8071fa2e9ab4a5798afba02dc5fcbfa41d60c4d13f30083e152da5ffc6b7c10"}
Nov 01 04:03:31.069471 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:31.069104    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-feacb8-qz857" event={"ID":"d4aaf866-5c93-48e1-b893-8bac00fd35ab","Type":"ContainerStarted","Data":"d1e200837ae12b328413957599612a1954d7e8a67cfa90bc8d7e4a1b5ffc9fb5"}
Nov 01 04:03:31.184003 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:31.183944    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-feacb8-qz857" podStartSLOduration=2.183931048 podStartE2EDuration="2.183931048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:03:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:03:31.183698447 +0000 UTC m=+3415.706705037" watchObservedRunningTime="2026-11-01 04:03:31.183931048 +0000 UTC m=+3415.706937636"
Nov 01 04:03:32.083676 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:32.083633    4930 generic.go:356] "Generic (PLEG): container finished" podID="d4aaf866-5c93-48e1-b893-8bac00fd35ab" containerID="f8071fa2e9ab4a5798afba02dc5fcbfa41d60c4d13f30083e152da5ffc6b7c10" exitCode=0
Nov 01 04:03:32.084200 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:32.083721    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-feacb8-qz857" event={"ID":"d4aaf866-5c93-48e1-b893-8bac00fd35ab","Type":"ContainerDied","Data":"f8071fa2e9ab4a5798afba02dc5fcbfa41d60c4d13f30083e152da5ffc6b7c10"}
Nov 01 04:03:33.241721 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:33.241692    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-feacb8-qz857"
Nov 01 04:03:33.312041 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:33.312010    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4aaf866-5c93-48e1-b893-8bac00fd35ab-kube-api-access-p9s8m\" (UniqueName: \"kubernetes.io/projected/d4aaf866-5c93-48e1-b893-8bac00fd35ab-kube-api-access-p9s8m\") pod \"d4aaf866-5c93-48e1-b893-8bac00fd35ab\" (UID: \"d4aaf866-5c93-48e1-b893-8bac00fd35ab\") "
Nov 01 04:03:33.312221 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:33.312168    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4aaf866-5c93-48e1-b893-8bac00fd35ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4aaf866-5c93-48e1-b893-8bac00fd35ab-maas-api-xks-ca\") pod \"d4aaf866-5c93-48e1-b893-8bac00fd35ab\" (UID: \"d4aaf866-5c93-48e1-b893-8bac00fd35ab\") "
Nov 01 04:03:33.312221 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:33.312202    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4aaf866-5c93-48e1-b893-8bac00fd35ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4aaf866-5c93-48e1-b893-8bac00fd35ab-maas-api-openshift-ca\") pod \"d4aaf866-5c93-48e1-b893-8bac00fd35ab\" (UID: \"d4aaf866-5c93-48e1-b893-8bac00fd35ab\") "
Nov 01 04:03:33.312653 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:33.312630    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4aaf866-5c93-48e1-b893-8bac00fd35ab-maas-api-openshift-ca" pod "d4aaf866-5c93-48e1-b893-8bac00fd35ab" (UID: "d4aaf866-5c93-48e1-b893-8bac00fd35ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:03:33.314243 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:33.314185    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4aaf866-5c93-48e1-b893-8bac00fd35ab-kube-api-access-p9s8m" pod "d4aaf866-5c93-48e1-b893-8bac00fd35ab" (UID: "d4aaf866-5c93-48e1-b893-8bac00fd35ab"). InnerVolumeSpecName "kube-api-access-p9s8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:03:33.314386 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:33.314368    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4aaf866-5c93-48e1-b893-8bac00fd35ab-maas-api-xks-ca" pod "d4aaf866-5c93-48e1-b893-8bac00fd35ab" (UID: "d4aaf866-5c93-48e1-b893-8bac00fd35ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:03:33.413360 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:33.413269    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4aaf866-5c93-48e1-b893-8bac00fd35ab-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:03:33.413360 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:33.413323    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4aaf866-5c93-48e1-b893-8bac00fd35ab-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:03:33.413360 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:33.413342    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9s8m\" (UniqueName: \"kubernetes.io/projected/d4aaf866-5c93-48e1-b893-8bac00fd35ab-kube-api-access-p9s8m\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:03:34.065124 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:34.065085    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:03:34.097018 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:34.096989    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-feacb8-qz857"
Nov 01 04:03:34.097154 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:34.097022    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-feacb8-qz857" event={"ID":"d4aaf866-5c93-48e1-b893-8bac00fd35ab","Type":"ContainerDied","Data":"d1e200837ae12b328413957599612a1954d7e8a67cfa90bc8d7e4a1b5ffc9fb5"}
Nov 01 04:03:34.097154 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:34.097067    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1e200837ae12b328413957599612a1954d7e8a67cfa90bc8d7e4a1b5ffc9fb5"
Nov 01 04:03:34.501184 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:34.501126    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:03:37.047867 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:37.047835    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6v5jl"]
Nov 01 04:03:37.097495 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:37.097465    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-6v5jl"]
Nov 01 04:03:37.620604 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:37.620573    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f5f13d9-80f9-40eb-a583-e622e57ee99d" path="/var/lib/kubelet/pods/9f5f13d9-80f9-40eb-a583-e622e57ee99d/volumes"
Nov 01 04:03:38.974500 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:38.974459    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:03:40.948345 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:40.948316    4930 scope.go:122] "RemoveContainer" containerID="ccefd226612ec3e1520f00e71175ecea83ffabd70229168695bba41d1975870d"
Nov 01 04:03:40.957675 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:40.957654    4930 scope.go:122] "RemoveContainer" containerID="aad41478a96ae91d6c9017a1783dabe053b8cea19c29c21166a7d315a0b2d1ab"
Nov 01 04:03:40.966189 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:40.966167    4930 scope.go:122] "RemoveContainer" containerID="da4a11eaab9de794cc945dc6df6a005c1aff874c57fa6c854808220d07775039"
Nov 01 04:03:41.506344 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:41.506308    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-feacb8-75f48cddcfzthf"]
Nov 01 04:03:41.620171 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:41.620141    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-feacb8-fd9f6th7dt"]
Nov 01 04:03:41.867944 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:41.867905    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:03:42.063747 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:42.063709    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:03:43.785891 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:43.785850    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:03:46.230551 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:46.230507    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:03:46.507058 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:46.506960    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-feacb8-75f48cddcfzthf" podUID="a4eaeaed-f66c-4719-adf2-759a555e07e6" containerName="payload-processing" containerID="cri-o://7bd6c7764533cc62d49dafc1b9708152bee682921dc841e6326b7f61420c59fb" gracePeriod=25
Nov 01 04:03:46.621250 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:46.621219    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-feacb8-fd9f6th7dt" podUID="b722df12-81b7-46a9-8fe3-0047132bf1be" containerName="payload-pre-processing" containerID="cri-o://e6b735e207078be8432e538c73466557512b7742e0d6ccd0e67e6ebcb55ba8df" gracePeriod=25
Nov 01 04:03:48.106797 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:48.106759    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:03:49.121223 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:03:49.121189    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 04:03:49.672784 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:49.672741    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:03:49.927593 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:49.927478    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:03:50.266842 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:50.266760    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-feacb8-qz857"]
Nov 01 04:03:50.315864 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:50.315835    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-feacb8-qz857"]
Nov 01 04:03:50.537575 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:50.537470    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-feacb8-fd9f6th7dt" podUID="b722df12-81b7-46a9-8fe3-0047132bf1be" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 04:03:50.537752 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:50.537647    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-feacb8-75f48cddcfzthf" podUID="a4eaeaed-f66c-4719-adf2-759a555e07e6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 04:03:50.690232 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:50.690199    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:03:51.619538 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:51.619504    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4aaf866-5c93-48e1-b893-8bac00fd35ab" path="/var/lib/kubelet/pods/d4aaf866-5c93-48e1-b893-8bac00fd35ab/volumes"
Nov 01 04:03:56.061908 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:56.061873    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:03:56.672244 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:56.672200    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:03:56.862733 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:56.862701    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:03:56.982911 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:56.982833    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"]
Nov 01 04:03:56.983096 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:56.983072    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s" podUID="01a0ed6d-e285-4062-9baf-fd36504d1f43" containerName="istio-proxy" containerID="cri-o://fcbd152aef85a055256333b1c4fd2de393770e04054e9d23efb2dd665475f6a0" gracePeriod=30
Nov 01 04:03:57.013204 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:57.013172    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:03:58.548158 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:58.548125    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ll4dv"]
Nov 01 04:03:58.597607 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:58.597571    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ll4dv"]
Nov 01 04:03:59.620099 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:03:59.620057    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="417b98ad-5bb3-4be6-b51b-25569c5e4f90" path="/var/lib/kubelet/pods/417b98ad-5bb3-4be6-b51b-25569c5e4f90/volumes"
Nov 01 04:04:00.536700 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:00.536663    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-feacb8-75f48cddcfzthf" podUID="a4eaeaed-f66c-4719-adf2-759a555e07e6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 04:04:00.536904 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:00.536671    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-feacb8-fd9f6th7dt" podUID="b722df12-81b7-46a9-8fe3-0047132bf1be" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 04:04:02.235020 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:02.234996    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"
Nov 01 04:04:02.267409 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:02.267353    4930 generic.go:356] "Generic (PLEG): container finished" podID="01a0ed6d-e285-4062-9baf-fd36504d1f43" containerID="fcbd152aef85a055256333b1c4fd2de393770e04054e9d23efb2dd665475f6a0" exitCode=0
Nov 01 04:04:02.267524 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:02.267407    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"
Nov 01 04:04:02.267524 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:02.267430    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s" event={"ID":"01a0ed6d-e285-4062-9baf-fd36504d1f43","Type":"ContainerDied","Data":"fcbd152aef85a055256333b1c4fd2de393770e04054e9d23efb2dd665475f6a0"}
Nov 01 04:04:02.267524 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:02.267484    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s" event={"ID":"01a0ed6d-e285-4062-9baf-fd36504d1f43","Type":"ContainerDied","Data":"7213385fd846f2a7830b9371968803405cbba87c3532a7cc0b3e3849b44c87ca"}
Nov 01 04:04:02.267524 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:02.267509    4930 scope.go:122] "RemoveContainer" containerID="fcbd152aef85a055256333b1c4fd2de393770e04054e9d23efb2dd665475f6a0"
Nov 01 04:04:02.276839 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:02.276823    4930 scope.go:122] "RemoveContainer" containerID="1f8f619174d0de1e49caa98c29003e62baf35cc032637515ce3b6c16cfb41b04"
Nov 01 04:04:02.285114 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:02.285097    4930 scope.go:122] "RemoveContainer" containerID="fcbd152aef85a055256333b1c4fd2de393770e04054e9d23efb2dd665475f6a0"
Nov 01 04:04:02.285359 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:04:02.285344    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcbd152aef85a055256333b1c4fd2de393770e04054e9d23efb2dd665475f6a0\": container with ID starting with fcbd152aef85a055256333b1c4fd2de393770e04054e9d23efb2dd665475f6a0 not found: ID does not exist" containerID="fcbd152aef85a055256333b1c4fd2de393770e04054e9d23efb2dd665475f6a0"
Nov 01 04:04:02.285425 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:02.285365    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcbd152aef85a055256333b1c4fd2de393770e04054e9d23efb2dd665475f6a0"} err="failed to get container status \"fcbd152aef85a055256333b1c4fd2de393770e04054e9d23efb2dd665475f6a0\": rpc error: code = NotFound desc = could not find container \"fcbd152aef85a055256333b1c4fd2de393770e04054e9d23efb2dd665475f6a0\": container with ID starting with fcbd152aef85a055256333b1c4fd2de393770e04054e9d23efb2dd665475f6a0 not found: ID does not exist"
Nov 01 04:04:02.285425 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:02.285380    4930 scope.go:122] "RemoveContainer" containerID="1f8f619174d0de1e49caa98c29003e62baf35cc032637515ce3b6c16cfb41b04"
Nov 01 04:04:02.285710 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:04:02.285692    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f8f619174d0de1e49caa98c29003e62baf35cc032637515ce3b6c16cfb41b04\": container with ID starting with 1f8f619174d0de1e49caa98c29003e62baf35cc032637515ce3b6c16cfb41b04 not found: ID does not exist" containerID="1f8f619174d0de1e49caa98c29003e62baf35cc032637515ce3b6c16cfb41b04"
Nov 01 04:04:02.285774 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:02.285722    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f8f619174d0de1e49caa98c29003e62baf35cc032637515ce3b6c16cfb41b04"} err="failed to get container status \"1f8f619174d0de1e49caa98c29003e62baf35cc032637515ce3b6c16cfb41b04\": rpc error: code = NotFound desc = could not find container \"1f8f619174d0de1e49caa98c29003e62baf35cc032637515ce3b6c16cfb41b04\": container with ID starting with 1f8f619174d0de1e49caa98c29003e62baf35cc032637515ce3b6c16cfb41b04 not found: ID does not exist"
Nov 01 04:04:02.315770 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:02.315744    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a0ed6d-e285-4062-9baf-fd36504d1f43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a0ed6d-e285-4062-9baf-fd36504d1f43-istio-envoy\") pod \"01a0ed6d-e285-4062-9baf-fd36504d1f43\" (UID: \"01a0ed6d-e285-4062-9baf-fd36504d1f43\") "
Nov 01 04:04:02.315881 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:02.315799    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a0ed6d-e285-4062-9baf-fd36504d1f43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a0ed6d-e285-4062-9baf-fd36504d1f43-workload-certs\") pod \"01a0ed6d-e285-4062-9baf-fd36504d1f43\" (UID: \"01a0ed6d-e285-4062-9baf-fd36504d1f43\") "
Nov 01 04:04:02.315881 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:02.315832    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01a0ed6d-e285-4062-9baf-fd36504d1f43-istio-token\" (UniqueName: \"kubernetes.io/projected/01a0ed6d-e285-4062-9baf-fd36504d1f43-istio-token\") pod \"01a0ed6d-e285-4062-9baf-fd36504d1f43\" (UID: \"01a0ed6d-e285-4062-9baf-fd36504d1f43\") "
Nov 01 04:04:02.315881 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:02.315871    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a0ed6d-e285-4062-9baf-fd36504d1f43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a0ed6d-e285-4062-9baf-fd36504d1f43-workload-socket\") pod \"01a0ed6d-e285-4062-9baf-fd36504d1f43\" (UID: \"01a0ed6d-e285-4062-9baf-fd36504d1f43\") "
Nov 01 04:04:02.316056 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:02.315910    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a0ed6d-e285-4062-9baf-fd36504d1f43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a0ed6d-e285-4062-9baf-fd36504d1f43-istio-data\") pod \"01a0ed6d-e285-4062-9baf-fd36504d1f43\" (UID: \"01a0ed6d-e285-4062-9baf-fd36504d1f43\") "
Nov 01 04:04:02.316056 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:02.315948    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a0ed6d-e285-4062-9baf-fd36504d1f43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a0ed6d-e285-4062-9baf-fd36504d1f43-credential-socket\") pod \"01a0ed6d-e285-4062-9baf-fd36504d1f43\" (UID: \"01a0ed6d-e285-4062-9baf-fd36504d1f43\") "
Nov 01 04:04:02.316056 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:02.316015    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01a0ed6d-e285-4062-9baf-fd36504d1f43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a0ed6d-e285-4062-9baf-fd36504d1f43-istiod-ca-cert\") pod \"01a0ed6d-e285-4062-9baf-fd36504d1f43\" (UID: \"01a0ed6d-e285-4062-9baf-fd36504d1f43\") "
Nov 01 04:04:02.316222 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:02.316056    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01a0ed6d-e285-4062-9baf-fd36504d1f43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a0ed6d-e285-4062-9baf-fd36504d1f43-istio-podinfo\") pod \"01a0ed6d-e285-4062-9baf-fd36504d1f43\" (UID: \"01a0ed6d-e285-4062-9baf-fd36504d1f43\") "
Nov 01 04:04:02.316222 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:02.316084    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a0ed6d-e285-4062-9baf-fd36504d1f43-workload-certs" pod "01a0ed6d-e285-4062-9baf-fd36504d1f43" (UID: "01a0ed6d-e285-4062-9baf-fd36504d1f43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:04:02.316222 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:02.316123    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01a0ed6d-e285-4062-9baf-fd36504d1f43-kube-api-access-xktm8\" (UniqueName: \"kubernetes.io/projected/01a0ed6d-e285-4062-9baf-fd36504d1f43-kube-api-access-xktm8\") pod \"01a0ed6d-e285-4062-9baf-fd36504d1f43\" (UID: \"01a0ed6d-e285-4062-9baf-fd36504d1f43\") "
Nov 01 04:04:02.316393 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:02.316215    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a0ed6d-e285-4062-9baf-fd36504d1f43-workload-socket" pod "01a0ed6d-e285-4062-9baf-fd36504d1f43" (UID: "01a0ed6d-e285-4062-9baf-fd36504d1f43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:04:02.316393 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:02.316283    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a0ed6d-e285-4062-9baf-fd36504d1f43-istio-data" pod "01a0ed6d-e285-4062-9baf-fd36504d1f43" (UID: "01a0ed6d-e285-4062-9baf-fd36504d1f43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:04:02.316393 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:02.316344    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a0ed6d-e285-4062-9baf-fd36504d1f43-credential-socket" pod "01a0ed6d-e285-4062-9baf-fd36504d1f43" (UID: "01a0ed6d-e285-4062-9baf-fd36504d1f43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:04:02.316570 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:02.316405    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01a0ed6d-e285-4062-9baf-fd36504d1f43-istiod-ca-cert" pod "01a0ed6d-e285-4062-9baf-fd36504d1f43" (UID: "01a0ed6d-e285-4062-9baf-fd36504d1f43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:04:02.316570 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:02.316540    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a0ed6d-e285-4062-9baf-fd36504d1f43-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:04:02.316662 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:02.316580    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a0ed6d-e285-4062-9baf-fd36504d1f43-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:04:02.316662 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:02.316597    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a0ed6d-e285-4062-9baf-fd36504d1f43-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:04:02.316662 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:02.316612    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a0ed6d-e285-4062-9baf-fd36504d1f43-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:04:02.316662 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:02.316628    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a0ed6d-e285-4062-9baf-fd36504d1f43-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:04:02.318048 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:02.318027    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a0ed6d-e285-4062-9baf-fd36504d1f43-istio-token" pod "01a0ed6d-e285-4062-9baf-fd36504d1f43" (UID: "01a0ed6d-e285-4062-9baf-fd36504d1f43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:04:02.318142 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:02.318097    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01a0ed6d-e285-4062-9baf-fd36504d1f43-istio-podinfo" pod "01a0ed6d-e285-4062-9baf-fd36504d1f43" (UID: "01a0ed6d-e285-4062-9baf-fd36504d1f43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:04:02.318380 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:02.318366    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a0ed6d-e285-4062-9baf-fd36504d1f43-istio-envoy" pod "01a0ed6d-e285-4062-9baf-fd36504d1f43" (UID: "01a0ed6d-e285-4062-9baf-fd36504d1f43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:04:02.318454 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:02.318438    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a0ed6d-e285-4062-9baf-fd36504d1f43-kube-api-access-xktm8" pod "01a0ed6d-e285-4062-9baf-fd36504d1f43" (UID: "01a0ed6d-e285-4062-9baf-fd36504d1f43"). InnerVolumeSpecName "kube-api-access-xktm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:04:02.417733 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:02.417705    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a0ed6d-e285-4062-9baf-fd36504d1f43-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:04:02.417836 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:02.417737    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xktm8\" (UniqueName: \"kubernetes.io/projected/01a0ed6d-e285-4062-9baf-fd36504d1f43-kube-api-access-xktm8\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:04:02.417836 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:02.417754    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a0ed6d-e285-4062-9baf-fd36504d1f43-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:04:02.417836 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:02.417767    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01a0ed6d-e285-4062-9baf-fd36504d1f43-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:04:02.689916 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:02.689880    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"]
Nov 01 04:04:02.695981 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:02.695958    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-feacb8-openshift-default-578799b757l4v2s"]
Nov 01 04:04:03.626855 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:03.626821    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01a0ed6d-e285-4062-9baf-fd36504d1f43" path="/var/lib/kubelet/pods/01a0ed6d-e285-4062-9baf-fd36504d1f43/volumes"
Nov 01 04:04:06.757347 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:06.757326    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-feacb8-75f48cddcfzthf"
Nov 01 04:04:06.855886 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:06.855858    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4eaeaed-f66c-4719-adf2-759a555e07e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4eaeaed-f66c-4719-adf2-759a555e07e6-plugins-config-volume\") pod \"a4eaeaed-f66c-4719-adf2-759a555e07e6\" (UID: \"a4eaeaed-f66c-4719-adf2-759a555e07e6\") "
Nov 01 04:04:06.856026 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:06.855941    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4eaeaed-f66c-4719-adf2-759a555e07e6-kube-api-access-vbmfk\" (UniqueName: \"kubernetes.io/projected/a4eaeaed-f66c-4719-adf2-759a555e07e6-kube-api-access-vbmfk\") pod \"a4eaeaed-f66c-4719-adf2-759a555e07e6\" (UID: \"a4eaeaed-f66c-4719-adf2-759a555e07e6\") "
Nov 01 04:04:06.856301 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:06.856275    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4eaeaed-f66c-4719-adf2-759a555e07e6-plugins-config-volume" pod "a4eaeaed-f66c-4719-adf2-759a555e07e6" (UID: "a4eaeaed-f66c-4719-adf2-759a555e07e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:04:06.857997 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:06.857966    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4eaeaed-f66c-4719-adf2-759a555e07e6-kube-api-access-vbmfk" pod "a4eaeaed-f66c-4719-adf2-759a555e07e6" (UID: "a4eaeaed-f66c-4719-adf2-759a555e07e6"). InnerVolumeSpecName "kube-api-access-vbmfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:04:06.877092 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:06.877070    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-feacb8-fd9f6th7dt"
Nov 01 04:04:06.957362 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:06.957335    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b722df12-81b7-46a9-8fe3-0047132bf1be-kube-api-access-m6z69\" (UniqueName: \"kubernetes.io/projected/b722df12-81b7-46a9-8fe3-0047132bf1be-kube-api-access-m6z69\") pod \"b722df12-81b7-46a9-8fe3-0047132bf1be\" (UID: \"b722df12-81b7-46a9-8fe3-0047132bf1be\") "
Nov 01 04:04:06.957480 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:06.957442    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b722df12-81b7-46a9-8fe3-0047132bf1be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b722df12-81b7-46a9-8fe3-0047132bf1be-plugins-config-volume\") pod \"b722df12-81b7-46a9-8fe3-0047132bf1be\" (UID: \"b722df12-81b7-46a9-8fe3-0047132bf1be\") "
Nov 01 04:04:06.957665 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:06.957652    4930 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4eaeaed-f66c-4719-adf2-759a555e07e6-plugins-config-volume\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:04:06.957718 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:06.957666    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbmfk\" (UniqueName: \"kubernetes.io/projected/a4eaeaed-f66c-4719-adf2-759a555e07e6-kube-api-access-vbmfk\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:04:06.957866 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:06.957845    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b722df12-81b7-46a9-8fe3-0047132bf1be-plugins-config-volume" pod "b722df12-81b7-46a9-8fe3-0047132bf1be" (UID: "b722df12-81b7-46a9-8fe3-0047132bf1be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:04:06.959591 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:06.959547    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b722df12-81b7-46a9-8fe3-0047132bf1be-kube-api-access-m6z69" pod "b722df12-81b7-46a9-8fe3-0047132bf1be" (UID: "b722df12-81b7-46a9-8fe3-0047132bf1be"). InnerVolumeSpecName "kube-api-access-m6z69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:04:07.048503 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:07.048420    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-z9fd9"]
Nov 01 04:04:07.051469 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:07.051441    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-z9fd9"]
Nov 01 04:04:07.058261 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:07.058237    4930 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b722df12-81b7-46a9-8fe3-0047132bf1be-plugins-config-volume\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:04:07.058329 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:07.058265    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6z69\" (UniqueName: \"kubernetes.io/projected/b722df12-81b7-46a9-8fe3-0047132bf1be-kube-api-access-m6z69\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:04:07.298128 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:07.298095    4930 generic.go:356] "Generic (PLEG): container finished" podID="b722df12-81b7-46a9-8fe3-0047132bf1be" containerID="e6b735e207078be8432e538c73466557512b7742e0d6ccd0e67e6ebcb55ba8df" exitCode=0
Nov 01 04:04:07.298349 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:07.298149    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-feacb8-fd9f6th7dt"
Nov 01 04:04:07.298349 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:07.298178    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-feacb8-fd9f6th7dt" event={"ID":"b722df12-81b7-46a9-8fe3-0047132bf1be","Type":"ContainerDied","Data":"e6b735e207078be8432e538c73466557512b7742e0d6ccd0e67e6ebcb55ba8df"}
Nov 01 04:04:07.298349 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:07.298232    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-feacb8-fd9f6th7dt" event={"ID":"b722df12-81b7-46a9-8fe3-0047132bf1be","Type":"ContainerDied","Data":"e5dc074f5ab528c0b31a0878d679963b09513f406543134de4f17b40091d43a6"}
Nov 01 04:04:07.298349 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:07.298258    4930 scope.go:122] "RemoveContainer" containerID="e6b735e207078be8432e538c73466557512b7742e0d6ccd0e67e6ebcb55ba8df"
Nov 01 04:04:07.299610 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:07.299586    4930 generic.go:356] "Generic (PLEG): container finished" podID="a4eaeaed-f66c-4719-adf2-759a555e07e6" containerID="7bd6c7764533cc62d49dafc1b9708152bee682921dc841e6326b7f61420c59fb" exitCode=0
Nov 01 04:04:07.299710 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:07.299630    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-feacb8-75f48cddcfzthf"
Nov 01 04:04:07.299710 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:07.299647    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-feacb8-75f48cddcfzthf" event={"ID":"a4eaeaed-f66c-4719-adf2-759a555e07e6","Type":"ContainerDied","Data":"7bd6c7764533cc62d49dafc1b9708152bee682921dc841e6326b7f61420c59fb"}
Nov 01 04:04:07.299710 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:07.299692    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-feacb8-75f48cddcfzthf" event={"ID":"a4eaeaed-f66c-4719-adf2-759a555e07e6","Type":"ContainerDied","Data":"d71754c8dcb6f4a4d7b76ea2933843325449095779e1f0f15aec6e376e5ef2d6"}
Nov 01 04:04:07.309207 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:07.309189    4930 scope.go:122] "RemoveContainer" containerID="e6b735e207078be8432e538c73466557512b7742e0d6ccd0e67e6ebcb55ba8df"
Nov 01 04:04:07.309480 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:04:07.309461    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6b735e207078be8432e538c73466557512b7742e0d6ccd0e67e6ebcb55ba8df\": container with ID starting with e6b735e207078be8432e538c73466557512b7742e0d6ccd0e67e6ebcb55ba8df not found: ID does not exist" containerID="e6b735e207078be8432e538c73466557512b7742e0d6ccd0e67e6ebcb55ba8df"
Nov 01 04:04:07.309583 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:07.309488    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6b735e207078be8432e538c73466557512b7742e0d6ccd0e67e6ebcb55ba8df"} err="failed to get container status \"e6b735e207078be8432e538c73466557512b7742e0d6ccd0e67e6ebcb55ba8df\": rpc error: code = NotFound desc = could not find container \"e6b735e207078be8432e538c73466557512b7742e0d6ccd0e67e6ebcb55ba8df\": container with ID starting with e6b735e207078be8432e538c73466557512b7742e0d6ccd0e67e6ebcb55ba8df not found: ID does not exist"
Nov 01 04:04:07.309583 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:07.309510    4930 scope.go:122] "RemoveContainer" containerID="7bd6c7764533cc62d49dafc1b9708152bee682921dc841e6326b7f61420c59fb"
Nov 01 04:04:07.317764 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:07.317747    4930 scope.go:122] "RemoveContainer" containerID="7bd6c7764533cc62d49dafc1b9708152bee682921dc841e6326b7f61420c59fb"
Nov 01 04:04:07.318020 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:04:07.318001    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bd6c7764533cc62d49dafc1b9708152bee682921dc841e6326b7f61420c59fb\": container with ID starting with 7bd6c7764533cc62d49dafc1b9708152bee682921dc841e6326b7f61420c59fb not found: ID does not exist" containerID="7bd6c7764533cc62d49dafc1b9708152bee682921dc841e6326b7f61420c59fb"
Nov 01 04:04:07.318076 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:07.318026    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bd6c7764533cc62d49dafc1b9708152bee682921dc841e6326b7f61420c59fb"} err="failed to get container status \"7bd6c7764533cc62d49dafc1b9708152bee682921dc841e6326b7f61420c59fb\": rpc error: code = NotFound desc = could not find container \"7bd6c7764533cc62d49dafc1b9708152bee682921dc841e6326b7f61420c59fb\": container with ID starting with 7bd6c7764533cc62d49dafc1b9708152bee682921dc841e6326b7f61420c59fb not found: ID does not exist"
Nov 01 04:04:07.462715 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:07.462690    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-feacb8-75f48cddcfzthf"]
Nov 01 04:04:07.468706 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:07.468684    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-feacb8-75f48cddcfzthf"]
Nov 01 04:04:07.578281 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:07.578230    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-feacb8-fd9f6th7dt"]
Nov 01 04:04:07.582312 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:07.582288    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-feacb8-fd9f6th7dt"]
Nov 01 04:04:07.619356 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:07.619331    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8144c0f7-1376-4854-85c4-3bdc5e03dd86" path="/var/lib/kubelet/pods/8144c0f7-1376-4854-85c4-3bdc5e03dd86/volumes"
Nov 01 04:04:07.619804 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:07.619786    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4eaeaed-f66c-4719-adf2-759a555e07e6" path="/var/lib/kubelet/pods/a4eaeaed-f66c-4719-adf2-759a555e07e6/volumes"
Nov 01 04:04:07.620143 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:07.620131    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b722df12-81b7-46a9-8fe3-0047132bf1be" path="/var/lib/kubelet/pods/b722df12-81b7-46a9-8fe3-0047132bf1be/volumes"
Nov 01 04:04:11.506135 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:11.506103    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:04:12.087807 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:12.087768    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:04:18.083229 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:18.083150    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:04:19.120655 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:04:19.120617    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 04:04:20.563056 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:20.563025    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b4sdg"]
Nov 01 04:04:20.563550 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:20.563221    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b4sdg" podUID="e4edaa96-e7f5-480f-9c85-b7189aedb11c" containerName="manager" containerID="cri-o://9d389f98887308f1f25fd47f24ba4ad9c22240696ad69ad51a8111fed9c43d7c" gracePeriod=10
Nov 01 04:04:20.825744 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:20.825688    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b4sdg"
Nov 01 04:04:20.902395 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:20.902366    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4edaa96-e7f5-480f-9c85-b7189aedb11c-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e4edaa96-e7f5-480f-9c85-b7189aedb11c-extensions-socket-volume\") pod \"e4edaa96-e7f5-480f-9c85-b7189aedb11c\" (UID: \"e4edaa96-e7f5-480f-9c85-b7189aedb11c\") "
Nov 01 04:04:20.902532 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:20.902414    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4edaa96-e7f5-480f-9c85-b7189aedb11c-kube-api-access-58tn4\" (UniqueName: \"kubernetes.io/projected/e4edaa96-e7f5-480f-9c85-b7189aedb11c-kube-api-access-58tn4\") pod \"e4edaa96-e7f5-480f-9c85-b7189aedb11c\" (UID: \"e4edaa96-e7f5-480f-9c85-b7189aedb11c\") "
Nov 01 04:04:20.902757 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:20.902734    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4edaa96-e7f5-480f-9c85-b7189aedb11c-extensions-socket-volume" pod "e4edaa96-e7f5-480f-9c85-b7189aedb11c" (UID: "e4edaa96-e7f5-480f-9c85-b7189aedb11c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:04:20.904451 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:20.904430    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4edaa96-e7f5-480f-9c85-b7189aedb11c-kube-api-access-58tn4" pod "e4edaa96-e7f5-480f-9c85-b7189aedb11c" (UID: "e4edaa96-e7f5-480f-9c85-b7189aedb11c"). InnerVolumeSpecName "kube-api-access-58tn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:04:21.003595 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:21.003551    4930 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e4edaa96-e7f5-480f-9c85-b7189aedb11c-extensions-socket-volume\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:04:21.003595 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:21.003592    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58tn4\" (UniqueName: \"kubernetes.io/projected/e4edaa96-e7f5-480f-9c85-b7189aedb11c-kube-api-access-58tn4\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:04:21.385804 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:21.385769    4930 generic.go:356] "Generic (PLEG): container finished" podID="e4edaa96-e7f5-480f-9c85-b7189aedb11c" containerID="9d389f98887308f1f25fd47f24ba4ad9c22240696ad69ad51a8111fed9c43d7c" exitCode=0
Nov 01 04:04:21.385984 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:21.385818    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b4sdg"
Nov 01 04:04:21.385984 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:21.385853    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b4sdg" event={"ID":"e4edaa96-e7f5-480f-9c85-b7189aedb11c","Type":"ContainerDied","Data":"9d389f98887308f1f25fd47f24ba4ad9c22240696ad69ad51a8111fed9c43d7c"}
Nov 01 04:04:21.385984 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:21.385919    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b4sdg" event={"ID":"e4edaa96-e7f5-480f-9c85-b7189aedb11c","Type":"ContainerDied","Data":"416be4b2e947684c2cc3799ab343fb45dbb0f8216e72e905e467235ceaa50d67"}
Nov 01 04:04:21.385984 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:21.385955    4930 scope.go:122] "RemoveContainer" containerID="9d389f98887308f1f25fd47f24ba4ad9c22240696ad69ad51a8111fed9c43d7c"
Nov 01 04:04:21.396681 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:21.396662    4930 scope.go:122] "RemoveContainer" containerID="9d389f98887308f1f25fd47f24ba4ad9c22240696ad69ad51a8111fed9c43d7c"
Nov 01 04:04:21.396920 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:04:21.396904    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d389f98887308f1f25fd47f24ba4ad9c22240696ad69ad51a8111fed9c43d7c\": container with ID starting with 9d389f98887308f1f25fd47f24ba4ad9c22240696ad69ad51a8111fed9c43d7c not found: ID does not exist" containerID="9d389f98887308f1f25fd47f24ba4ad9c22240696ad69ad51a8111fed9c43d7c"
Nov 01 04:04:21.396980 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:21.396927    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d389f98887308f1f25fd47f24ba4ad9c22240696ad69ad51a8111fed9c43d7c"} err="failed to get container status \"9d389f98887308f1f25fd47f24ba4ad9c22240696ad69ad51a8111fed9c43d7c\": rpc error: code = NotFound desc = could not find container \"9d389f98887308f1f25fd47f24ba4ad9c22240696ad69ad51a8111fed9c43d7c\": container with ID starting with 9d389f98887308f1f25fd47f24ba4ad9c22240696ad69ad51a8111fed9c43d7c not found: ID does not exist"
Nov 01 04:04:21.550416 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:21.550385    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b4sdg"]
Nov 01 04:04:21.600975 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:21.600944    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b4sdg"]
Nov 01 04:04:21.619050 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:21.619025    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4edaa96-e7f5-480f-9c85-b7189aedb11c" path="/var/lib/kubelet/pods/e4edaa96-e7f5-480f-9c85-b7189aedb11c/volumes"
Nov 01 04:04:31.048711 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:31.048675    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rk84h"]
Nov 01 04:04:31.098057 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:31.098025    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rk84h"]
Nov 01 04:04:31.618336 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:31.618300    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eafdd6e0-4d42-4ba7-b342-df7fd1179a5e" path="/var/lib/kubelet/pods/eafdd6e0-4d42-4ba7-b342-df7fd1179a5e/volumes"
Nov 01 04:04:41.002654 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:41.002614    4930 scope.go:122] "RemoveContainer" containerID="bfeb771d42d44a92baa66a308928ca88cc271839100df4d07de616ca1106f547"
Nov 01 04:04:41.011683 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:41.011660    4930 scope.go:122] "RemoveContainer" containerID="2651565c673d69df216db29fc376953be126d4b1e047726fe2ccbade991b0ed7"
Nov 01 04:04:46.589552 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:46.589515    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:04:49.120976 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:04:49.120939    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 04:04:56.425385 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:04:56.425346    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:05:19.120051 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:05:19.120007    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 04:05:23.439202 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:23.439164    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:05:24.178295 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:24.178254    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-d4pzp"]
Nov 01 04:05:24.178746 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:24.178729    4930 cpu_manager.go:414] "containerMap: removing container" podUID="b722df12-81b7-46a9-8fe3-0047132bf1be" containerName="payload-pre-processing"
Nov 01 04:05:24.178809 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:24.178748    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b722df12-81b7-46a9-8fe3-0047132bf1be" containerName="payload-pre-processing"
Nov 01 04:05:24.178809 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:24.178760    4930 cpu_manager.go:414] "containerMap: removing container" podUID="01a0ed6d-e285-4062-9baf-fd36504d1f43" containerName="istio-proxy"
Nov 01 04:05:24.178809 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:24.178765    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01a0ed6d-e285-4062-9baf-fd36504d1f43" containerName="istio-proxy"
Nov 01 04:05:24.178809 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:24.178780    4930 cpu_manager.go:414] "containerMap: removing container" podUID="01a0ed6d-e285-4062-9baf-fd36504d1f43" containerName="istio-proxy"
Nov 01 04:05:24.178809 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:24.178785    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01a0ed6d-e285-4062-9baf-fd36504d1f43" containerName="istio-proxy"
Nov 01 04:05:24.178809 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:24.178794    4930 cpu_manager.go:414] "containerMap: removing container" podUID="e4edaa96-e7f5-480f-9c85-b7189aedb11c" containerName="manager"
Nov 01 04:05:24.178809 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:24.178799    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4edaa96-e7f5-480f-9c85-b7189aedb11c" containerName="manager"
Nov 01 04:05:24.178809 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:24.178804    4930 cpu_manager.go:414] "containerMap: removing container" podUID="d4aaf866-5c93-48e1-b893-8bac00fd35ab" containerName="revoke-keys"
Nov 01 04:05:24.178809 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:24.178811    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4aaf866-5c93-48e1-b893-8bac00fd35ab" containerName="revoke-keys"
Nov 01 04:05:24.179099 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:24.178835    4930 cpu_manager.go:414] "containerMap: removing container" podUID="a4eaeaed-f66c-4719-adf2-759a555e07e6" containerName="payload-processing"
Nov 01 04:05:24.179099 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:24.178840    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4eaeaed-f66c-4719-adf2-759a555e07e6" containerName="payload-processing"
Nov 01 04:05:24.179099 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:24.178923    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4eaeaed-f66c-4719-adf2-759a555e07e6" containerName="payload-processing"
Nov 01 04:05:24.179099 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:24.178936    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="b722df12-81b7-46a9-8fe3-0047132bf1be" containerName="payload-pre-processing"
Nov 01 04:05:24.179099 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:24.178943    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="01a0ed6d-e285-4062-9baf-fd36504d1f43" containerName="istio-proxy"
Nov 01 04:05:24.179099 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:24.178949    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="01a0ed6d-e285-4062-9baf-fd36504d1f43" containerName="istio-proxy"
Nov 01 04:05:24.179099 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:24.178954    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4edaa96-e7f5-480f-9c85-b7189aedb11c" containerName="manager"
Nov 01 04:05:24.179099 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:24.178960    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4aaf866-5c93-48e1-b893-8bac00fd35ab" containerName="revoke-keys"
Nov 01 04:05:24.182254 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:24.182239    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-d4pzp"
Nov 01 04:05:24.194093 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:24.194065    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f32a73e-8d68-4f8a-bae1-a85c34d4fd80-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-d4pzp\" (UID: \"1f32a73e-8d68-4f8a-bae1-a85c34d4fd80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-d4pzp"
Nov 01 04:05:24.194246 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:24.194122    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdhxr\" (UniqueName: \"kubernetes.io/projected/1f32a73e-8d68-4f8a-bae1-a85c34d4fd80-kube-api-access-zdhxr\") pod \"kuadrant-operator-controller-manager-678d87766f-d4pzp\" (UID: \"1f32a73e-8d68-4f8a-bae1-a85c34d4fd80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-d4pzp"
Nov 01 04:05:24.237730 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:24.237700    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-d4pzp"]
Nov 01 04:05:24.284322 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:24.284297    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-mvk79\""
Nov 01 04:05:24.294985 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:24.294950    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f32a73e-8d68-4f8a-bae1-a85c34d4fd80-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-d4pzp\" (UID: \"1f32a73e-8d68-4f8a-bae1-a85c34d4fd80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-d4pzp"
Nov 01 04:05:24.295093 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:24.295003    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdhxr\" (UniqueName: \"kubernetes.io/projected/1f32a73e-8d68-4f8a-bae1-a85c34d4fd80-kube-api-access-zdhxr\") pod \"kuadrant-operator-controller-manager-678d87766f-d4pzp\" (UID: \"1f32a73e-8d68-4f8a-bae1-a85c34d4fd80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-d4pzp"
Nov 01 04:05:24.295339 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:24.295315    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f32a73e-8d68-4f8a-bae1-a85c34d4fd80-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-d4pzp\" (UID: \"1f32a73e-8d68-4f8a-bae1-a85c34d4fd80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-d4pzp"
Nov 01 04:05:24.359840 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:24.359805    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdhxr\" (UniqueName: \"kubernetes.io/projected/1f32a73e-8d68-4f8a-bae1-a85c34d4fd80-kube-api-access-zdhxr\") pod \"kuadrant-operator-controller-manager-678d87766f-d4pzp\" (UID: \"1f32a73e-8d68-4f8a-bae1-a85c34d4fd80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-d4pzp"
Nov 01 04:05:24.495938 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:24.495861    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-d4pzp"
Nov 01 04:05:24.972575 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:24.972533    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-d4pzp"]
Nov 01 04:05:24.974722 ip-10-0-141-122 kubenswrapper[4930]: W1101 04:05:24.974694    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f32a73e_8d68_4f8a_bae1_a85c34d4fd80.slice/crio-ca88b91b0e70c7a3bc091fc3fe1458acc20e6b751b5874bdc55c5c59ea55e107 WatchSource:0}: Error finding container ca88b91b0e70c7a3bc091fc3fe1458acc20e6b751b5874bdc55c5c59ea55e107: Status 404 returned error can't find the container with id ca88b91b0e70c7a3bc091fc3fe1458acc20e6b751b5874bdc55c5c59ea55e107
Nov 01 04:05:25.735002 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:25.734968    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-d4pzp" event={"ID":"1f32a73e-8d68-4f8a-bae1-a85c34d4fd80","Type":"ContainerStarted","Data":"6f4bbec2c4100186a05e420035177877d3b081a9160a21df9f443169ebdc3596"}
Nov 01 04:05:25.735434 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:25.735032    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-d4pzp" event={"ID":"1f32a73e-8d68-4f8a-bae1-a85c34d4fd80","Type":"ContainerStarted","Data":"ca88b91b0e70c7a3bc091fc3fe1458acc20e6b751b5874bdc55c5c59ea55e107"}
Nov 01 04:05:25.735434 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:25.735060    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-d4pzp"
Nov 01 04:05:25.886552 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:25.886501    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-d4pzp" podStartSLOduration=1.886487351 podStartE2EDuration="1.886487351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:05:25.884274686 +0000 UTC m=+3530.407281274" watchObservedRunningTime="2026-11-01 04:05:25.886487351 +0000 UTC m=+3530.409493940"
Nov 01 04:05:36.742717 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:36.742685    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-d4pzp"
Nov 01 04:05:41.056627 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:41.056595    4930 scope.go:122] "RemoveContainer" containerID="40a8c0bdeaf311bfb618414fbb11cd32b16bc86af375e51ef17174f0ccd68b89"
Nov 01 04:05:41.065486 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:41.065466    4930 scope.go:122] "RemoveContainer" containerID="50f73c007d717a4dfc69e36db2b354ac8446ec5cf1167452fa7e4fb9036c23b3"
Nov 01 04:05:41.073430 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:41.073409    4930 scope.go:122] "RemoveContainer" containerID="d3c07d87e876284b71748627c25ce2e2a562d4d20b73ba69d93b5f534d2d7455"
Nov 01 04:05:47.053633 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:47.051940    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:05:49.122008 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:05:49.121969    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 04:05:49.655361 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:49.655328    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7zrj9"]
Nov 01 04:05:49.660300 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:49.660276    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7zrj9"
Nov 01 04:05:49.712181 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:49.712149    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 04:05:49.761747 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:49.761716    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:05:49.761747 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:49.761716    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 04:05:49.761978 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:49.761823    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 04:05:49.764210 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:49.764189    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7zrj9"]
Nov 01 04:05:49.791533 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:49.791504    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg6wv\" (UniqueName: \"kubernetes.io/projected/d35568fe-47b1-4f8d-849f-5a85a1b554fe-kube-api-access-dg6wv\") pod \"maas-api-revoke-sub-d576e68b-7zrj9\" (UID: \"d35568fe-47b1-4f8d-849f-5a85a1b554fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7zrj9"
Nov 01 04:05:49.791679 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:49.791641    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d35568fe-47b1-4f8d-849f-5a85a1b554fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-7zrj9\" (UID: \"d35568fe-47b1-4f8d-849f-5a85a1b554fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7zrj9"
Nov 01 04:05:49.791722 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:49.791697    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d35568fe-47b1-4f8d-849f-5a85a1b554fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-7zrj9\" (UID: \"d35568fe-47b1-4f8d-849f-5a85a1b554fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7zrj9"
Nov 01 04:05:49.893014 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:49.892974    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg6wv\" (UniqueName: \"kubernetes.io/projected/d35568fe-47b1-4f8d-849f-5a85a1b554fe-kube-api-access-dg6wv\") pod \"maas-api-revoke-sub-d576e68b-7zrj9\" (UID: \"d35568fe-47b1-4f8d-849f-5a85a1b554fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7zrj9"
Nov 01 04:05:49.893221 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:49.893061    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d35568fe-47b1-4f8d-849f-5a85a1b554fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-7zrj9\" (UID: \"d35568fe-47b1-4f8d-849f-5a85a1b554fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7zrj9"
Nov 01 04:05:49.893221 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:49.893094    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d35568fe-47b1-4f8d-849f-5a85a1b554fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-7zrj9\" (UID: \"d35568fe-47b1-4f8d-849f-5a85a1b554fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7zrj9"
Nov 01 04:05:49.893901 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:49.893873    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d35568fe-47b1-4f8d-849f-5a85a1b554fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-7zrj9\" (UID: \"d35568fe-47b1-4f8d-849f-5a85a1b554fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7zrj9"
Nov 01 04:05:49.895705 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:49.895678    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d35568fe-47b1-4f8d-849f-5a85a1b554fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-7zrj9\" (UID: \"d35568fe-47b1-4f8d-849f-5a85a1b554fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7zrj9"
Nov 01 04:05:49.956102 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:49.956033    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg6wv\" (UniqueName: \"kubernetes.io/projected/d35568fe-47b1-4f8d-849f-5a85a1b554fe-kube-api-access-dg6wv\") pod \"maas-api-revoke-sub-d576e68b-7zrj9\" (UID: \"d35568fe-47b1-4f8d-849f-5a85a1b554fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7zrj9"
Nov 01 04:05:49.972819 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:49.972794    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7zrj9"
Nov 01 04:05:50.350772 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:50.350739    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7zrj9"]
Nov 01 04:05:50.352885 ip-10-0-141-122 kubenswrapper[4930]: W1101 04:05:50.352858    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd35568fe_47b1_4f8d_849f_5a85a1b554fe.slice/crio-0c9af44f15e5d3bf1ce6abc6e69f828c90b1da1725db1daf400d43f6c3d24b53 WatchSource:0}: Error finding container 0c9af44f15e5d3bf1ce6abc6e69f828c90b1da1725db1daf400d43f6c3d24b53: Status 404 returned error can't find the container with id 0c9af44f15e5d3bf1ce6abc6e69f828c90b1da1725db1daf400d43f6c3d24b53
Nov 01 04:05:50.910990 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:50.910953    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7zrj9" event={"ID":"d35568fe-47b1-4f8d-849f-5a85a1b554fe","Type":"ContainerStarted","Data":"b924be943a87f5481cbab24e4532389636f19ddc94c333d5e1bfe6141d54e030"}
Nov 01 04:05:50.911166 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:50.911005    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7zrj9" event={"ID":"d35568fe-47b1-4f8d-849f-5a85a1b554fe","Type":"ContainerStarted","Data":"0c9af44f15e5d3bf1ce6abc6e69f828c90b1da1725db1daf400d43f6c3d24b53"}
Nov 01 04:05:51.026890 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:51.026785    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7zrj9" podStartSLOduration=2.026770958 podStartE2EDuration="2.026770958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:05:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:05:51.02566663 +0000 UTC m=+3555.548673218" watchObservedRunningTime="2026-11-01 04:05:51.026770958 +0000 UTC m=+3555.549777545"
Nov 01 04:05:51.917837 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:51.917801    4930 generic.go:356] "Generic (PLEG): container finished" podID="d35568fe-47b1-4f8d-849f-5a85a1b554fe" containerID="b924be943a87f5481cbab24e4532389636f19ddc94c333d5e1bfe6141d54e030" exitCode=0
Nov 01 04:05:51.918263 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:51.917842    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7zrj9" event={"ID":"d35568fe-47b1-4f8d-849f-5a85a1b554fe","Type":"ContainerDied","Data":"b924be943a87f5481cbab24e4532389636f19ddc94c333d5e1bfe6141d54e030"}
Nov 01 04:05:53.068384 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:53.068358    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7zrj9"
Nov 01 04:05:53.124027 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:53.123998    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d35568fe-47b1-4f8d-849f-5a85a1b554fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d35568fe-47b1-4f8d-849f-5a85a1b554fe-maas-api-openshift-ca\") pod \"d35568fe-47b1-4f8d-849f-5a85a1b554fe\" (UID: \"d35568fe-47b1-4f8d-849f-5a85a1b554fe\") "
Nov 01 04:05:53.124197 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:53.124090    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d35568fe-47b1-4f8d-849f-5a85a1b554fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d35568fe-47b1-4f8d-849f-5a85a1b554fe-maas-api-xks-ca\") pod \"d35568fe-47b1-4f8d-849f-5a85a1b554fe\" (UID: \"d35568fe-47b1-4f8d-849f-5a85a1b554fe\") "
Nov 01 04:05:53.124197 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:53.124186    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d35568fe-47b1-4f8d-849f-5a85a1b554fe-kube-api-access-dg6wv\" (UniqueName: \"kubernetes.io/projected/d35568fe-47b1-4f8d-849f-5a85a1b554fe-kube-api-access-dg6wv\") pod \"d35568fe-47b1-4f8d-849f-5a85a1b554fe\" (UID: \"d35568fe-47b1-4f8d-849f-5a85a1b554fe\") "
Nov 01 04:05:53.124441 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:53.124420    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d35568fe-47b1-4f8d-849f-5a85a1b554fe-maas-api-openshift-ca" pod "d35568fe-47b1-4f8d-849f-5a85a1b554fe" (UID: "d35568fe-47b1-4f8d-849f-5a85a1b554fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:05:53.124622 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:53.124603    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d35568fe-47b1-4f8d-849f-5a85a1b554fe-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:05:53.126163 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:53.126139    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d35568fe-47b1-4f8d-849f-5a85a1b554fe-maas-api-xks-ca" pod "d35568fe-47b1-4f8d-849f-5a85a1b554fe" (UID: "d35568fe-47b1-4f8d-849f-5a85a1b554fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:05:53.126271 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:53.126191    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d35568fe-47b1-4f8d-849f-5a85a1b554fe-kube-api-access-dg6wv" pod "d35568fe-47b1-4f8d-849f-5a85a1b554fe" (UID: "d35568fe-47b1-4f8d-849f-5a85a1b554fe"). InnerVolumeSpecName "kube-api-access-dg6wv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:05:53.225992 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:53.225907    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d35568fe-47b1-4f8d-849f-5a85a1b554fe-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:05:53.225992 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:53.225936    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg6wv\" (UniqueName: \"kubernetes.io/projected/d35568fe-47b1-4f8d-849f-5a85a1b554fe-kube-api-access-dg6wv\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:05:53.929476 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:53.929391    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7zrj9"
Nov 01 04:05:53.929476 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:53.929419    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7zrj9" event={"ID":"d35568fe-47b1-4f8d-849f-5a85a1b554fe","Type":"ContainerDied","Data":"0c9af44f15e5d3bf1ce6abc6e69f828c90b1da1725db1daf400d43f6c3d24b53"}
Nov 01 04:05:53.929476 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:53.929464    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c9af44f15e5d3bf1ce6abc6e69f828c90b1da1725db1daf400d43f6c3d24b53"
Nov 01 04:05:56.758095 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:05:56.758057    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:06:04.061176 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:04.061142    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nlssn"]
Nov 01 04:06:04.109742 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:04.109707    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-nlssn"]
Nov 01 04:06:05.443913 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:05.443873    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:06:05.619578 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:05.619523    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="702790ea-5079-4431-ba0c-674bf255859a" path="/var/lib/kubelet/pods/702790ea-5079-4431-ba0c-674bf255859a/volumes"
Nov 01 04:06:14.051404 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:14.051363    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dw724"]
Nov 01 04:06:14.099990 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:14.099956    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-dw724"]
Nov 01 04:06:15.619527 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:15.619490    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47773233-705b-48a6-b893-315fe79f75df" path="/var/lib/kubelet/pods/47773233-705b-48a6-b893-315fe79f75df/volumes"
Nov 01 04:06:18.723167 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:18.723132    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-psfgj"]
Nov 01 04:06:18.723628 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:18.723613    4930 cpu_manager.go:414] "containerMap: removing container" podUID="d35568fe-47b1-4f8d-849f-5a85a1b554fe" containerName="revoke-keys"
Nov 01 04:06:18.723688 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:18.723630    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d35568fe-47b1-4f8d-849f-5a85a1b554fe" containerName="revoke-keys"
Nov 01 04:06:18.723736 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:18.723727    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="d35568fe-47b1-4f8d-849f-5a85a1b554fe" containerName="revoke-keys"
Nov 01 04:06:18.726969 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:18.726952    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-psfgj"
Nov 01 04:06:18.779370 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:18.779339    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 04:06:18.812066 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:18.812038    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tc9b\" (UniqueName: \"kubernetes.io/projected/0667bc47-e82f-42f6-bd76-8f942e668f06-kube-api-access-6tc9b\") pod \"maas-api-revoke-sub-d2bc93bc-psfgj\" (UID: \"0667bc47-e82f-42f6-bd76-8f942e668f06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-psfgj"
Nov 01 04:06:18.812234 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:18.812087    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0667bc47-e82f-42f6-bd76-8f942e668f06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-psfgj\" (UID: \"0667bc47-e82f-42f6-bd76-8f942e668f06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-psfgj"
Nov 01 04:06:18.812234 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:18.812182    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0667bc47-e82f-42f6-bd76-8f942e668f06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-psfgj\" (UID: \"0667bc47-e82f-42f6-bd76-8f942e668f06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-psfgj"
Nov 01 04:06:18.814212 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:18.814188    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 04:06:18.829008 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:18.828978    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:06:18.829167 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:18.829153    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 04:06:18.831475 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:18.831454    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-psfgj"]
Nov 01 04:06:18.912946 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:18.912920    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0667bc47-e82f-42f6-bd76-8f942e668f06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-psfgj\" (UID: \"0667bc47-e82f-42f6-bd76-8f942e668f06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-psfgj"
Nov 01 04:06:18.913084 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:18.912985    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tc9b\" (UniqueName: \"kubernetes.io/projected/0667bc47-e82f-42f6-bd76-8f942e668f06-kube-api-access-6tc9b\") pod \"maas-api-revoke-sub-d2bc93bc-psfgj\" (UID: \"0667bc47-e82f-42f6-bd76-8f942e668f06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-psfgj"
Nov 01 04:06:18.913084 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:18.913030    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0667bc47-e82f-42f6-bd76-8f942e668f06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-psfgj\" (UID: \"0667bc47-e82f-42f6-bd76-8f942e668f06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-psfgj"
Nov 01 04:06:18.913549 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:18.913530    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0667bc47-e82f-42f6-bd76-8f942e668f06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-psfgj\" (UID: \"0667bc47-e82f-42f6-bd76-8f942e668f06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-psfgj"
Nov 01 04:06:18.915389 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:18.915365    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0667bc47-e82f-42f6-bd76-8f942e668f06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-psfgj\" (UID: \"0667bc47-e82f-42f6-bd76-8f942e668f06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-psfgj"
Nov 01 04:06:18.973479 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:18.973421    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tc9b\" (UniqueName: \"kubernetes.io/projected/0667bc47-e82f-42f6-bd76-8f942e668f06-kube-api-access-6tc9b\") pod \"maas-api-revoke-sub-d2bc93bc-psfgj\" (UID: \"0667bc47-e82f-42f6-bd76-8f942e668f06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-psfgj"
Nov 01 04:06:19.040212 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:19.040186    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-psfgj"
Nov 01 04:06:19.132709 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:06:19.132678    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 04:06:19.415101 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:19.415074    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-psfgj"]
Nov 01 04:06:19.416405 ip-10-0-141-122 kubenswrapper[4930]: W1101 04:06:19.416375    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0667bc47_e82f_42f6_bd76_8f942e668f06.slice/crio-42d80e2b82ebbabfc9281c64b84ab6fd18015c6d10cce2c718cd237696a8e009 WatchSource:0}: Error finding container 42d80e2b82ebbabfc9281c64b84ab6fd18015c6d10cce2c718cd237696a8e009: Status 404 returned error can't find the container with id 42d80e2b82ebbabfc9281c64b84ab6fd18015c6d10cce2c718cd237696a8e009
Nov 01 04:06:20.118182 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:20.118144    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-psfgj" event={"ID":"0667bc47-e82f-42f6-bd76-8f942e668f06","Type":"ContainerStarted","Data":"35cfee4777a6a1cdd84299451097e79bdeada1cf6d7c838523000edd77afaa0a"}
Nov 01 04:06:20.118644 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:20.118210    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-psfgj" event={"ID":"0667bc47-e82f-42f6-bd76-8f942e668f06","Type":"ContainerStarted","Data":"42d80e2b82ebbabfc9281c64b84ab6fd18015c6d10cce2c718cd237696a8e009"}
Nov 01 04:06:20.237506 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:20.237458    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-psfgj" podStartSLOduration=2.237444499 podStartE2EDuration="2.237444499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:06:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:06:20.234481242 +0000 UTC m=+3584.757487829" watchObservedRunningTime="2026-11-01 04:06:20.237444499 +0000 UTC m=+3584.760451086"
Nov 01 04:06:21.125112 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:21.125084    4930 generic.go:356] "Generic (PLEG): container finished" podID="0667bc47-e82f-42f6-bd76-8f942e668f06" containerID="35cfee4777a6a1cdd84299451097e79bdeada1cf6d7c838523000edd77afaa0a" exitCode=0
Nov 01 04:06:21.125512 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:21.125172    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-psfgj" event={"ID":"0667bc47-e82f-42f6-bd76-8f942e668f06","Type":"ContainerDied","Data":"35cfee4777a6a1cdd84299451097e79bdeada1cf6d7c838523000edd77afaa0a"}
Nov 01 04:06:22.270419 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:22.270396    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-psfgj"
Nov 01 04:06:22.343912 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:22.343889    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0667bc47-e82f-42f6-bd76-8f942e668f06-kube-api-access-6tc9b\" (UniqueName: \"kubernetes.io/projected/0667bc47-e82f-42f6-bd76-8f942e668f06-kube-api-access-6tc9b\") pod \"0667bc47-e82f-42f6-bd76-8f942e668f06\" (UID: \"0667bc47-e82f-42f6-bd76-8f942e668f06\") "
Nov 01 04:06:22.344084 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:22.343955    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0667bc47-e82f-42f6-bd76-8f942e668f06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0667bc47-e82f-42f6-bd76-8f942e668f06-maas-api-xks-ca\") pod \"0667bc47-e82f-42f6-bd76-8f942e668f06\" (UID: \"0667bc47-e82f-42f6-bd76-8f942e668f06\") "
Nov 01 04:06:22.344084 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:22.343996    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0667bc47-e82f-42f6-bd76-8f942e668f06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0667bc47-e82f-42f6-bd76-8f942e668f06-maas-api-openshift-ca\") pod \"0667bc47-e82f-42f6-bd76-8f942e668f06\" (UID: \"0667bc47-e82f-42f6-bd76-8f942e668f06\") "
Nov 01 04:06:22.344384 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:22.344360    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0667bc47-e82f-42f6-bd76-8f942e668f06-maas-api-openshift-ca" pod "0667bc47-e82f-42f6-bd76-8f942e668f06" (UID: "0667bc47-e82f-42f6-bd76-8f942e668f06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:06:22.345941 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:22.345913    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0667bc47-e82f-42f6-bd76-8f942e668f06-kube-api-access-6tc9b" pod "0667bc47-e82f-42f6-bd76-8f942e668f06" (UID: "0667bc47-e82f-42f6-bd76-8f942e668f06"). InnerVolumeSpecName "kube-api-access-6tc9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:06:22.346041 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:22.345958    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0667bc47-e82f-42f6-bd76-8f942e668f06-maas-api-xks-ca" pod "0667bc47-e82f-42f6-bd76-8f942e668f06" (UID: "0667bc47-e82f-42f6-bd76-8f942e668f06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:06:22.444767 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:22.444687    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0667bc47-e82f-42f6-bd76-8f942e668f06-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:06:22.444767 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:22.444714    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tc9b\" (UniqueName: \"kubernetes.io/projected/0667bc47-e82f-42f6-bd76-8f942e668f06-kube-api-access-6tc9b\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:06:22.444767 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:22.444725    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0667bc47-e82f-42f6-bd76-8f942e668f06-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:06:23.137193 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:23.137159    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-psfgj"
Nov 01 04:06:23.137374 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:23.137186    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-psfgj" event={"ID":"0667bc47-e82f-42f6-bd76-8f942e668f06","Type":"ContainerDied","Data":"42d80e2b82ebbabfc9281c64b84ab6fd18015c6d10cce2c718cd237696a8e009"}
Nov 01 04:06:23.137374 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:23.137240    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42d80e2b82ebbabfc9281c64b84ab6fd18015c6d10cce2c718cd237696a8e009"
Nov 01 04:06:31.156855 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:31.156822    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:06:32.003225 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:32.003188    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"]
Nov 01 04:06:32.003637 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:32.003622    4930 cpu_manager.go:414] "containerMap: removing container" podUID="0667bc47-e82f-42f6-bd76-8f942e668f06" containerName="revoke-keys"
Nov 01 04:06:32.003693 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:32.003638    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0667bc47-e82f-42f6-bd76-8f942e668f06" containerName="revoke-keys"
Nov 01 04:06:32.003739 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:32.003730    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="0667bc47-e82f-42f6-bd76-8f942e668f06" containerName="revoke-keys"
Nov 01 04:06:32.007060 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:32.007040    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"
Nov 01 04:06:32.074319 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:32.074275    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-077ac6-openshift-default-dockercfg-64m5k\""
Nov 01 04:06:32.088357 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:32.088326    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:06:32.112911 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:32.112884    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"]
Nov 01 04:06:32.134043 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:32.134020    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee888f27-32db-468c-a2df-7a5bceb25246-istio-token\") pod \"e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h\" (UID: \"ee888f27-32db-468c-a2df-7a5bceb25246\") " pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"
Nov 01 04:06:32.134208 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:32.134052    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee888f27-32db-468c-a2df-7a5bceb25246-istio-podinfo\") pod \"e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h\" (UID: \"ee888f27-32db-468c-a2df-7a5bceb25246\") " pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"
Nov 01 04:06:32.134208 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:32.134092    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee888f27-32db-468c-a2df-7a5bceb25246-istio-envoy\") pod \"e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h\" (UID: \"ee888f27-32db-468c-a2df-7a5bceb25246\") " pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"
Nov 01 04:06:32.134208 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:32.134160    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee888f27-32db-468c-a2df-7a5bceb25246-istio-data\") pod \"e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h\" (UID: \"ee888f27-32db-468c-a2df-7a5bceb25246\") " pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"
Nov 01 04:06:32.134358 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:32.134302    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee888f27-32db-468c-a2df-7a5bceb25246-credential-socket\") pod \"e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h\" (UID: \"ee888f27-32db-468c-a2df-7a5bceb25246\") " pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"
Nov 01 04:06:32.134406 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:32.134376    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee888f27-32db-468c-a2df-7a5bceb25246-workload-socket\") pod \"e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h\" (UID: \"ee888f27-32db-468c-a2df-7a5bceb25246\") " pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"
Nov 01 04:06:32.134406 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:32.134396    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee888f27-32db-468c-a2df-7a5bceb25246-workload-certs\") pod \"e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h\" (UID: \"ee888f27-32db-468c-a2df-7a5bceb25246\") " pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"
Nov 01 04:06:32.134501 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:32.134419    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqhkl\" (UniqueName: \"kubernetes.io/projected/ee888f27-32db-468c-a2df-7a5bceb25246-kube-api-access-zqhkl\") pod \"e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h\" (UID: \"ee888f27-32db-468c-a2df-7a5bceb25246\") " pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"
Nov 01 04:06:32.134501 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:32.134440    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee888f27-32db-468c-a2df-7a5bceb25246-istiod-ca-cert\") pod \"e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h\" (UID: \"ee888f27-32db-468c-a2df-7a5bceb25246\") " pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"
Nov 01 04:06:32.235403 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:32.235362    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee888f27-32db-468c-a2df-7a5bceb25246-istio-token\") pod \"e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h\" (UID: \"ee888f27-32db-468c-a2df-7a5bceb25246\") " pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"
Nov 01 04:06:32.235403 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:32.235401    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee888f27-32db-468c-a2df-7a5bceb25246-istio-podinfo\") pod \"e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h\" (UID: \"ee888f27-32db-468c-a2df-7a5bceb25246\") " pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"
Nov 01 04:06:32.235962 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:32.235446    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee888f27-32db-468c-a2df-7a5bceb25246-istio-envoy\") pod \"e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h\" (UID: \"ee888f27-32db-468c-a2df-7a5bceb25246\") " pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"
Nov 01 04:06:32.235962 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:32.235477    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee888f27-32db-468c-a2df-7a5bceb25246-istio-data\") pod \"e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h\" (UID: \"ee888f27-32db-468c-a2df-7a5bceb25246\") " pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"
Nov 01 04:06:32.235962 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:32.235524    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee888f27-32db-468c-a2df-7a5bceb25246-credential-socket\") pod \"e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h\" (UID: \"ee888f27-32db-468c-a2df-7a5bceb25246\") " pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"
Nov 01 04:06:32.235962 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:32.235582    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee888f27-32db-468c-a2df-7a5bceb25246-workload-socket\") pod \"e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h\" (UID: \"ee888f27-32db-468c-a2df-7a5bceb25246\") " pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"
Nov 01 04:06:32.235962 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:32.235607    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee888f27-32db-468c-a2df-7a5bceb25246-workload-certs\") pod \"e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h\" (UID: \"ee888f27-32db-468c-a2df-7a5bceb25246\") " pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"
Nov 01 04:06:32.235962 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:32.235639    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqhkl\" (UniqueName: \"kubernetes.io/projected/ee888f27-32db-468c-a2df-7a5bceb25246-kube-api-access-zqhkl\") pod \"e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h\" (UID: \"ee888f27-32db-468c-a2df-7a5bceb25246\") " pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"
Nov 01 04:06:32.235962 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:32.235680    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee888f27-32db-468c-a2df-7a5bceb25246-istiod-ca-cert\") pod \"e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h\" (UID: \"ee888f27-32db-468c-a2df-7a5bceb25246\") " pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"
Nov 01 04:06:32.236341 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:32.236050    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee888f27-32db-468c-a2df-7a5bceb25246-istio-data\") pod \"e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h\" (UID: \"ee888f27-32db-468c-a2df-7a5bceb25246\") " pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"
Nov 01 04:06:32.236341 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:32.236291    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee888f27-32db-468c-a2df-7a5bceb25246-credential-socket\") pod \"e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h\" (UID: \"ee888f27-32db-468c-a2df-7a5bceb25246\") " pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"
Nov 01 04:06:32.236341 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:32.236329    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee888f27-32db-468c-a2df-7a5bceb25246-workload-socket\") pod \"e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h\" (UID: \"ee888f27-32db-468c-a2df-7a5bceb25246\") " pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"
Nov 01 04:06:32.236473 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:32.236429    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee888f27-32db-468c-a2df-7a5bceb25246-workload-certs\") pod \"e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h\" (UID: \"ee888f27-32db-468c-a2df-7a5bceb25246\") " pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"
Nov 01 04:06:32.236673 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:32.236655    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee888f27-32db-468c-a2df-7a5bceb25246-istiod-ca-cert\") pod \"e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h\" (UID: \"ee888f27-32db-468c-a2df-7a5bceb25246\") " pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"
Nov 01 04:06:32.237868 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:32.237843    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee888f27-32db-468c-a2df-7a5bceb25246-istio-envoy\") pod \"e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h\" (UID: \"ee888f27-32db-468c-a2df-7a5bceb25246\") " pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"
Nov 01 04:06:32.238081 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:32.238066    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee888f27-32db-468c-a2df-7a5bceb25246-istio-podinfo\") pod \"e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h\" (UID: \"ee888f27-32db-468c-a2df-7a5bceb25246\") " pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"
Nov 01 04:06:32.298543 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:32.298484    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqhkl\" (UniqueName: \"kubernetes.io/projected/ee888f27-32db-468c-a2df-7a5bceb25246-kube-api-access-zqhkl\") pod \"e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h\" (UID: \"ee888f27-32db-468c-a2df-7a5bceb25246\") " pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"
Nov 01 04:06:32.298543 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:32.298511    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee888f27-32db-468c-a2df-7a5bceb25246-istio-token\") pod \"e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h\" (UID: \"ee888f27-32db-468c-a2df-7a5bceb25246\") " pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"
Nov 01 04:06:32.320164 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:32.320141    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"
Nov 01 04:06:32.424289 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:32.424259    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:06:32.760005 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:32.759978    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"]
Nov 01 04:06:32.761791 ip-10-0-141-122 kubenswrapper[4930]: W1101 04:06:32.761757    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee888f27_32db_468c_a2df_7a5bceb25246.slice/crio-ea32b25998bc1e2db843a4a268f38aa31496149db77ed8d62eb009a130a3841f WatchSource:0}: Error finding container ea32b25998bc1e2db843a4a268f38aa31496149db77ed8d62eb009a130a3841f: Status 404 returned error can't find the container with id ea32b25998bc1e2db843a4a268f38aa31496149db77ed8d62eb009a130a3841f
Nov 01 04:06:32.764025 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:32.763991    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:06:32.764151 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:32.764072    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:06:32.764151 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:32.764119    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:06:33.192490 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:33.192453    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h" event={"ID":"ee888f27-32db-468c-a2df-7a5bceb25246","Type":"ContainerStarted","Data":"1df684fd4d0a2d0e0f8a9d88b6815f74a4fae0b4bd0e919602e278f4ca1bc921"}
Nov 01 04:06:33.192677 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:33.192508    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h" event={"ID":"ee888f27-32db-468c-a2df-7a5bceb25246","Type":"ContainerStarted","Data":"ea32b25998bc1e2db843a4a268f38aa31496149db77ed8d62eb009a130a3841f"}
Nov 01 04:06:33.314629 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:33.314550    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h" podStartSLOduration=2.314535495 podStartE2EDuration="2.314535495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:06:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:06:33.311319458 +0000 UTC m=+3597.834326047" watchObservedRunningTime="2026-11-01 04:06:33.314535495 +0000 UTC m=+3597.837542083"
Nov 01 04:06:33.321188 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:33.321159    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"
Nov 01 04:06:33.327453 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:33.327429    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"
Nov 01 04:06:34.197828 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:34.197795    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"
Nov 01 04:06:34.198736 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:34.198714    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"
Nov 01 04:06:36.258847 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:36.258812    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:06:36.461995 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:36.461958    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:06:36.613164 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:36.613130    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:06:38.421288 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:38.421258    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jh955_f29ea61e-2a78-4c11-9566-f52497e135ff/console-operator/2.log"
Nov 01 04:06:38.472482 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:38.472455    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jh955_f29ea61e-2a78-4c11-9566-f52497e135ff/console-operator/2.log"
Nov 01 04:06:39.513997 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:39.513961    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:06:39.705614 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:39.705547    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:06:40.094250 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:40.094202    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-077ac6-67bf4cf86j4nnw"]
Nov 01 04:06:40.101595 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:40.099063    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-077ac6-67bf4cf86j4nnw"
Nov 01 04:06:40.148670 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:40.148638    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-077ac6-67bf4cf86j4nnw"]
Nov 01 04:06:40.195219 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:40.195195    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-077ac6\""
Nov 01 04:06:40.195391 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:40.195294    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-999d8449-dockercfg-lcpjg\""
Nov 01 04:06:40.214258 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:40.214234    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e29d20b-433b-4fa1-bd90-707e83a245f2-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-077ac6-67bf4cf86j4nnw\" (UID: \"7e29d20b-433b-4fa1-bd90-707e83a245f2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-077ac6-67bf4cf86j4nnw"
Nov 01 04:06:40.214401 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:40.214270    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9x5t\" (UniqueName: \"kubernetes.io/projected/7e29d20b-433b-4fa1-bd90-707e83a245f2-kube-api-access-x9x5t\") pod \"payload-processing-e2e-worker-single-main-077ac6-67bf4cf86j4nnw\" (UID: \"7e29d20b-433b-4fa1-bd90-707e83a245f2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-077ac6-67bf4cf86j4nnw"
Nov 01 04:06:40.256871 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:40.256840    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-077ac6-7cd74zxv4f"]
Nov 01 04:06:40.260626 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:40.260606    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-077ac6-7cd74zxv4f"
Nov 01 04:06:40.315752 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:40.315723    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e29d20b-433b-4fa1-bd90-707e83a245f2-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-077ac6-67bf4cf86j4nnw\" (UID: \"7e29d20b-433b-4fa1-bd90-707e83a245f2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-077ac6-67bf4cf86j4nnw"
Nov 01 04:06:40.315891 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:40.315765    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9x5t\" (UniqueName: \"kubernetes.io/projected/7e29d20b-433b-4fa1-bd90-707e83a245f2-kube-api-access-x9x5t\") pod \"payload-processing-e2e-worker-single-main-077ac6-67bf4cf86j4nnw\" (UID: \"7e29d20b-433b-4fa1-bd90-707e83a245f2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-077ac6-67bf4cf86j4nnw"
Nov 01 04:06:40.316692 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:40.316670    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e29d20b-433b-4fa1-bd90-707e83a245f2-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-077ac6-67bf4cf86j4nnw\" (UID: \"7e29d20b-433b-4fa1-bd90-707e83a245f2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-077ac6-67bf4cf86j4nnw"
Nov 01 04:06:40.318984 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:40.318962    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-077ac6-7cd74zxv4f"]
Nov 01 04:06:40.376650 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:40.376575    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9x5t\" (UniqueName: \"kubernetes.io/projected/7e29d20b-433b-4fa1-bd90-707e83a245f2-kube-api-access-x9x5t\") pod \"payload-processing-e2e-worker-single-main-077ac6-67bf4cf86j4nnw\" (UID: \"7e29d20b-433b-4fa1-bd90-707e83a245f2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-077ac6-67bf4cf86j4nnw"
Nov 01 04:06:40.416805 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:40.416779    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8d38fbc-ffe4-46b8-b273-9cc2d8838550-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-077ac6-7cd74zxv4f\" (UID: \"c8d38fbc-ffe4-46b8-b273-9cc2d8838550\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-077ac6-7cd74zxv4f"
Nov 01 04:06:40.416950 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:40.416824    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h22xg\" (UniqueName: \"kubernetes.io/projected/c8d38fbc-ffe4-46b8-b273-9cc2d8838550-kube-api-access-h22xg\") pod \"payload-pre-processing-e2e-worker-single-main-077ac6-7cd74zxv4f\" (UID: \"c8d38fbc-ffe4-46b8-b273-9cc2d8838550\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-077ac6-7cd74zxv4f"
Nov 01 04:06:40.419581 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:40.419567    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-077ac6-67bf4cf86j4nnw"
Nov 01 04:06:40.518268 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:40.518238    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8d38fbc-ffe4-46b8-b273-9cc2d8838550-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-077ac6-7cd74zxv4f\" (UID: \"c8d38fbc-ffe4-46b8-b273-9cc2d8838550\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-077ac6-7cd74zxv4f"
Nov 01 04:06:40.518752 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:40.518723    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h22xg\" (UniqueName: \"kubernetes.io/projected/c8d38fbc-ffe4-46b8-b273-9cc2d8838550-kube-api-access-h22xg\") pod \"payload-pre-processing-e2e-worker-single-main-077ac6-7cd74zxv4f\" (UID: \"c8d38fbc-ffe4-46b8-b273-9cc2d8838550\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-077ac6-7cd74zxv4f"
Nov 01 04:06:40.520074 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:40.520054    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8d38fbc-ffe4-46b8-b273-9cc2d8838550-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-077ac6-7cd74zxv4f\" (UID: \"c8d38fbc-ffe4-46b8-b273-9cc2d8838550\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-077ac6-7cd74zxv4f"
Nov 01 04:06:40.579434 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:40.579414    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h22xg\" (UniqueName: \"kubernetes.io/projected/c8d38fbc-ffe4-46b8-b273-9cc2d8838550-kube-api-access-h22xg\") pod \"payload-pre-processing-e2e-worker-single-main-077ac6-7cd74zxv4f\" (UID: \"c8d38fbc-ffe4-46b8-b273-9cc2d8838550\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-077ac6-7cd74zxv4f"
Nov 01 04:06:40.792257 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:40.792231    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-077ac6-67bf4cf86j4nnw"]
Nov 01 04:06:40.794376 ip-10-0-141-122 kubenswrapper[4930]: W1101 04:06:40.794346    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e29d20b_433b_4fa1_bd90_707e83a245f2.slice/crio-17ae723209c30508736274bf314061e3ee6928a22fa00e63e36e45dad3b85b46 WatchSource:0}: Error finding container 17ae723209c30508736274bf314061e3ee6928a22fa00e63e36e45dad3b85b46: Status 404 returned error can't find the container with id 17ae723209c30508736274bf314061e3ee6928a22fa00e63e36e45dad3b85b46
Nov 01 04:06:40.873388 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:40.873367    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-077ac6-7cd74zxv4f"
Nov 01 04:06:41.237299 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:41.237255    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-077ac6-67bf4cf86j4nnw" event={"ID":"7e29d20b-433b-4fa1-bd90-707e83a245f2","Type":"ContainerStarted","Data":"962ffba62f91481e80bc7277b1dd0d371af3445e529f5661b1f2791108f3cffc"}
Nov 01 04:06:41.237472 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:41.237323    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-077ac6-67bf4cf86j4nnw" event={"ID":"7e29d20b-433b-4fa1-bd90-707e83a245f2","Type":"ContainerStarted","Data":"17ae723209c30508736274bf314061e3ee6928a22fa00e63e36e45dad3b85b46"}
Nov 01 04:06:41.237472 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:41.237443    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-077ac6-67bf4cf86j4nnw"
Nov 01 04:06:41.249620 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:41.249596    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-077ac6-7cd74zxv4f"]
Nov 01 04:06:41.251033 ip-10-0-141-122 kubenswrapper[4930]: W1101 04:06:41.251013    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8d38fbc_ffe4_46b8_b273_9cc2d8838550.slice/crio-46a937903505f5610ff2ec9ea4c183c3025f3a3810f5aa26c0a5e731bcaeed06 WatchSource:0}: Error finding container 46a937903505f5610ff2ec9ea4c183c3025f3a3810f5aa26c0a5e731bcaeed06: Status 404 returned error can't find the container with id 46a937903505f5610ff2ec9ea4c183c3025f3a3810f5aa26c0a5e731bcaeed06
Nov 01 04:06:41.355336 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:41.355289    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-077ac6-67bf4cf86j4nnw" podStartSLOduration=1.355277326 podStartE2EDuration="1.355277326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:06:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:06:41.353307949 +0000 UTC m=+3605.876314538" watchObservedRunningTime="2026-11-01 04:06:41.355277326 +0000 UTC m=+3605.878283914"
Nov 01 04:06:42.245744 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:42.245446    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-077ac6-7cd74zxv4f" event={"ID":"c8d38fbc-ffe4-46b8-b273-9cc2d8838550","Type":"ContainerStarted","Data":"6a763464fd1bdd43060d568372f183d03d342f1dbd78cfc93c221d6622a982a0"}
Nov 01 04:06:42.246251 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:42.245948    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-077ac6-7cd74zxv4f" event={"ID":"c8d38fbc-ffe4-46b8-b273-9cc2d8838550","Type":"ContainerStarted","Data":"46a937903505f5610ff2ec9ea4c183c3025f3a3810f5aa26c0a5e731bcaeed06"}
Nov 01 04:06:42.363232 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:42.363169    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-077ac6-7cd74zxv4f" podStartSLOduration=2.363155735 podStartE2EDuration="2.363155735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:06:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:06:42.360354344 +0000 UTC m=+3606.883360933" watchObservedRunningTime="2026-11-01 04:06:42.363155735 +0000 UTC m=+3606.886162323"
Nov 01 04:06:42.451583 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:42.451527    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:06:43.251960 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:43.251921    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-077ac6-7cd74zxv4f"
Nov 01 04:06:43.564510 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:43.564423    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:06:47.372631 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:47.372540    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:06:47.514671 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:47.514634    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:06:49.121106 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:06:49.121065    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 04:06:52.248980 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:52.248949    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-077ac6-67bf4cf86j4nnw"
Nov 01 04:06:52.305337 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:52.305297    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:06:54.285476 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:54.285441    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-077ac6-7cd74zxv4f"
Nov 01 04:06:57.160920 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:57.160886    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:06:57.307173 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:06:57.307128    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:06:59.970292 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:06:59.970264    4930 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee888f27_32db_468c_a2df_7a5bceb25246.slice/crio-1df684fd4d0a2d0e0f8a9d88b6815f74a4fae0b4bd0e919602e278f4ca1bc921.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 04:07:00.352924 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:07:00.352893    4930 generic.go:356] "Generic (PLEG): container finished" podID="ee888f27-32db-468c-a2df-7a5bceb25246" containerID="1df684fd4d0a2d0e0f8a9d88b6815f74a4fae0b4bd0e919602e278f4ca1bc921" exitCode=0
Nov 01 04:07:00.353092 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:07:00.352959    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h" event={"ID":"ee888f27-32db-468c-a2df-7a5bceb25246","Type":"ContainerDied","Data":"1df684fd4d0a2d0e0f8a9d88b6815f74a4fae0b4bd0e919602e278f4ca1bc921"}
Nov 01 04:07:00.353378 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:07:00.353355    4930 scope.go:122] "RemoveContainer" containerID="1df684fd4d0a2d0e0f8a9d88b6815f74a4fae0b4bd0e919602e278f4ca1bc921"
Nov 01 04:07:00.354590 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:07:00.354535    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:07:00.354712 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:07:00.354650    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:07:00.354712 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:07:00.354685    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:07:01.085211 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:07:01.085168    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:07:01.386368 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:07:01.386279    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h" event={"ID":"ee888f27-32db-468c-a2df-7a5bceb25246","Type":"ContainerStarted","Data":"3ebf7d8cc5a843827242cb283c527fd6c754fff4ae9ccd2ae164f489ef92aa14"}
Nov 01 04:07:01.706810 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:07:01.706704    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:07:01.871045 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:07:01.871010    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:07:02.321171 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:07:02.321132    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"
Nov 01 04:07:02.322528 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:07:02.322504    4930 patch_prober.go:29] interesting pod/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body=
Nov 01 04:07:02.322623 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:07:02.322567    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h" podUID="ee888f27-32db-468c-a2df-7a5bceb25246" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused"
Nov 01 04:07:02.415897 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:07:02.415857    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:07:03.321232 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:07:03.321198    4930 patch_prober.go:29] interesting pod/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body=
Nov 01 04:07:03.321662 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:07:03.321256    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h" podUID="ee888f27-32db-468c-a2df-7a5bceb25246" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused"
Nov 01 04:07:04.198049 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:07:04.198011    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"
Nov 01 04:07:04.321322 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:07:04.321287    4930 patch_prober.go:29] interesting pod/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body=
Nov 01 04:07:04.321740 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:07:04.321340    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h" podUID="ee888f27-32db-468c-a2df-7a5bceb25246" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused"
Nov 01 04:07:04.484637 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:07:04.458696    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:07:05.320948 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:07:05.320913    4930 patch_prober.go:29] interesting pod/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body=
Nov 01 04:07:05.321151 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:07:05.320968    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h" podUID="ee888f27-32db-468c-a2df-7a5bceb25246" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused"
Nov 01 04:07:06.321014 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:07:06.320970    4930 patch_prober.go:29] interesting pod/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused" start-of-body=
Nov 01 04:07:06.321580 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:07:06.321022    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h" podUID="ee888f27-32db-468c-a2df-7a5bceb25246" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.30:15021/healthz/ready\": dial tcp 10.133.1.30:15021: connect: connection refused"
Nov 01 04:07:07.274194 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:07:07.274144    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:07:07.328183 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:07:07.328149    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"
Nov 01 04:07:07.329931 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:07:07.329908    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"
Nov 01 04:07:12.072884 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:07:12.072850    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:07:16.886177 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:07:16.886084    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:07:19.121394 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:07:19.121362    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 04:07:21.680867 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:07:21.680820    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:07:26.499188 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:07:26.499152    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:07:31.271521 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:07:31.271486    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:07:36.086211 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:07:36.086175    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:07:40.883902 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:07:40.883860    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:07:41.113315 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:07:41.113282    4930 scope.go:122] "RemoveContainer" containerID="f6a22f209212e5e8613fe5bf34ebe67cf0c179162ab0974877597877c0b92bde"
Nov 01 04:07:41.122439 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:07:41.122416    4930 scope.go:122] "RemoveContainer" containerID="bdb37430a7a465b7c3e70658b908c107ff1d75e864260633e066ff347ca67909"
Nov 01 04:07:41.130124 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:07:41.130101    4930 scope.go:122] "RemoveContainer" containerID="8c9a7071e67dc3715e51f78581a4716f1de5ffe41cd8a4facfc6c370bc746cda"
Nov 01 04:07:45.679847 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:07:45.679803    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:07:47.710360 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:07:47.710327    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:07:49.122462 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:07:49.122418    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 04:07:50.480055 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:07:50.480007    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:07:55.363883 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:07:55.363846    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:07:57.448128 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:07:57.448088    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:08:00.183213 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:00.183181    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:08:00.950960 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:00.950924    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:08:01.502391 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:01.502354    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-077ac6-frdbj"]
Nov 01 04:08:01.507701 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:01.507679    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-077ac6-frdbj"
Nov 01 04:08:01.564966 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:01.564938    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-077ac6-frdbj"]
Nov 01 04:08:01.583847 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:01.583819    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ba78ea-ac58-4e51-b241-6cc364e8fd23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-077ac6-frdbj\" (UID: \"f1ba78ea-ac58-4e51-b241-6cc364e8fd23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-077ac6-frdbj"
Nov 01 04:08:01.584019 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:01.583853    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5qgm\" (UniqueName: \"kubernetes.io/projected/f1ba78ea-ac58-4e51-b241-6cc364e8fd23-kube-api-access-l5qgm\") pod \"maas-api-revoke-keys-e2e-worker-single-main-077ac6-frdbj\" (UID: \"f1ba78ea-ac58-4e51-b241-6cc364e8fd23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-077ac6-frdbj"
Nov 01 04:08:01.584019 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:01.583985    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ba78ea-ac58-4e51-b241-6cc364e8fd23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-077ac6-frdbj\" (UID: \"f1ba78ea-ac58-4e51-b241-6cc364e8fd23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-077ac6-frdbj"
Nov 01 04:08:01.610872 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:01.610845    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 04:08:01.610872 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:01.610867    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 04:08:01.610872 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:01.610848    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:08:01.611085 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:01.610924    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 04:08:01.684806 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:01.684774    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ba78ea-ac58-4e51-b241-6cc364e8fd23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-077ac6-frdbj\" (UID: \"f1ba78ea-ac58-4e51-b241-6cc364e8fd23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-077ac6-frdbj"
Nov 01 04:08:01.684806 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:01.684807    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5qgm\" (UniqueName: \"kubernetes.io/projected/f1ba78ea-ac58-4e51-b241-6cc364e8fd23-kube-api-access-l5qgm\") pod \"maas-api-revoke-keys-e2e-worker-single-main-077ac6-frdbj\" (UID: \"f1ba78ea-ac58-4e51-b241-6cc364e8fd23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-077ac6-frdbj"
Nov 01 04:08:01.685007 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:01.684985    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ba78ea-ac58-4e51-b241-6cc364e8fd23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-077ac6-frdbj\" (UID: \"f1ba78ea-ac58-4e51-b241-6cc364e8fd23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-077ac6-frdbj"
Nov 01 04:08:01.685488 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:01.685464    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ba78ea-ac58-4e51-b241-6cc364e8fd23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-077ac6-frdbj\" (UID: \"f1ba78ea-ac58-4e51-b241-6cc364e8fd23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-077ac6-frdbj"
Nov 01 04:08:01.687479 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:01.687453    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ba78ea-ac58-4e51-b241-6cc364e8fd23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-077ac6-frdbj\" (UID: \"f1ba78ea-ac58-4e51-b241-6cc364e8fd23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-077ac6-frdbj"
Nov 01 04:08:01.746252 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:01.746220    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5qgm\" (UniqueName: \"kubernetes.io/projected/f1ba78ea-ac58-4e51-b241-6cc364e8fd23-kube-api-access-l5qgm\") pod \"maas-api-revoke-keys-e2e-worker-single-main-077ac6-frdbj\" (UID: \"f1ba78ea-ac58-4e51-b241-6cc364e8fd23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-077ac6-frdbj"
Nov 01 04:08:01.823292 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:01.823216    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-077ac6-frdbj"
Nov 01 04:08:02.197782 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:02.197750    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-077ac6-frdbj"]
Nov 01 04:08:02.199417 ip-10-0-141-122 kubenswrapper[4930]: W1101 04:08:02.199387    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ba78ea_ac58_4e51_b241_6cc364e8fd23.slice/crio-faa00e15981e9694fc1c764044f7c0ef209bb21b98c262cd75c043978dbf9739 WatchSource:0}: Error finding container faa00e15981e9694fc1c764044f7c0ef209bb21b98c262cd75c043978dbf9739: Status 404 returned error can't find the container with id faa00e15981e9694fc1c764044f7c0ef209bb21b98c262cd75c043978dbf9739
Nov 01 04:08:02.756253 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:02.756171    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-077ac6-frdbj" event={"ID":"f1ba78ea-ac58-4e51-b241-6cc364e8fd23","Type":"ContainerStarted","Data":"1e61ea38119102bd1b07eb971cc7319560708727ac1c205e8b6e1a2613e5c9ff"}
Nov 01 04:08:02.756253 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:02.756237    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-077ac6-frdbj" event={"ID":"f1ba78ea-ac58-4e51-b241-6cc364e8fd23","Type":"ContainerStarted","Data":"faa00e15981e9694fc1c764044f7c0ef209bb21b98c262cd75c043978dbf9739"}
Nov 01 04:08:02.873597 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:02.873531    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-077ac6-frdbj" podStartSLOduration=1.873516524 podStartE2EDuration="1.873516524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:08:02.871780262 +0000 UTC m=+3687.394786853" watchObservedRunningTime="2026-11-01 04:08:02.873516524 +0000 UTC m=+3687.396523112"
Nov 01 04:08:03.763003 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:03.762967    4930 generic.go:356] "Generic (PLEG): container finished" podID="f1ba78ea-ac58-4e51-b241-6cc364e8fd23" containerID="1e61ea38119102bd1b07eb971cc7319560708727ac1c205e8b6e1a2613e5c9ff" exitCode=0
Nov 01 04:08:03.763428 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:03.763056    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-077ac6-frdbj" event={"ID":"f1ba78ea-ac58-4e51-b241-6cc364e8fd23","Type":"ContainerDied","Data":"1e61ea38119102bd1b07eb971cc7319560708727ac1c205e8b6e1a2613e5c9ff"}
Nov 01 04:08:04.921087 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:04.921059    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-077ac6-frdbj"
Nov 01 04:08:05.020795 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:05.020729    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1ba78ea-ac58-4e51-b241-6cc364e8fd23-kube-api-access-l5qgm\" (UniqueName: \"kubernetes.io/projected/f1ba78ea-ac58-4e51-b241-6cc364e8fd23-kube-api-access-l5qgm\") pod \"f1ba78ea-ac58-4e51-b241-6cc364e8fd23\" (UID: \"f1ba78ea-ac58-4e51-b241-6cc364e8fd23\") "
Nov 01 04:08:05.020957 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:05.020794    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1ba78ea-ac58-4e51-b241-6cc364e8fd23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ba78ea-ac58-4e51-b241-6cc364e8fd23-maas-api-xks-ca\") pod \"f1ba78ea-ac58-4e51-b241-6cc364e8fd23\" (UID: \"f1ba78ea-ac58-4e51-b241-6cc364e8fd23\") "
Nov 01 04:08:05.020957 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:05.020862    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1ba78ea-ac58-4e51-b241-6cc364e8fd23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ba78ea-ac58-4e51-b241-6cc364e8fd23-maas-api-openshift-ca\") pod \"f1ba78ea-ac58-4e51-b241-6cc364e8fd23\" (UID: \"f1ba78ea-ac58-4e51-b241-6cc364e8fd23\") "
Nov 01 04:08:05.021259 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:05.021235    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1ba78ea-ac58-4e51-b241-6cc364e8fd23-maas-api-openshift-ca" pod "f1ba78ea-ac58-4e51-b241-6cc364e8fd23" (UID: "f1ba78ea-ac58-4e51-b241-6cc364e8fd23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:08:05.022757 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:05.022735    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1ba78ea-ac58-4e51-b241-6cc364e8fd23-kube-api-access-l5qgm" pod "f1ba78ea-ac58-4e51-b241-6cc364e8fd23" (UID: "f1ba78ea-ac58-4e51-b241-6cc364e8fd23"). InnerVolumeSpecName "kube-api-access-l5qgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:08:05.022850 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:05.022834    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1ba78ea-ac58-4e51-b241-6cc364e8fd23-maas-api-xks-ca" pod "f1ba78ea-ac58-4e51-b241-6cc364e8fd23" (UID: "f1ba78ea-ac58-4e51-b241-6cc364e8fd23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:08:05.122459 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:05.122423    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5qgm\" (UniqueName: \"kubernetes.io/projected/f1ba78ea-ac58-4e51-b241-6cc364e8fd23-kube-api-access-l5qgm\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:08:05.122459 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:05.122454    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ba78ea-ac58-4e51-b241-6cc364e8fd23-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:08:05.122693 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:05.122471    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ba78ea-ac58-4e51-b241-6cc364e8fd23-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:08:05.261488 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:05.261452    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:08:05.775441 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:05.775401    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-077ac6-frdbj" event={"ID":"f1ba78ea-ac58-4e51-b241-6cc364e8fd23","Type":"ContainerDied","Data":"faa00e15981e9694fc1c764044f7c0ef209bb21b98c262cd75c043978dbf9739"}
Nov 01 04:08:05.775616 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:05.775459    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="faa00e15981e9694fc1c764044f7c0ef209bb21b98c262cd75c043978dbf9739"
Nov 01 04:08:05.775616 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:05.775411    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-077ac6-frdbj"
Nov 01 04:08:06.077886 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:06.077803    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:08:10.081479 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:10.081440    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:08:12.983719 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:12.983676    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-077ac6-67bf4cf86j4nnw"]
Nov 01 04:08:13.088921 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:13.088890    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-077ac6-7cd74zxv4f"]
Nov 01 04:08:13.341734 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:13.341702    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:08:13.544473 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:13.544437    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:08:14.885035 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:14.884986    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:08:17.984667 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:17.984600    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-077ac6-67bf4cf86j4nnw" podUID="7e29d20b-433b-4fa1-bd90-707e83a245f2" containerName="payload-processing" containerID="cri-o://962ffba62f91481e80bc7277b1dd0d371af3445e529f5661b1f2791108f3cffc" gracePeriod=25
Nov 01 04:08:18.089571 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:18.089525    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-077ac6-7cd74zxv4f" podUID="c8d38fbc-ffe4-46b8-b273-9cc2d8838550" containerName="payload-pre-processing" containerID="cri-o://6a763464fd1bdd43060d568372f183d03d342f1dbd78cfc93c221d6622a982a0" gracePeriod=25
Nov 01 04:08:19.082948 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:19.082916    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:08:19.121388 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:08:19.121355    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 04:08:20.580864 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:20.580824    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:08:20.864722 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:20.864690    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:08:21.198687 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:21.198605    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-077ac6-frdbj"]
Nov 01 04:08:21.247833 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:21.247799    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-077ac6-frdbj"]
Nov 01 04:08:21.619649 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:21.619611    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1ba78ea-ac58-4e51-b241-6cc364e8fd23" path="/var/lib/kubelet/pods/f1ba78ea-ac58-4e51-b241-6cc364e8fd23/volumes"
Nov 01 04:08:21.644322 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:21.644293    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:08:22.247429 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:22.247394    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-077ac6-67bf4cf86j4nnw" podUID="7e29d20b-433b-4fa1-bd90-707e83a245f2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 04:08:24.258487 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:24.258443    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-077ac6-7cd74zxv4f" podUID="c8d38fbc-ffe4-46b8-b273-9cc2d8838550" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 04:08:27.475311 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:27.475272    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:08:27.620901 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:27.620866    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:08:27.834343 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:27.834321    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"]
Nov 01 04:08:27.834710 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:27.834674    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h" podUID="ee888f27-32db-468c-a2df-7a5bceb25246" containerName="istio-proxy" containerID="cri-o://3ebf7d8cc5a843827242cb283c527fd6c754fff4ae9ccd2ae164f489ef92aa14" gracePeriod=30
Nov 01 04:08:27.845928 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:27.845892    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:08:31.895758 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:31.895710    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:08:32.247795 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:32.247704    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-077ac6-67bf4cf86j4nnw" podUID="7e29d20b-433b-4fa1-bd90-707e83a245f2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 04:08:32.971110 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:32.971075    4930 generic.go:356] "Generic (PLEG): container finished" podID="ee888f27-32db-468c-a2df-7a5bceb25246" containerID="3ebf7d8cc5a843827242cb283c527fd6c754fff4ae9ccd2ae164f489ef92aa14" exitCode=0
Nov 01 04:08:32.971540 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:32.971148    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h" event={"ID":"ee888f27-32db-468c-a2df-7a5bceb25246","Type":"ContainerDied","Data":"3ebf7d8cc5a843827242cb283c527fd6c754fff4ae9ccd2ae164f489ef92aa14"}
Nov 01 04:08:32.971540 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:32.971211    4930 scope.go:122] "RemoveContainer" containerID="1df684fd4d0a2d0e0f8a9d88b6815f74a4fae0b4bd0e919602e278f4ca1bc921"
Nov 01 04:08:33.086011 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:33.085988    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"
Nov 01 04:08:33.207757 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:33.207680    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee888f27-32db-468c-a2df-7a5bceb25246-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee888f27-32db-468c-a2df-7a5bceb25246-istio-data\") pod \"ee888f27-32db-468c-a2df-7a5bceb25246\" (UID: \"ee888f27-32db-468c-a2df-7a5bceb25246\") "
Nov 01 04:08:33.207757 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:33.207729    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee888f27-32db-468c-a2df-7a5bceb25246-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee888f27-32db-468c-a2df-7a5bceb25246-credential-socket\") pod \"ee888f27-32db-468c-a2df-7a5bceb25246\" (UID: \"ee888f27-32db-468c-a2df-7a5bceb25246\") "
Nov 01 04:08:33.207978 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:33.207759    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee888f27-32db-468c-a2df-7a5bceb25246-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee888f27-32db-468c-a2df-7a5bceb25246-istiod-ca-cert\") pod \"ee888f27-32db-468c-a2df-7a5bceb25246\" (UID: \"ee888f27-32db-468c-a2df-7a5bceb25246\") "
Nov 01 04:08:33.207978 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:33.207795    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee888f27-32db-468c-a2df-7a5bceb25246-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee888f27-32db-468c-a2df-7a5bceb25246-workload-certs\") pod \"ee888f27-32db-468c-a2df-7a5bceb25246\" (UID: \"ee888f27-32db-468c-a2df-7a5bceb25246\") "
Nov 01 04:08:33.207978 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:33.207880    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee888f27-32db-468c-a2df-7a5bceb25246-istio-token\" (UniqueName: \"kubernetes.io/projected/ee888f27-32db-468c-a2df-7a5bceb25246-istio-token\") pod \"ee888f27-32db-468c-a2df-7a5bceb25246\" (UID: \"ee888f27-32db-468c-a2df-7a5bceb25246\") "
Nov 01 04:08:33.207978 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:33.207927    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee888f27-32db-468c-a2df-7a5bceb25246-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee888f27-32db-468c-a2df-7a5bceb25246-istio-podinfo\") pod \"ee888f27-32db-468c-a2df-7a5bceb25246\" (UID: \"ee888f27-32db-468c-a2df-7a5bceb25246\") "
Nov 01 04:08:33.207978 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:33.207968    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee888f27-32db-468c-a2df-7a5bceb25246-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee888f27-32db-468c-a2df-7a5bceb25246-istio-envoy\") pod \"ee888f27-32db-468c-a2df-7a5bceb25246\" (UID: \"ee888f27-32db-468c-a2df-7a5bceb25246\") "
Nov 01 04:08:33.208261 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:33.207969    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee888f27-32db-468c-a2df-7a5bceb25246-istio-data" pod "ee888f27-32db-468c-a2df-7a5bceb25246" (UID: "ee888f27-32db-468c-a2df-7a5bceb25246"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:33.208261 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:33.208005    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee888f27-32db-468c-a2df-7a5bceb25246-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee888f27-32db-468c-a2df-7a5bceb25246-workload-socket\") pod \"ee888f27-32db-468c-a2df-7a5bceb25246\" (UID: \"ee888f27-32db-468c-a2df-7a5bceb25246\") "
Nov 01 04:08:33.208261 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:33.208052    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee888f27-32db-468c-a2df-7a5bceb25246-credential-socket" pod "ee888f27-32db-468c-a2df-7a5bceb25246" (UID: "ee888f27-32db-468c-a2df-7a5bceb25246"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:33.208261 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:33.208064    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee888f27-32db-468c-a2df-7a5bceb25246-kube-api-access-zqhkl\" (UniqueName: \"kubernetes.io/projected/ee888f27-32db-468c-a2df-7a5bceb25246-kube-api-access-zqhkl\") pod \"ee888f27-32db-468c-a2df-7a5bceb25246\" (UID: \"ee888f27-32db-468c-a2df-7a5bceb25246\") "
Nov 01 04:08:33.208261 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:33.208203    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee888f27-32db-468c-a2df-7a5bceb25246-istiod-ca-cert" pod "ee888f27-32db-468c-a2df-7a5bceb25246" (UID: "ee888f27-32db-468c-a2df-7a5bceb25246"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:08:33.208515 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:33.208293    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee888f27-32db-468c-a2df-7a5bceb25246-workload-certs" pod "ee888f27-32db-468c-a2df-7a5bceb25246" (UID: "ee888f27-32db-468c-a2df-7a5bceb25246"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:33.208654 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:33.208626    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee888f27-32db-468c-a2df-7a5bceb25246-workload-socket" pod "ee888f27-32db-468c-a2df-7a5bceb25246" (UID: "ee888f27-32db-468c-a2df-7a5bceb25246"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:33.208654 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:33.208648    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee888f27-32db-468c-a2df-7a5bceb25246-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:08:33.208842 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:33.208671    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee888f27-32db-468c-a2df-7a5bceb25246-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:08:33.208842 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:33.208689    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee888f27-32db-468c-a2df-7a5bceb25246-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:08:33.208842 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:33.208704    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee888f27-32db-468c-a2df-7a5bceb25246-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:08:33.210763 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:33.210738    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee888f27-32db-468c-a2df-7a5bceb25246-istio-podinfo" pod "ee888f27-32db-468c-a2df-7a5bceb25246" (UID: "ee888f27-32db-468c-a2df-7a5bceb25246"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:08:33.210763 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:33.210749    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee888f27-32db-468c-a2df-7a5bceb25246-kube-api-access-zqhkl" pod "ee888f27-32db-468c-a2df-7a5bceb25246" (UID: "ee888f27-32db-468c-a2df-7a5bceb25246"). InnerVolumeSpecName "kube-api-access-zqhkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:08:33.210931 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:33.210742    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee888f27-32db-468c-a2df-7a5bceb25246-istio-envoy" pod "ee888f27-32db-468c-a2df-7a5bceb25246" (UID: "ee888f27-32db-468c-a2df-7a5bceb25246"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:08:33.210931 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:33.210790    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee888f27-32db-468c-a2df-7a5bceb25246-istio-token" pod "ee888f27-32db-468c-a2df-7a5bceb25246" (UID: "ee888f27-32db-468c-a2df-7a5bceb25246"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:08:33.310234 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:33.310205    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee888f27-32db-468c-a2df-7a5bceb25246-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:08:33.310234 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:33.310231    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee888f27-32db-468c-a2df-7a5bceb25246-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:08:33.310234 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:33.310240    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee888f27-32db-468c-a2df-7a5bceb25246-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:08:33.310474 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:33.310250    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqhkl\" (UniqueName: \"kubernetes.io/projected/ee888f27-32db-468c-a2df-7a5bceb25246-kube-api-access-zqhkl\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:08:33.310474 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:33.310262    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee888f27-32db-468c-a2df-7a5bceb25246-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:08:33.980122 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:33.980045    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"
Nov 01 04:08:33.980580 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:33.980036    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h" event={"ID":"ee888f27-32db-468c-a2df-7a5bceb25246","Type":"ContainerDied","Data":"ea32b25998bc1e2db843a4a268f38aa31496149db77ed8d62eb009a130a3841f"}
Nov 01 04:08:33.980580 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:33.980211    4930 scope.go:122] "RemoveContainer" containerID="3ebf7d8cc5a843827242cb283c527fd6c754fff4ae9ccd2ae164f489ef92aa14"
Nov 01 04:08:34.197841 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:34.197794    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"]
Nov 01 04:08:34.200616 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:34.200592    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-077ac6-openshift-default-9f959995c-b2w2h"]
Nov 01 04:08:34.258445 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:34.258413    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-077ac6-7cd74zxv4f" podUID="c8d38fbc-ffe4-46b8-b273-9cc2d8838550" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 04:08:35.314114 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:35.314079    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m"]
Nov 01 04:08:35.314530 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:35.314516    4930 cpu_manager.go:414] "containerMap: removing container" podUID="ee888f27-32db-468c-a2df-7a5bceb25246" containerName="istio-proxy"
Nov 01 04:08:35.314591 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:35.314530    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee888f27-32db-468c-a2df-7a5bceb25246" containerName="istio-proxy"
Nov 01 04:08:35.314591 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:35.314544    4930 cpu_manager.go:414] "containerMap: removing container" podUID="f1ba78ea-ac58-4e51-b241-6cc364e8fd23" containerName="revoke-keys"
Nov 01 04:08:35.314591 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:35.314549    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1ba78ea-ac58-4e51-b241-6cc364e8fd23" containerName="revoke-keys"
Nov 01 04:08:35.314591 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:35.314574    4930 cpu_manager.go:414] "containerMap: removing container" podUID="ee888f27-32db-468c-a2df-7a5bceb25246" containerName="istio-proxy"
Nov 01 04:08:35.314591 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:35.314579    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee888f27-32db-468c-a2df-7a5bceb25246" containerName="istio-proxy"
Nov 01 04:08:35.315407 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:35.315376    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee888f27-32db-468c-a2df-7a5bceb25246" containerName="istio-proxy"
Nov 01 04:08:35.315580 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:35.315551    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1ba78ea-ac58-4e51-b241-6cc364e8fd23" containerName="revoke-keys"
Nov 01 04:08:35.315681 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:35.315670    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee888f27-32db-468c-a2df-7a5bceb25246" containerName="istio-proxy"
Nov 01 04:08:35.321957 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:35.321928    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m"
Nov 01 04:08:35.377530 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:35.377497    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m"]
Nov 01 04:08:35.405384 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:35.405353    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:08:35.423854 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:35.423831    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-d0082e19-kserve-self-signed-certs\""
Nov 01 04:08:35.433248 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:35.433221    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2254\" (UniqueName: \"kubernetes.io/projected/74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a-kube-api-access-f2254\") pod \"e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m\" (UID: \"74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a\") " pod="llm/e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m"
Nov 01 04:08:35.433360 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:35.433252    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a-home\") pod \"e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m\" (UID: \"74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a\") " pod="llm/e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m"
Nov 01 04:08:35.433360 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:35.433296    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a-dshm\") pod \"e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m\" (UID: \"74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a\") " pod="llm/e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m"
Nov 01 04:08:35.433463 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:35.433399    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a-model-cache\") pod \"e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m\" (UID: \"74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a\") " pod="llm/e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m"
Nov 01 04:08:35.433463 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:35.433424    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a-tmp-dir\") pod \"e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m\" (UID: \"74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a\") " pod="llm/e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m"
Nov 01 04:08:35.433463 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:35.433448    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a-tls-certs\") pod \"e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m\" (UID: \"74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a\") " pod="llm/e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m"
Nov 01 04:08:35.534154 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:35.534117    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a-dshm\") pod \"e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m\" (UID: \"74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a\") " pod="llm/e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m"
Nov 01 04:08:35.534347 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:35.534161    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a-model-cache\") pod \"e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m\" (UID: \"74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a\") " pod="llm/e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m"
Nov 01 04:08:35.534347 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:35.534196    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a-tmp-dir\") pod \"e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m\" (UID: \"74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a\") " pod="llm/e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m"
Nov 01 04:08:35.534347 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:35.534227    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a-tls-certs\") pod \"e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m\" (UID: \"74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a\") " pod="llm/e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m"
Nov 01 04:08:35.534347 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:35.534318    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2254\" (UniqueName: \"kubernetes.io/projected/74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a-kube-api-access-f2254\") pod \"e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m\" (UID: \"74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a\") " pod="llm/e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m"
Nov 01 04:08:35.534617 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:35.534348    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a-home\") pod \"e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m\" (UID: \"74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a\") " pod="llm/e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m"
Nov 01 04:08:35.534699 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:35.534672    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a-tmp-dir\") pod \"e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m\" (UID: \"74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a\") " pod="llm/e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m"
Nov 01 04:08:35.534766 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:35.534703    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a-model-cache\") pod \"e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m\" (UID: \"74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a\") " pod="llm/e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m"
Nov 01 04:08:35.534766 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:35.534736    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a-home\") pod \"e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m\" (UID: \"74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a\") " pod="llm/e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m"
Nov 01 04:08:35.536612 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:35.536585    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a-dshm\") pod \"e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m\" (UID: \"74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a\") " pod="llm/e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m"
Nov 01 04:08:35.536828 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:35.536808    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a-tls-certs\") pod \"e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m\" (UID: \"74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a\") " pod="llm/e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m"
Nov 01 04:08:35.595512 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:35.595484    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2254\" (UniqueName: \"kubernetes.io/projected/74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a-kube-api-access-f2254\") pod \"e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m\" (UID: \"74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a\") " pod="llm/e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m"
Nov 01 04:08:35.618898 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:35.618873    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee888f27-32db-468c-a2df-7a5bceb25246" path="/var/lib/kubelet/pods/ee888f27-32db-468c-a2df-7a5bceb25246/volumes"
Nov 01 04:08:35.635593 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:35.635572    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m"
Nov 01 04:08:36.017515 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:36.017492    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m"]
Nov 01 04:08:36.018889 ip-10-0-141-122 kubenswrapper[4930]: W1101 04:08:36.018850    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74e2ce1b_fda7_4dcb_ac24_df9f4de5a48a.slice/crio-c9af8cd621bdcb06d18396a8595f297eecf1bb6805cb2bc5555e2c4cd8ff5179 WatchSource:0}: Error finding container c9af8cd621bdcb06d18396a8595f297eecf1bb6805cb2bc5555e2c4cd8ff5179: Status 404 returned error can't find the container with id c9af8cd621bdcb06d18396a8595f297eecf1bb6805cb2bc5555e2c4cd8ff5179
Nov 01 04:08:37.000894 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:37.000849    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m" event={"ID":"74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a","Type":"ContainerStarted","Data":"e60455984407d45e9dd9ae287141efd404557736103e13243c63576d0500c95d"}
Nov 01 04:08:37.000894 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:37.000900    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m" event={"ID":"74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a","Type":"ContainerStarted","Data":"c9af8cd621bdcb06d18396a8595f297eecf1bb6805cb2bc5555e2c4cd8ff5179"}
Nov 01 04:08:37.119688 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:37.119636    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m" podStartSLOduration=2.119622678 podStartE2EDuration="2.119622678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:08:37.117834954 +0000 UTC m=+3721.640841539" watchObservedRunningTime="2026-11-01 04:08:37.119622678 +0000 UTC m=+3721.642629302"
Nov 01 04:08:37.636315 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:37.636267    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m"
Nov 01 04:08:38.245456 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:38.245393    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-077ac6-67bf4cf86j4nnw"
Nov 01 04:08:38.339486 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:38.339458    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-077ac6-7cd74zxv4f"
Nov 01 04:08:38.363931 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:38.363902    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e29d20b-433b-4fa1-bd90-707e83a245f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e29d20b-433b-4fa1-bd90-707e83a245f2-plugins-config-volume\") pod \"7e29d20b-433b-4fa1-bd90-707e83a245f2\" (UID: \"7e29d20b-433b-4fa1-bd90-707e83a245f2\") "
Nov 01 04:08:38.364092 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:38.363965    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e29d20b-433b-4fa1-bd90-707e83a245f2-kube-api-access-x9x5t\" (UniqueName: \"kubernetes.io/projected/7e29d20b-433b-4fa1-bd90-707e83a245f2-kube-api-access-x9x5t\") pod \"7e29d20b-433b-4fa1-bd90-707e83a245f2\" (UID: \"7e29d20b-433b-4fa1-bd90-707e83a245f2\") "
Nov 01 04:08:38.364840 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:38.364815    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e29d20b-433b-4fa1-bd90-707e83a245f2-plugins-config-volume" pod "7e29d20b-433b-4fa1-bd90-707e83a245f2" (UID: "7e29d20b-433b-4fa1-bd90-707e83a245f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:08:38.366451 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:38.366430    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e29d20b-433b-4fa1-bd90-707e83a245f2-kube-api-access-x9x5t" pod "7e29d20b-433b-4fa1-bd90-707e83a245f2" (UID: "7e29d20b-433b-4fa1-bd90-707e83a245f2"). InnerVolumeSpecName "kube-api-access-x9x5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:08:38.465311 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:38.465283    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8d38fbc-ffe4-46b8-b273-9cc2d8838550-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8d38fbc-ffe4-46b8-b273-9cc2d8838550-plugins-config-volume\") pod \"c8d38fbc-ffe4-46b8-b273-9cc2d8838550\" (UID: \"c8d38fbc-ffe4-46b8-b273-9cc2d8838550\") "
Nov 01 04:08:38.465427 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:38.465386    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8d38fbc-ffe4-46b8-b273-9cc2d8838550-kube-api-access-h22xg\" (UniqueName: \"kubernetes.io/projected/c8d38fbc-ffe4-46b8-b273-9cc2d8838550-kube-api-access-h22xg\") pod \"c8d38fbc-ffe4-46b8-b273-9cc2d8838550\" (UID: \"c8d38fbc-ffe4-46b8-b273-9cc2d8838550\") "
Nov 01 04:08:38.465658 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:38.465645    4930 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e29d20b-433b-4fa1-bd90-707e83a245f2-plugins-config-volume\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:08:38.465706 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:38.465660    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9x5t\" (UniqueName: \"kubernetes.io/projected/7e29d20b-433b-4fa1-bd90-707e83a245f2-kube-api-access-x9x5t\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:08:38.465747 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:38.465724    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8d38fbc-ffe4-46b8-b273-9cc2d8838550-plugins-config-volume" pod "c8d38fbc-ffe4-46b8-b273-9cc2d8838550" (UID: "c8d38fbc-ffe4-46b8-b273-9cc2d8838550"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:08:38.467396 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:38.467374    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8d38fbc-ffe4-46b8-b273-9cc2d8838550-kube-api-access-h22xg" pod "c8d38fbc-ffe4-46b8-b273-9cc2d8838550" (UID: "c8d38fbc-ffe4-46b8-b273-9cc2d8838550"). InnerVolumeSpecName "kube-api-access-h22xg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:08:38.567187 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:38.567110    4930 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8d38fbc-ffe4-46b8-b273-9cc2d8838550-plugins-config-volume\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:08:38.567187 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:38.567143    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h22xg\" (UniqueName: \"kubernetes.io/projected/c8d38fbc-ffe4-46b8-b273-9cc2d8838550-kube-api-access-h22xg\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:08:39.012758 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:39.012728    4930 generic.go:356] "Generic (PLEG): container finished" podID="c8d38fbc-ffe4-46b8-b273-9cc2d8838550" containerID="6a763464fd1bdd43060d568372f183d03d342f1dbd78cfc93c221d6622a982a0" exitCode=0
Nov 01 04:08:39.012915 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:39.012776    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-077ac6-7cd74zxv4f"
Nov 01 04:08:39.012915 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:39.012808    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-077ac6-7cd74zxv4f" event={"ID":"c8d38fbc-ffe4-46b8-b273-9cc2d8838550","Type":"ContainerDied","Data":"6a763464fd1bdd43060d568372f183d03d342f1dbd78cfc93c221d6622a982a0"}
Nov 01 04:08:39.012915 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:39.012864    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-077ac6-7cd74zxv4f" event={"ID":"c8d38fbc-ffe4-46b8-b273-9cc2d8838550","Type":"ContainerDied","Data":"46a937903505f5610ff2ec9ea4c183c3025f3a3810f5aa26c0a5e731bcaeed06"}
Nov 01 04:08:39.012915 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:39.012893    4930 scope.go:122] "RemoveContainer" containerID="6a763464fd1bdd43060d568372f183d03d342f1dbd78cfc93c221d6622a982a0"
Nov 01 04:08:39.014350 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:39.014324    4930 generic.go:356] "Generic (PLEG): container finished" podID="7e29d20b-433b-4fa1-bd90-707e83a245f2" containerID="962ffba62f91481e80bc7277b1dd0d371af3445e529f5661b1f2791108f3cffc" exitCode=0
Nov 01 04:08:39.014468 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:39.014366    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-077ac6-67bf4cf86j4nnw"
Nov 01 04:08:39.014468 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:39.014413    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-077ac6-67bf4cf86j4nnw" event={"ID":"7e29d20b-433b-4fa1-bd90-707e83a245f2","Type":"ContainerDied","Data":"962ffba62f91481e80bc7277b1dd0d371af3445e529f5661b1f2791108f3cffc"}
Nov 01 04:08:39.014468 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:39.014457    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-077ac6-67bf4cf86j4nnw" event={"ID":"7e29d20b-433b-4fa1-bd90-707e83a245f2","Type":"ContainerDied","Data":"17ae723209c30508736274bf314061e3ee6928a22fa00e63e36e45dad3b85b46"}
Nov 01 04:08:39.022712 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:39.022690    4930 scope.go:122] "RemoveContainer" containerID="6a763464fd1bdd43060d568372f183d03d342f1dbd78cfc93c221d6622a982a0"
Nov 01 04:08:39.023154 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:08:39.022996    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a763464fd1bdd43060d568372f183d03d342f1dbd78cfc93c221d6622a982a0\": container with ID starting with 6a763464fd1bdd43060d568372f183d03d342f1dbd78cfc93c221d6622a982a0 not found: ID does not exist" containerID="6a763464fd1bdd43060d568372f183d03d342f1dbd78cfc93c221d6622a982a0"
Nov 01 04:08:39.023154 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:39.023045    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a763464fd1bdd43060d568372f183d03d342f1dbd78cfc93c221d6622a982a0"} err="failed to get container status \"6a763464fd1bdd43060d568372f183d03d342f1dbd78cfc93c221d6622a982a0\": rpc error: code = NotFound desc = could not find container \"6a763464fd1bdd43060d568372f183d03d342f1dbd78cfc93c221d6622a982a0\": container with ID starting with 6a763464fd1bdd43060d568372f183d03d342f1dbd78cfc93c221d6622a982a0 not found: ID does not exist"
Nov 01 04:08:39.023154 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:39.023067    4930 scope.go:122] "RemoveContainer" containerID="962ffba62f91481e80bc7277b1dd0d371af3445e529f5661b1f2791108f3cffc"
Nov 01 04:08:39.031809 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:39.031784    4930 scope.go:122] "RemoveContainer" containerID="962ffba62f91481e80bc7277b1dd0d371af3445e529f5661b1f2791108f3cffc"
Nov 01 04:08:39.032095 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:08:39.032069    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"962ffba62f91481e80bc7277b1dd0d371af3445e529f5661b1f2791108f3cffc\": container with ID starting with 962ffba62f91481e80bc7277b1dd0d371af3445e529f5661b1f2791108f3cffc not found: ID does not exist" containerID="962ffba62f91481e80bc7277b1dd0d371af3445e529f5661b1f2791108f3cffc"
Nov 01 04:08:39.032176 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:39.032101    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"962ffba62f91481e80bc7277b1dd0d371af3445e529f5661b1f2791108f3cffc"} err="failed to get container status \"962ffba62f91481e80bc7277b1dd0d371af3445e529f5661b1f2791108f3cffc\": rpc error: code = NotFound desc = could not find container \"962ffba62f91481e80bc7277b1dd0d371af3445e529f5661b1f2791108f3cffc\": container with ID starting with 962ffba62f91481e80bc7277b1dd0d371af3445e529f5661b1f2791108f3cffc not found: ID does not exist"
Nov 01 04:08:39.053194 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:08:39.053173    4930 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8d38fbc_ffe4_46b8_b273_9cc2d8838550.slice\": RecentStats: unable to find data in memory cache]"
Nov 01 04:08:39.180264 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:39.180224    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-077ac6-67bf4cf86j4nnw"]
Nov 01 04:08:39.229767 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:39.229722    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-077ac6-67bf4cf86j4nnw"]
Nov 01 04:08:39.295218 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:39.295129    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-077ac6-7cd74zxv4f"]
Nov 01 04:08:39.298763 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:39.298734    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-077ac6-7cd74zxv4f"]
Nov 01 04:08:39.621068 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:39.621031    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e29d20b-433b-4fa1-bd90-707e83a245f2" path="/var/lib/kubelet/pods/7e29d20b-433b-4fa1-bd90-707e83a245f2/volumes"
Nov 01 04:08:39.621539 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:39.621519    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8d38fbc-ffe4-46b8-b273-9cc2d8838550" path="/var/lib/kubelet/pods/c8d38fbc-ffe4-46b8-b273-9cc2d8838550/volumes"
Nov 01 04:08:43.662876 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:43.662844    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:08:44.021581 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:44.021487    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:08:45.636106 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:45.636068    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m"
Nov 01 04:08:45.648565 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:45.648524    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m"
Nov 01 04:08:46.111274 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:46.111240    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m"
Nov 01 04:08:48.060319 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:48.060233    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:08:49.122519 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:08:49.122486    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 04:08:57.811867 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:57.811827    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:08:59.075286 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:08:59.075249    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:09:16.659092 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:09:16.659056    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:09:18.441319 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:09:18.441281    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:09:19.121344 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:09:19.121312    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 04:09:41.192719 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:09:41.192688    4930 scope.go:122] "RemoveContainer" containerID="f8071fa2e9ab4a5798afba02dc5fcbfa41d60c4d13f30083e152da5ffc6b7c10"
Nov 01 04:09:43.326018 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:09:43.325986    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:09:48.457787 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:09:48.457748    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:09:49.121678 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:09:49.121640    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 04:09:58.139627 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:09:58.139546    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:10:04.563604 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:04.563550    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m"]
Nov 01 04:10:04.603106 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:04.603073    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:10:15.340533 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:15.340498    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:10:19.122024 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:10:19.121939    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 04:10:19.611717 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:19.611680    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m" podUID="74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a" containerName="main" containerID="cri-o://e60455984407d45e9dd9ae287141efd404557736103e13243c63576d0500c95d" gracePeriod=45
Nov 01 04:10:20.100290 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:20.099957    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m" podUID="74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a" containerName="main" probeResult="failure" output="Get \"https://10.133.1.34:8000/ready\": dial tcp 10.133.1.34:8000: connect: connection refused"
Nov 01 04:10:20.902095 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:20.902039    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m"
Nov 01 04:10:20.938349 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:20.938311    4930 generic.go:356] "Generic (PLEG): container finished" podID="74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a" containerID="e60455984407d45e9dd9ae287141efd404557736103e13243c63576d0500c95d" exitCode=0
Nov 01 04:10:20.938715 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:20.938696    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m" event={"ID":"74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a","Type":"ContainerDied","Data":"e60455984407d45e9dd9ae287141efd404557736103e13243c63576d0500c95d"}
Nov 01 04:10:20.938854 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:20.938841    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m" event={"ID":"74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a","Type":"ContainerDied","Data":"c9af8cd621bdcb06d18396a8595f297eecf1bb6805cb2bc5555e2c4cd8ff5179"}
Nov 01 04:10:20.938987 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:20.938975    4930 scope.go:122] "RemoveContainer" containerID="e60455984407d45e9dd9ae287141efd404557736103e13243c63576d0500c95d"
Nov 01 04:10:20.939293 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:20.939280    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m"
Nov 01 04:10:21.067629 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:21.060226    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a-home\" (UniqueName: \"kubernetes.io/empty-dir/74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a-home\") pod \"74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a\" (UID: \"74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a\") "
Nov 01 04:10:21.067629 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:21.060285    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a-dshm\") pod \"74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a\" (UID: \"74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a\") "
Nov 01 04:10:21.067629 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:21.060357    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a-kube-api-access-f2254\" (UniqueName: \"kubernetes.io/projected/74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a-kube-api-access-f2254\") pod \"74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a\" (UID: \"74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a\") "
Nov 01 04:10:21.067629 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:21.060412    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a-tls-certs\" (UniqueName: \"kubernetes.io/secret/74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a-tls-certs\") pod \"74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a\" (UID: \"74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a\") "
Nov 01 04:10:21.067629 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:21.060453    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a-tmp-dir\") pod \"74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a\" (UID: \"74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a\") "
Nov 01 04:10:21.067629 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:21.060544    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a-model-cache\") pod \"74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a\" (UID: \"74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a\") "
Nov 01 04:10:21.069826 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:21.069738    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a-model-cache" pod "74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a" (UID: "74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:10:21.088580 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:21.088312    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a-home" pod "74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a" (UID: "74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:10:21.096689 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:21.096629    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a-tmp-dir" pod "74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a" (UID: "74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:10:21.109691 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:21.109644    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a-tls-certs" pod "74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a" (UID: "74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:10:21.133696 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:21.133665    4930 scope.go:122] "RemoveContainer" containerID="e60455984407d45e9dd9ae287141efd404557736103e13243c63576d0500c95d"
Nov 01 04:10:21.134182 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:10:21.134146    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e60455984407d45e9dd9ae287141efd404557736103e13243c63576d0500c95d\": container with ID starting with e60455984407d45e9dd9ae287141efd404557736103e13243c63576d0500c95d not found: ID does not exist" containerID="e60455984407d45e9dd9ae287141efd404557736103e13243c63576d0500c95d"
Nov 01 04:10:21.134302 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:21.134197    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e60455984407d45e9dd9ae287141efd404557736103e13243c63576d0500c95d"} err="failed to get container status \"e60455984407d45e9dd9ae287141efd404557736103e13243c63576d0500c95d\": rpc error: code = NotFound desc = could not find container \"e60455984407d45e9dd9ae287141efd404557736103e13243c63576d0500c95d\": container with ID starting with e60455984407d45e9dd9ae287141efd404557736103e13243c63576d0500c95d not found: ID does not exist"
Nov 01 04:10:21.138176 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:21.138128    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a-kube-api-access-f2254" pod "74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a" (UID: "74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a"). InnerVolumeSpecName "kube-api-access-f2254". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:10:21.139661 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:21.139610    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a-dshm" pod "74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a" (UID: "74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:10:21.172612 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:21.163881    4930 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a-home\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:10:21.172612 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:21.163913    4930 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a-dshm\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:10:21.172612 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:21.163930    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2254\" (UniqueName: \"kubernetes.io/projected/74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a-kube-api-access-f2254\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:10:21.172612 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:21.163945    4930 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a-tls-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:10:21.172612 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:21.163959    4930 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a-tmp-dir\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:10:21.172612 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:21.163973    4930 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a-model-cache\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:10:21.442578 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:21.439613    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m"]
Nov 01 04:10:21.445583 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:21.443492    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-d0082e19-kserve-cb67b5d89-9tx5m"]
Nov 01 04:10:21.619530 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:21.619494    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a" path="/var/lib/kubelet/pods/74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a/volumes"
Nov 01 04:10:48.756578 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:48.756096    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:10:49.108747 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:49.108712    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea63aa93-5ntmv"]
Nov 01 04:10:49.109387 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:49.109365    4930 cpu_manager.go:414] "containerMap: removing container" podUID="7e29d20b-433b-4fa1-bd90-707e83a245f2" containerName="payload-processing"
Nov 01 04:10:49.109472 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:49.109390    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e29d20b-433b-4fa1-bd90-707e83a245f2" containerName="payload-processing"
Nov 01 04:10:49.109472 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:49.109404    4930 cpu_manager.go:414] "containerMap: removing container" podUID="c8d38fbc-ffe4-46b8-b273-9cc2d8838550" containerName="payload-pre-processing"
Nov 01 04:10:49.109472 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:49.109413    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8d38fbc-ffe4-46b8-b273-9cc2d8838550" containerName="payload-pre-processing"
Nov 01 04:10:49.109472 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:49.109451    4930 cpu_manager.go:414] "containerMap: removing container" podUID="74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a" containerName="main"
Nov 01 04:10:49.109472 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:49.109459    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a" containerName="main"
Nov 01 04:10:49.109743 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:49.109590    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e29d20b-433b-4fa1-bd90-707e83a245f2" containerName="payload-processing"
Nov 01 04:10:49.109743 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:49.109607    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8d38fbc-ffe4-46b8-b273-9cc2d8838550" containerName="payload-pre-processing"
Nov 01 04:10:49.109743 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:49.109622    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="74e2ce1b-fda7-4dcb-ac24-df9f4de5a48a" containerName="main"
Nov 01 04:10:49.112973 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:49.112947    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea63aa93-5ntmv"
Nov 01 04:10:49.125404 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:10:49.125376    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 04:10:49.166967 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:49.166918    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 04:10:49.167179 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:49.167009    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 04:10:49.216230 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:49.216201    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:10:49.216385 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:49.216256    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h8fq5\""
Nov 01 04:10:49.218709 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:49.218689    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea63aa93-5ntmv"]
Nov 01 04:10:49.242627 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:49.242601    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdnff\" (UniqueName: \"kubernetes.io/projected/b2ad2d57-84ba-42c2-95cb-0492edd90fd3-kube-api-access-mdnff\") pod \"maas-api-revoke-sub-ea63aa93-5ntmv\" (UID: \"b2ad2d57-84ba-42c2-95cb-0492edd90fd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea63aa93-5ntmv"
Nov 01 04:10:49.242781 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:49.242761    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2ad2d57-84ba-42c2-95cb-0492edd90fd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea63aa93-5ntmv\" (UID: \"b2ad2d57-84ba-42c2-95cb-0492edd90fd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea63aa93-5ntmv"
Nov 01 04:10:49.242853 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:49.242821    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2ad2d57-84ba-42c2-95cb-0492edd90fd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea63aa93-5ntmv\" (UID: \"b2ad2d57-84ba-42c2-95cb-0492edd90fd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea63aa93-5ntmv"
Nov 01 04:10:49.343456 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:49.343426    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdnff\" (UniqueName: \"kubernetes.io/projected/b2ad2d57-84ba-42c2-95cb-0492edd90fd3-kube-api-access-mdnff\") pod \"maas-api-revoke-sub-ea63aa93-5ntmv\" (UID: \"b2ad2d57-84ba-42c2-95cb-0492edd90fd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea63aa93-5ntmv"
Nov 01 04:10:49.343602 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:49.343526    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2ad2d57-84ba-42c2-95cb-0492edd90fd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea63aa93-5ntmv\" (UID: \"b2ad2d57-84ba-42c2-95cb-0492edd90fd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea63aa93-5ntmv"
Nov 01 04:10:49.343602 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:49.343591    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2ad2d57-84ba-42c2-95cb-0492edd90fd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea63aa93-5ntmv\" (UID: \"b2ad2d57-84ba-42c2-95cb-0492edd90fd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea63aa93-5ntmv"
Nov 01 04:10:49.344164 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:49.344145    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2ad2d57-84ba-42c2-95cb-0492edd90fd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea63aa93-5ntmv\" (UID: \"b2ad2d57-84ba-42c2-95cb-0492edd90fd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea63aa93-5ntmv"
Nov 01 04:10:49.345896 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:49.345874    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2ad2d57-84ba-42c2-95cb-0492edd90fd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea63aa93-5ntmv\" (UID: \"b2ad2d57-84ba-42c2-95cb-0492edd90fd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea63aa93-5ntmv"
Nov 01 04:10:49.403992 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:49.403942    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdnff\" (UniqueName: \"kubernetes.io/projected/b2ad2d57-84ba-42c2-95cb-0492edd90fd3-kube-api-access-mdnff\") pod \"maas-api-revoke-sub-ea63aa93-5ntmv\" (UID: \"b2ad2d57-84ba-42c2-95cb-0492edd90fd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea63aa93-5ntmv"
Nov 01 04:10:49.426721 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:49.426703    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea63aa93-5ntmv"
Nov 01 04:10:49.798982 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:49.798951    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea63aa93-5ntmv"]
Nov 01 04:10:49.799710 ip-10-0-141-122 kubenswrapper[4930]: W1101 04:10:49.799680    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2ad2d57_84ba_42c2_95cb_0492edd90fd3.slice/crio-a2af1abeb46c17c1a631929370561c25b9064543fd5780c5ed8e0f639db53585 WatchSource:0}: Error finding container a2af1abeb46c17c1a631929370561c25b9064543fd5780c5ed8e0f639db53585: Status 404 returned error can't find the container with id a2af1abeb46c17c1a631929370561c25b9064543fd5780c5ed8e0f639db53585
Nov 01 04:10:50.159605 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:50.159566    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea63aa93-5ntmv" event={"ID":"b2ad2d57-84ba-42c2-95cb-0492edd90fd3","Type":"ContainerStarted","Data":"bba2e2e4565f4f38251c28750715bc4b36275c76f624a0de6d4eaefdb9720a88"}
Nov 01 04:10:50.159740 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:50.159638    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea63aa93-5ntmv" event={"ID":"b2ad2d57-84ba-42c2-95cb-0492edd90fd3","Type":"ContainerStarted","Data":"a2af1abeb46c17c1a631929370561c25b9064543fd5780c5ed8e0f639db53585"}
Nov 01 04:10:50.275279 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:50.275222    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea63aa93-5ntmv" podStartSLOduration=1.275208568 podStartE2EDuration="1.275208568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:10:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:10:50.274711902 +0000 UTC m=+3854.797718490" watchObservedRunningTime="2026-11-01 04:10:50.275208568 +0000 UTC m=+3854.798215157"
Nov 01 04:10:51.174473 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:51.174441    4930 generic.go:356] "Generic (PLEG): container finished" podID="b2ad2d57-84ba-42c2-95cb-0492edd90fd3" containerID="bba2e2e4565f4f38251c28750715bc4b36275c76f624a0de6d4eaefdb9720a88" exitCode=0
Nov 01 04:10:51.175080 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:51.174633    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea63aa93-5ntmv" event={"ID":"b2ad2d57-84ba-42c2-95cb-0492edd90fd3","Type":"ContainerDied","Data":"bba2e2e4565f4f38251c28750715bc4b36275c76f624a0de6d4eaefdb9720a88"}
Nov 01 04:10:52.317924 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:52.317899    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea63aa93-5ntmv"
Nov 01 04:10:52.376439 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:52.376413    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2ad2d57-84ba-42c2-95cb-0492edd90fd3-kube-api-access-mdnff\" (UniqueName: \"kubernetes.io/projected/b2ad2d57-84ba-42c2-95cb-0492edd90fd3-kube-api-access-mdnff\") pod \"b2ad2d57-84ba-42c2-95cb-0492edd90fd3\" (UID: \"b2ad2d57-84ba-42c2-95cb-0492edd90fd3\") "
Nov 01 04:10:52.376597 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:52.376502    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2ad2d57-84ba-42c2-95cb-0492edd90fd3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2ad2d57-84ba-42c2-95cb-0492edd90fd3-maas-api-xks-ca\") pod \"b2ad2d57-84ba-42c2-95cb-0492edd90fd3\" (UID: \"b2ad2d57-84ba-42c2-95cb-0492edd90fd3\") "
Nov 01 04:10:52.376597 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:52.376526    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2ad2d57-84ba-42c2-95cb-0492edd90fd3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2ad2d57-84ba-42c2-95cb-0492edd90fd3-maas-api-openshift-ca\") pod \"b2ad2d57-84ba-42c2-95cb-0492edd90fd3\" (UID: \"b2ad2d57-84ba-42c2-95cb-0492edd90fd3\") "
Nov 01 04:10:52.376980 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:52.376953    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2ad2d57-84ba-42c2-95cb-0492edd90fd3-maas-api-openshift-ca" pod "b2ad2d57-84ba-42c2-95cb-0492edd90fd3" (UID: "b2ad2d57-84ba-42c2-95cb-0492edd90fd3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:10:52.378432 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:52.378411    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2ad2d57-84ba-42c2-95cb-0492edd90fd3-kube-api-access-mdnff" pod "b2ad2d57-84ba-42c2-95cb-0492edd90fd3" (UID: "b2ad2d57-84ba-42c2-95cb-0492edd90fd3"). InnerVolumeSpecName "kube-api-access-mdnff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:10:52.378612 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:52.378593    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2ad2d57-84ba-42c2-95cb-0492edd90fd3-maas-api-xks-ca" pod "b2ad2d57-84ba-42c2-95cb-0492edd90fd3" (UID: "b2ad2d57-84ba-42c2-95cb-0492edd90fd3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:10:52.478217 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:52.478146    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdnff\" (UniqueName: \"kubernetes.io/projected/b2ad2d57-84ba-42c2-95cb-0492edd90fd3-kube-api-access-mdnff\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:10:52.478217 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:52.478174    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2ad2d57-84ba-42c2-95cb-0492edd90fd3-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:10:52.478217 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:52.478185    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2ad2d57-84ba-42c2-95cb-0492edd90fd3-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:10:53.187405 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:53.187374    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea63aa93-5ntmv"
Nov 01 04:10:53.187405 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:53.187391    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea63aa93-5ntmv" event={"ID":"b2ad2d57-84ba-42c2-95cb-0492edd90fd3","Type":"ContainerDied","Data":"a2af1abeb46c17c1a631929370561c25b9064543fd5780c5ed8e0f639db53585"}
Nov 01 04:10:53.187636 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:53.187433    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2af1abeb46c17c1a631929370561c25b9064543fd5780c5ed8e0f639db53585"
Nov 01 04:10:54.049701 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:54.049667    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7zrj9"]
Nov 01 04:10:54.099112 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:54.099080    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-7zrj9"]
Nov 01 04:10:55.620941 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:55.620905    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d35568fe-47b1-4f8d-849f-5a85a1b554fe" path="/var/lib/kubelet/pods/d35568fe-47b1-4f8d-849f-5a85a1b554fe/volumes"
Nov 01 04:10:58.479242 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:10:58.479205    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:11:08.818652 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:08.818607    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:11:09.362411 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:09.362378    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"]
Nov 01 04:11:09.362838 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:09.362825    4930 cpu_manager.go:414] "containerMap: removing container" podUID="b2ad2d57-84ba-42c2-95cb-0492edd90fd3" containerName="revoke-keys"
Nov 01 04:11:09.362893 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:09.362840    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2ad2d57-84ba-42c2-95cb-0492edd90fd3" containerName="revoke-keys"
Nov 01 04:11:09.362936 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:09.362930    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2ad2d57-84ba-42c2-95cb-0492edd90fd3" containerName="revoke-keys"
Nov 01 04:11:09.366321 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:09.366305    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"
Nov 01 04:11:09.418849 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:09.418814    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ca6b5505-openshift-default-dockercfg-4mn4x\""
Nov 01 04:11:09.450620 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:09.450576    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:11:09.471714 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:09.471664    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"]
Nov 01 04:11:09.536143 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:09.536100    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/243fb857-1985-4313-9d36-1cc66cb6e503-credential-socket\") pod \"e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5\" (UID: \"243fb857-1985-4313-9d36-1cc66cb6e503\") " pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"
Nov 01 04:11:09.536365 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:09.536155    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv6lp\" (UniqueName: \"kubernetes.io/projected/243fb857-1985-4313-9d36-1cc66cb6e503-kube-api-access-xv6lp\") pod \"e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5\" (UID: \"243fb857-1985-4313-9d36-1cc66cb6e503\") " pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"
Nov 01 04:11:09.536365 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:09.536213    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/243fb857-1985-4313-9d36-1cc66cb6e503-istio-token\") pod \"e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5\" (UID: \"243fb857-1985-4313-9d36-1cc66cb6e503\") " pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"
Nov 01 04:11:09.536365 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:09.536245    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/243fb857-1985-4313-9d36-1cc66cb6e503-workload-certs\") pod \"e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5\" (UID: \"243fb857-1985-4313-9d36-1cc66cb6e503\") " pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"
Nov 01 04:11:09.536365 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:09.536270    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/243fb857-1985-4313-9d36-1cc66cb6e503-istio-envoy\") pod \"e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5\" (UID: \"243fb857-1985-4313-9d36-1cc66cb6e503\") " pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"
Nov 01 04:11:09.536365 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:09.536328    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/243fb857-1985-4313-9d36-1cc66cb6e503-istiod-ca-cert\") pod \"e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5\" (UID: \"243fb857-1985-4313-9d36-1cc66cb6e503\") " pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"
Nov 01 04:11:09.536745 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:09.536368    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/243fb857-1985-4313-9d36-1cc66cb6e503-istio-data\") pod \"e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5\" (UID: \"243fb857-1985-4313-9d36-1cc66cb6e503\") " pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"
Nov 01 04:11:09.536745 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:09.536462    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/243fb857-1985-4313-9d36-1cc66cb6e503-istio-podinfo\") pod \"e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5\" (UID: \"243fb857-1985-4313-9d36-1cc66cb6e503\") " pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"
Nov 01 04:11:09.536745 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:09.536485    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/243fb857-1985-4313-9d36-1cc66cb6e503-workload-socket\") pod \"e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5\" (UID: \"243fb857-1985-4313-9d36-1cc66cb6e503\") " pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"
Nov 01 04:11:09.637859 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:09.637784    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/243fb857-1985-4313-9d36-1cc66cb6e503-credential-socket\") pod \"e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5\" (UID: \"243fb857-1985-4313-9d36-1cc66cb6e503\") " pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"
Nov 01 04:11:09.637859 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:09.637819    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv6lp\" (UniqueName: \"kubernetes.io/projected/243fb857-1985-4313-9d36-1cc66cb6e503-kube-api-access-xv6lp\") pod \"e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5\" (UID: \"243fb857-1985-4313-9d36-1cc66cb6e503\") " pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"
Nov 01 04:11:09.637859 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:09.637855    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/243fb857-1985-4313-9d36-1cc66cb6e503-istio-token\") pod \"e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5\" (UID: \"243fb857-1985-4313-9d36-1cc66cb6e503\") " pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"
Nov 01 04:11:09.638153 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:09.637880    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/243fb857-1985-4313-9d36-1cc66cb6e503-workload-certs\") pod \"e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5\" (UID: \"243fb857-1985-4313-9d36-1cc66cb6e503\") " pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"
Nov 01 04:11:09.638153 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:09.637906    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/243fb857-1985-4313-9d36-1cc66cb6e503-istio-envoy\") pod \"e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5\" (UID: \"243fb857-1985-4313-9d36-1cc66cb6e503\") " pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"
Nov 01 04:11:09.638153 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:09.637977    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/243fb857-1985-4313-9d36-1cc66cb6e503-istiod-ca-cert\") pod \"e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5\" (UID: \"243fb857-1985-4313-9d36-1cc66cb6e503\") " pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"
Nov 01 04:11:09.638153 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:09.638019    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/243fb857-1985-4313-9d36-1cc66cb6e503-istio-data\") pod \"e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5\" (UID: \"243fb857-1985-4313-9d36-1cc66cb6e503\") " pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"
Nov 01 04:11:09.638153 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:09.638110    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/243fb857-1985-4313-9d36-1cc66cb6e503-istio-podinfo\") pod \"e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5\" (UID: \"243fb857-1985-4313-9d36-1cc66cb6e503\") " pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"
Nov 01 04:11:09.638153 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:09.638135    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/243fb857-1985-4313-9d36-1cc66cb6e503-workload-socket\") pod \"e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5\" (UID: \"243fb857-1985-4313-9d36-1cc66cb6e503\") " pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"
Nov 01 04:11:09.638471 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:09.638291    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/243fb857-1985-4313-9d36-1cc66cb6e503-credential-socket\") pod \"e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5\" (UID: \"243fb857-1985-4313-9d36-1cc66cb6e503\") " pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"
Nov 01 04:11:09.638471 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:09.638315    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/243fb857-1985-4313-9d36-1cc66cb6e503-workload-certs\") pod \"e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5\" (UID: \"243fb857-1985-4313-9d36-1cc66cb6e503\") " pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"
Nov 01 04:11:09.638471 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:09.638443    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/243fb857-1985-4313-9d36-1cc66cb6e503-workload-socket\") pod \"e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5\" (UID: \"243fb857-1985-4313-9d36-1cc66cb6e503\") " pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"
Nov 01 04:11:09.638633 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:09.638526    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/243fb857-1985-4313-9d36-1cc66cb6e503-istio-data\") pod \"e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5\" (UID: \"243fb857-1985-4313-9d36-1cc66cb6e503\") " pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"
Nov 01 04:11:09.638754 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:09.638735    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/243fb857-1985-4313-9d36-1cc66cb6e503-istiod-ca-cert\") pod \"e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5\" (UID: \"243fb857-1985-4313-9d36-1cc66cb6e503\") " pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"
Nov 01 04:11:09.640288 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:09.640256    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/243fb857-1985-4313-9d36-1cc66cb6e503-istio-envoy\") pod \"e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5\" (UID: \"243fb857-1985-4313-9d36-1cc66cb6e503\") " pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"
Nov 01 04:11:09.640583 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:09.640541    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/243fb857-1985-4313-9d36-1cc66cb6e503-istio-podinfo\") pod \"e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5\" (UID: \"243fb857-1985-4313-9d36-1cc66cb6e503\") " pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"
Nov 01 04:11:09.677992 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:09.677952    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:11:09.700696 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:09.700666    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/243fb857-1985-4313-9d36-1cc66cb6e503-istio-token\") pod \"e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5\" (UID: \"243fb857-1985-4313-9d36-1cc66cb6e503\") " pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"
Nov 01 04:11:09.700849 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:09.700714    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv6lp\" (UniqueName: \"kubernetes.io/projected/243fb857-1985-4313-9d36-1cc66cb6e503-kube-api-access-xv6lp\") pod \"e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5\" (UID: \"243fb857-1985-4313-9d36-1cc66cb6e503\") " pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"
Nov 01 04:11:09.979620 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:09.979510    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"
Nov 01 04:11:10.409765 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:10.409734    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"]
Nov 01 04:11:10.411346 ip-10-0-141-122 kubenswrapper[4930]: W1101 04:11:10.411312    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod243fb857_1985_4313_9d36_1cc66cb6e503.slice/crio-3180f93a751e5b29157bc4089ab20f394e56854c7324f7bff1c9e9896cfd502a WatchSource:0}: Error finding container 3180f93a751e5b29157bc4089ab20f394e56854c7324f7bff1c9e9896cfd502a: Status 404 returned error can't find the container with id 3180f93a751e5b29157bc4089ab20f394e56854c7324f7bff1c9e9896cfd502a
Nov 01 04:11:10.413602 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:10.413542    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:11:10.413685 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:10.413638    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:11:10.413685 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:10.413665    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:11:11.285052 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:11.285015    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5" event={"ID":"243fb857-1985-4313-9d36-1cc66cb6e503","Type":"ContainerStarted","Data":"9eccbec3b560c1dba54f4bf5513d6e12e7f236b090d392b5620ab81ffc803b81"}
Nov 01 04:11:11.285481 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:11.285071    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5" event={"ID":"243fb857-1985-4313-9d36-1cc66cb6e503","Type":"ContainerStarted","Data":"3180f93a751e5b29157bc4089ab20f394e56854c7324f7bff1c9e9896cfd502a"}
Nov 01 04:11:11.980351 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:11.980320    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"
Nov 01 04:11:11.985609 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:11.985584    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"
Nov 01 04:11:12.109192 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:12.109102    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5" podStartSLOduration=3.109086022 podStartE2EDuration="3.109086022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:11:11.505398779 +0000 UTC m=+3876.028405367" watchObservedRunningTime="2026-11-01 04:11:12.109086022 +0000 UTC m=+3876.632092608"
Nov 01 04:11:12.290755 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:12.290671    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"
Nov 01 04:11:12.291495 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:12.291475    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"
Nov 01 04:11:13.583889 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:13.583856    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:11:13.757250 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:13.757212    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:11:13.907972 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:13.907932    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:11:16.845093 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:16.845058    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:11:16.995574 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:16.995526    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:11:17.240711 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:17.240625    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ca6b5505-6cf77bdd9f-qcs27"]
Nov 01 04:11:17.244296 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:17.244273    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ca6b5505-6cf77bdd9f-qcs27"
Nov 01 04:11:17.300236 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:17.300212    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ca6b5505\""
Nov 01 04:11:17.316313 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:17.316286    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqt9t\" (UniqueName: \"kubernetes.io/projected/97531ac4-7bfb-4e7a-bfae-eec9e76043c4-kube-api-access-qqt9t\") pod \"payload-pre-processing-e2e-ait-ca6b5505-6cf77bdd9f-qcs27\" (UID: \"97531ac4-7bfb-4e7a-bfae-eec9e76043c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ca6b5505-6cf77bdd9f-qcs27"
Nov 01 04:11:17.316440 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:17.316348    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97531ac4-7bfb-4e7a-bfae-eec9e76043c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ca6b5505-6cf77bdd9f-qcs27\" (UID: \"97531ac4-7bfb-4e7a-bfae-eec9e76043c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ca6b5505-6cf77bdd9f-qcs27"
Nov 01 04:11:17.348969 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:17.348941    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ca6b5505-6cf77bdd9f-qcs27"]
Nov 01 04:11:17.402101 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:17.402061    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ca6b5505-f5fc4bf45-9v25p"]
Nov 01 04:11:17.407599 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:17.407550    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ca6b5505-f5fc4bf45-9v25p"
Nov 01 04:11:17.420702 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:17.420676    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqt9t\" (UniqueName: \"kubernetes.io/projected/97531ac4-7bfb-4e7a-bfae-eec9e76043c4-kube-api-access-qqt9t\") pod \"payload-pre-processing-e2e-ait-ca6b5505-6cf77bdd9f-qcs27\" (UID: \"97531ac4-7bfb-4e7a-bfae-eec9e76043c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ca6b5505-6cf77bdd9f-qcs27"
Nov 01 04:11:17.420856 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:17.420720    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1048d330-ed78-4ab6-ad60-539e0637dc54-plugins-config-volume\") pod \"payload-processing-e2e-ait-ca6b5505-f5fc4bf45-9v25p\" (UID: \"1048d330-ed78-4ab6-ad60-539e0637dc54\") " pod="openshift-ingress/payload-processing-e2e-ait-ca6b5505-f5fc4bf45-9v25p"
Nov 01 04:11:17.420856 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:17.420768    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97531ac4-7bfb-4e7a-bfae-eec9e76043c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ca6b5505-6cf77bdd9f-qcs27\" (UID: \"97531ac4-7bfb-4e7a-bfae-eec9e76043c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ca6b5505-6cf77bdd9f-qcs27"
Nov 01 04:11:17.420981 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:17.420868    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7ftt\" (UniqueName: \"kubernetes.io/projected/1048d330-ed78-4ab6-ad60-539e0637dc54-kube-api-access-f7ftt\") pod \"payload-processing-e2e-ait-ca6b5505-f5fc4bf45-9v25p\" (UID: \"1048d330-ed78-4ab6-ad60-539e0637dc54\") " pod="openshift-ingress/payload-processing-e2e-ait-ca6b5505-f5fc4bf45-9v25p"
Nov 01 04:11:17.422034 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:17.421997    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97531ac4-7bfb-4e7a-bfae-eec9e76043c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ca6b5505-6cf77bdd9f-qcs27\" (UID: \"97531ac4-7bfb-4e7a-bfae-eec9e76043c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ca6b5505-6cf77bdd9f-qcs27"
Nov 01 04:11:17.435704 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:17.435678    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:11:17.454041 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:17.454011    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ca6b5505-f5fc4bf45-9v25p"]
Nov 01 04:11:17.464458 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:17.464429    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ca6b5505-dockercfg-79kl5\""
Nov 01 04:11:17.499793 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:17.483870    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqt9t\" (UniqueName: \"kubernetes.io/projected/97531ac4-7bfb-4e7a-bfae-eec9e76043c4-kube-api-access-qqt9t\") pod \"payload-pre-processing-e2e-ait-ca6b5505-6cf77bdd9f-qcs27\" (UID: \"97531ac4-7bfb-4e7a-bfae-eec9e76043c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ca6b5505-6cf77bdd9f-qcs27"
Nov 01 04:11:17.522533 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:17.522491    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1048d330-ed78-4ab6-ad60-539e0637dc54-plugins-config-volume\") pod \"payload-processing-e2e-ait-ca6b5505-f5fc4bf45-9v25p\" (UID: \"1048d330-ed78-4ab6-ad60-539e0637dc54\") " pod="openshift-ingress/payload-processing-e2e-ait-ca6b5505-f5fc4bf45-9v25p"
Nov 01 04:11:17.522734 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:17.522684    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7ftt\" (UniqueName: \"kubernetes.io/projected/1048d330-ed78-4ab6-ad60-539e0637dc54-kube-api-access-f7ftt\") pod \"payload-processing-e2e-ait-ca6b5505-f5fc4bf45-9v25p\" (UID: \"1048d330-ed78-4ab6-ad60-539e0637dc54\") " pod="openshift-ingress/payload-processing-e2e-ait-ca6b5505-f5fc4bf45-9v25p"
Nov 01 04:11:17.523329 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:17.523307    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1048d330-ed78-4ab6-ad60-539e0637dc54-plugins-config-volume\") pod \"payload-processing-e2e-ait-ca6b5505-f5fc4bf45-9v25p\" (UID: \"1048d330-ed78-4ab6-ad60-539e0637dc54\") " pod="openshift-ingress/payload-processing-e2e-ait-ca6b5505-f5fc4bf45-9v25p"
Nov 01 04:11:17.557817 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:17.557792    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ca6b5505-6cf77bdd9f-qcs27"
Nov 01 04:11:17.583461 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:17.583428    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7ftt\" (UniqueName: \"kubernetes.io/projected/1048d330-ed78-4ab6-ad60-539e0637dc54-kube-api-access-f7ftt\") pod \"payload-processing-e2e-ait-ca6b5505-f5fc4bf45-9v25p\" (UID: \"1048d330-ed78-4ab6-ad60-539e0637dc54\") " pod="openshift-ingress/payload-processing-e2e-ait-ca6b5505-f5fc4bf45-9v25p"
Nov 01 04:11:17.600312 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:17.600262    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:11:17.729099 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:17.729066    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ca6b5505-f5fc4bf45-9v25p"
Nov 01 04:11:17.750603 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:17.750522    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:11:17.932593 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:17.932546    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ca6b5505-6cf77bdd9f-qcs27"]
Nov 01 04:11:17.933199 ip-10-0-141-122 kubenswrapper[4930]: W1101 04:11:17.933174    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97531ac4_7bfb_4e7a_bfae_eec9e76043c4.slice/crio-7cf5673c91d358cdf4759b9a62762099d9ceabd7c03800ae04a202add3074f11 WatchSource:0}: Error finding container 7cf5673c91d358cdf4759b9a62762099d9ceabd7c03800ae04a202add3074f11: Status 404 returned error can't find the container with id 7cf5673c91d358cdf4759b9a62762099d9ceabd7c03800ae04a202add3074f11
Nov 01 04:11:18.102899 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:18.102873    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ca6b5505-f5fc4bf45-9v25p"]
Nov 01 04:11:18.104166 ip-10-0-141-122 kubenswrapper[4930]: W1101 04:11:18.104137    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1048d330_ed78_4ab6_ad60_539e0637dc54.slice/crio-ecd98ffd540e847522f2d5edf4f3b57060dec655ea9ff147ea18c48984a52b45 WatchSource:0}: Error finding container ecd98ffd540e847522f2d5edf4f3b57060dec655ea9ff147ea18c48984a52b45: Status 404 returned error can't find the container with id ecd98ffd540e847522f2d5edf4f3b57060dec655ea9ff147ea18c48984a52b45
Nov 01 04:11:18.325126 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:18.325090    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ca6b5505-6cf77bdd9f-qcs27" event={"ID":"97531ac4-7bfb-4e7a-bfae-eec9e76043c4","Type":"ContainerStarted","Data":"bf652bed95876a83dceea357534633c56a1ca893173790df06c5585ffd4d0d7b"}
Nov 01 04:11:18.325312 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:18.325154    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ca6b5505-6cf77bdd9f-qcs27" event={"ID":"97531ac4-7bfb-4e7a-bfae-eec9e76043c4","Type":"ContainerStarted","Data":"7cf5673c91d358cdf4759b9a62762099d9ceabd7c03800ae04a202add3074f11"}
Nov 01 04:11:18.325312 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:18.325200    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ca6b5505-6cf77bdd9f-qcs27"
Nov 01 04:11:18.326724 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:18.326700    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ca6b5505-f5fc4bf45-9v25p" event={"ID":"1048d330-ed78-4ab6-ad60-539e0637dc54","Type":"ContainerStarted","Data":"c4c948c4b468041b076f1461f3ca829f4d3ad209e97da253991536a17cdc4eea"}
Nov 01 04:11:18.326875 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:18.326743    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ca6b5505-f5fc4bf45-9v25p" event={"ID":"1048d330-ed78-4ab6-ad60-539e0637dc54","Type":"ContainerStarted","Data":"ecd98ffd540e847522f2d5edf4f3b57060dec655ea9ff147ea18c48984a52b45"}
Nov 01 04:11:18.326875 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:18.326770    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ca6b5505-f5fc4bf45-9v25p"
Nov 01 04:11:18.442434 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:18.442346    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ca6b5505-6cf77bdd9f-qcs27" podStartSLOduration=1.442333077 podStartE2EDuration="1.442333077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:11:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:11:18.441032251 +0000 UTC m=+3882.964038840" watchObservedRunningTime="2026-11-01 04:11:18.442333077 +0000 UTC m=+3882.965339665"
Nov 01 04:11:18.556957 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:18.556908    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ca6b5505-f5fc4bf45-9v25p" podStartSLOduration=1.556893449 podStartE2EDuration="1.556893449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:11:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:11:18.555778607 +0000 UTC m=+3883.078785222" watchObservedRunningTime="2026-11-01 04:11:18.556893449 +0000 UTC m=+3883.079900064"
Nov 01 04:11:19.120778 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:11:19.120748    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 04:11:22.771753 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:22.771710    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:11:23.051596 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:23.051487    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-psfgj"]
Nov 01 04:11:23.100959 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:23.100924    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-psfgj"]
Nov 01 04:11:23.619780 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:23.619746    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0667bc47-e82f-42f6-bd76-8f942e668f06" path="/var/lib/kubelet/pods/0667bc47-e82f-42f6-bd76-8f942e668f06/volumes"
Nov 01 04:11:27.613201 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:27.613163    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:11:29.335366 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:29.335334    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ca6b5505-f5fc4bf45-9v25p"
Nov 01 04:11:29.335816 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:29.335401    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ca6b5505-6cf77bdd9f-qcs27"
Nov 01 04:11:32.462070 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:32.462029    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:11:35.450053 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:35.449972    4930 generic.go:356] "Generic (PLEG): container finished" podID="243fb857-1985-4313-9d36-1cc66cb6e503" containerID="9eccbec3b560c1dba54f4bf5513d6e12e7f236b090d392b5620ab81ffc803b81" exitCode=0
Nov 01 04:11:35.450463 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:35.450047    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5" event={"ID":"243fb857-1985-4313-9d36-1cc66cb6e503","Type":"ContainerDied","Data":"9eccbec3b560c1dba54f4bf5513d6e12e7f236b090d392b5620ab81ffc803b81"}
Nov 01 04:11:35.450507 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:35.450464    4930 scope.go:122] "RemoveContainer" containerID="9eccbec3b560c1dba54f4bf5513d6e12e7f236b090d392b5620ab81ffc803b81"
Nov 01 04:11:35.459060 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:35.451917    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:11:35.459060 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:35.452118    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:11:35.459060 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:35.452230    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:11:36.457623 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:36.457584    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5" event={"ID":"243fb857-1985-4313-9d36-1cc66cb6e503","Type":"ContainerStarted","Data":"34fd0e7729096ad28394e67daea9e810234dbbc95f64b026e7eaa2eb8fbe55b5"}
Nov 01 04:11:36.980972 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:36.980934    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"
Nov 01 04:11:36.981354 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:36.981270    4930 patch_prober.go:29] interesting pod/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body=
Nov 01 04:11:36.981537 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:36.981514    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5" podUID="243fb857-1985-4313-9d36-1cc66cb6e503" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused"
Nov 01 04:11:37.804045 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:37.804010    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:11:37.980021 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:37.979981    4930 patch_prober.go:29] interesting pod/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body=
Nov 01 04:11:37.980343 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:37.980311    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5" podUID="243fb857-1985-4313-9d36-1cc66cb6e503" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused"
Nov 01 04:11:38.483839 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:38.483808    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jh955_f29ea61e-2a78-4c11-9566-f52497e135ff/console-operator/2.log"
Nov 01 04:11:38.515664 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:38.515638    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jh955_f29ea61e-2a78-4c11-9566-f52497e135ff/console-operator/2.log"
Nov 01 04:11:38.982582 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:38.981867    4930 patch_prober.go:29] interesting pod/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body=
Nov 01 04:11:38.982582 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:38.981929    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5" podUID="243fb857-1985-4313-9d36-1cc66cb6e503" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused"
Nov 01 04:11:39.981581 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:39.980979    4930 patch_prober.go:29] interesting pod/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body=
Nov 01 04:11:39.981581 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:39.981054    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5" podUID="243fb857-1985-4313-9d36-1cc66cb6e503" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused"
Nov 01 04:11:40.980043 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:40.980009    4930 patch_prober.go:29] interesting pod/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body=
Nov 01 04:11:40.980666 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:40.980620    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5" podUID="243fb857-1985-4313-9d36-1cc66cb6e503" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused"
Nov 01 04:11:41.984331 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:41.984298    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"
Nov 01 04:11:41.984752 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:41.984717    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"
Nov 01 04:11:41.985384 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:41.985365    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"
Nov 01 04:11:42.670041 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:42.670004    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:11:44.966129 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:44.966090    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca6b5505-6q55z"]
Nov 01 04:11:44.969884 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:44.969865    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca6b5505-6q55z"
Nov 01 04:11:45.022243 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:45.022214    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 04:11:45.071906 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:45.071877    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:11:45.071906 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:45.071898    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 04:11:45.072068 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:45.071974    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7br8\""
Nov 01 04:11:45.074868 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:45.074849    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca6b5505-6q55z"]
Nov 01 04:11:45.139694 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:45.139668    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4abebf5-8329-433f-b323-65f9eb4aafb6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ca6b5505-6q55z\" (UID: \"a4abebf5-8329-433f-b323-65f9eb4aafb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca6b5505-6q55z"
Nov 01 04:11:45.139826 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:45.139733    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4abebf5-8329-433f-b323-65f9eb4aafb6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ca6b5505-6q55z\" (UID: \"a4abebf5-8329-433f-b323-65f9eb4aafb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca6b5505-6q55z"
Nov 01 04:11:45.139881 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:45.139850    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r4wl\" (UniqueName: \"kubernetes.io/projected/a4abebf5-8329-433f-b323-65f9eb4aafb6-kube-api-access-6r4wl\") pod \"maas-api-revoke-keys-e2e-ait-ca6b5505-6q55z\" (UID: \"a4abebf5-8329-433f-b323-65f9eb4aafb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca6b5505-6q55z"
Nov 01 04:11:45.241430 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:45.241351    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r4wl\" (UniqueName: \"kubernetes.io/projected/a4abebf5-8329-433f-b323-65f9eb4aafb6-kube-api-access-6r4wl\") pod \"maas-api-revoke-keys-e2e-ait-ca6b5505-6q55z\" (UID: \"a4abebf5-8329-433f-b323-65f9eb4aafb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca6b5505-6q55z"
Nov 01 04:11:45.241430 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:45.241401    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4abebf5-8329-433f-b323-65f9eb4aafb6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ca6b5505-6q55z\" (UID: \"a4abebf5-8329-433f-b323-65f9eb4aafb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca6b5505-6q55z"
Nov 01 04:11:45.241677 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:45.241462    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4abebf5-8329-433f-b323-65f9eb4aafb6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ca6b5505-6q55z\" (UID: \"a4abebf5-8329-433f-b323-65f9eb4aafb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca6b5505-6q55z"
Nov 01 04:11:45.242144 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:45.242124    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4abebf5-8329-433f-b323-65f9eb4aafb6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ca6b5505-6q55z\" (UID: \"a4abebf5-8329-433f-b323-65f9eb4aafb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca6b5505-6q55z"
Nov 01 04:11:45.243834 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:45.243812    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4abebf5-8329-433f-b323-65f9eb4aafb6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ca6b5505-6q55z\" (UID: \"a4abebf5-8329-433f-b323-65f9eb4aafb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca6b5505-6q55z"
Nov 01 04:11:45.302045 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:45.302016    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r4wl\" (UniqueName: \"kubernetes.io/projected/a4abebf5-8329-433f-b323-65f9eb4aafb6-kube-api-access-6r4wl\") pod \"maas-api-revoke-keys-e2e-ait-ca6b5505-6q55z\" (UID: \"a4abebf5-8329-433f-b323-65f9eb4aafb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca6b5505-6q55z"
Nov 01 04:11:45.582350 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:45.582269    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca6b5505-6q55z"
Nov 01 04:11:45.955647 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:45.955612    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca6b5505-6q55z"]
Nov 01 04:11:45.957023 ip-10-0-141-122 kubenswrapper[4930]: W1101 04:11:45.956998    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4abebf5_8329_433f_b323_65f9eb4aafb6.slice/crio-b386d1d43dc65291609435f4b61349914529a10f3e2ce1a00934e97dbfdbe295 WatchSource:0}: Error finding container b386d1d43dc65291609435f4b61349914529a10f3e2ce1a00934e97dbfdbe295: Status 404 returned error can't find the container with id b386d1d43dc65291609435f4b61349914529a10f3e2ce1a00934e97dbfdbe295
Nov 01 04:11:46.538938 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:46.538852    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca6b5505-6q55z" event={"ID":"a4abebf5-8329-433f-b323-65f9eb4aafb6","Type":"ContainerStarted","Data":"50f8b9acfe883e35feee3fe3ec49177122f93dea391f6a05a56bdecec815388a"}
Nov 01 04:11:46.538938 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:46.538912    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca6b5505-6q55z" event={"ID":"a4abebf5-8329-433f-b323-65f9eb4aafb6","Type":"ContainerStarted","Data":"b386d1d43dc65291609435f4b61349914529a10f3e2ce1a00934e97dbfdbe295"}
Nov 01 04:11:46.689394 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:46.689337    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca6b5505-6q55z" podStartSLOduration=2.689321422 podStartE2EDuration="2.689321422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:11:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:11:46.687958624 +0000 UTC m=+3911.210965213" watchObservedRunningTime="2026-11-01 04:11:46.689321422 +0000 UTC m=+3911.212328011"
Nov 01 04:11:47.479444 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:47.479409    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:11:47.547141 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:47.547108    4930 generic.go:356] "Generic (PLEG): container finished" podID="a4abebf5-8329-433f-b323-65f9eb4aafb6" containerID="50f8b9acfe883e35feee3fe3ec49177122f93dea391f6a05a56bdecec815388a" exitCode=0
Nov 01 04:11:47.547551 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:47.547164    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca6b5505-6q55z" event={"ID":"a4abebf5-8329-433f-b323-65f9eb4aafb6","Type":"ContainerDied","Data":"50f8b9acfe883e35feee3fe3ec49177122f93dea391f6a05a56bdecec815388a"}
Nov 01 04:11:48.694515 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:48.694485    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca6b5505-6q55z"
Nov 01 04:11:48.873747 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:48.873714    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4abebf5-8329-433f-b323-65f9eb4aafb6-kube-api-access-6r4wl\" (UniqueName: \"kubernetes.io/projected/a4abebf5-8329-433f-b323-65f9eb4aafb6-kube-api-access-6r4wl\") pod \"a4abebf5-8329-433f-b323-65f9eb4aafb6\" (UID: \"a4abebf5-8329-433f-b323-65f9eb4aafb6\") "
Nov 01 04:11:48.873910 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:48.873807    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4abebf5-8329-433f-b323-65f9eb4aafb6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4abebf5-8329-433f-b323-65f9eb4aafb6-maas-api-openshift-ca\") pod \"a4abebf5-8329-433f-b323-65f9eb4aafb6\" (UID: \"a4abebf5-8329-433f-b323-65f9eb4aafb6\") "
Nov 01 04:11:48.873910 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:48.873835    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4abebf5-8329-433f-b323-65f9eb4aafb6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4abebf5-8329-433f-b323-65f9eb4aafb6-maas-api-xks-ca\") pod \"a4abebf5-8329-433f-b323-65f9eb4aafb6\" (UID: \"a4abebf5-8329-433f-b323-65f9eb4aafb6\") "
Nov 01 04:11:48.874177 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:48.874144    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4abebf5-8329-433f-b323-65f9eb4aafb6-maas-api-openshift-ca" pod "a4abebf5-8329-433f-b323-65f9eb4aafb6" (UID: "a4abebf5-8329-433f-b323-65f9eb4aafb6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:11:48.875747 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:48.875724    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4abebf5-8329-433f-b323-65f9eb4aafb6-kube-api-access-6r4wl" pod "a4abebf5-8329-433f-b323-65f9eb4aafb6" (UID: "a4abebf5-8329-433f-b323-65f9eb4aafb6"). InnerVolumeSpecName "kube-api-access-6r4wl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:11:48.875840 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:48.875768    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4abebf5-8329-433f-b323-65f9eb4aafb6-maas-api-xks-ca" pod "a4abebf5-8329-433f-b323-65f9eb4aafb6" (UID: "a4abebf5-8329-433f-b323-65f9eb4aafb6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:11:48.974721 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:48.974693    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r4wl\" (UniqueName: \"kubernetes.io/projected/a4abebf5-8329-433f-b323-65f9eb4aafb6-kube-api-access-6r4wl\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:11:48.974721 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:48.974718    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4abebf5-8329-433f-b323-65f9eb4aafb6-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:11:48.974872 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:48.974728    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4abebf5-8329-433f-b323-65f9eb4aafb6-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:11:49.070389 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:49.070347    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:11:49.119941 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:11:49.119911    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 04:11:49.559305 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:49.559275    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca6b5505-6q55z" event={"ID":"a4abebf5-8329-433f-b323-65f9eb4aafb6","Type":"ContainerDied","Data":"b386d1d43dc65291609435f4b61349914529a10f3e2ce1a00934e97dbfdbe295"}
Nov 01 04:11:49.559497 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:49.559317    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca6b5505-6q55z"
Nov 01 04:11:49.559497 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:49.559347    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b386d1d43dc65291609435f4b61349914529a10f3e2ce1a00934e97dbfdbe295"
Nov 01 04:11:52.265543 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:52.265514    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:11:56.445810 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:56.445776    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ca6b5505-f5fc4bf45-9v25p"]
Nov 01 04:11:56.458374 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:56.458340    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:11:56.557503 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:56.557469    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ca6b5505-6cf77bdd9f-qcs27"]
Nov 01 04:11:56.814291 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:56.814199    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:11:57.008330 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:57.008294    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:11:58.871141 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:11:58.871108    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:12:00.807694 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:00.807656    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:12:01.072715 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:01.072634    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:12:01.358799 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:01.358762    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca6b5505-6q55z"]
Nov 01 04:12:01.409274 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:01.409239    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ca6b5505-6q55z"]
Nov 01 04:12:01.446654 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:01.446611    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ca6b5505-f5fc4bf45-9v25p" podUID="1048d330-ed78-4ab6-ad60-539e0637dc54" containerName="payload-processing" containerID="cri-o://c4c948c4b468041b076f1461f3ca829f4d3ad209e97da253991536a17cdc4eea" gracePeriod=25
Nov 01 04:12:01.558720 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:01.558622    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ca6b5505-6cf77bdd9f-qcs27" podUID="97531ac4-7bfb-4e7a-bfae-eec9e76043c4" containerName="payload-pre-processing" containerID="cri-o://bf652bed95876a83dceea357534633c56a1ca893173790df06c5585ffd4d0d7b" gracePeriod=25
Nov 01 04:12:01.628938 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:01.628471    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4abebf5-8329-433f-b323-65f9eb4aafb6" path="/var/lib/kubelet/pods/a4abebf5-8329-433f-b323-65f9eb4aafb6/volumes"
Nov 01 04:12:01.634497 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:01.634466    4930 generic.go:356] "Generic (PLEG): container finished" podID="1048d330-ed78-4ab6-ad60-539e0637dc54" containerID="c4c948c4b468041b076f1461f3ca829f4d3ad209e97da253991536a17cdc4eea" exitCode=0
Nov 01 04:12:01.634695 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:01.634674    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ca6b5505-f5fc4bf45-9v25p" event={"ID":"1048d330-ed78-4ab6-ad60-539e0637dc54","Type":"ContainerDied","Data":"c4c948c4b468041b076f1461f3ca829f4d3ad209e97da253991536a17cdc4eea"}
Nov 01 04:12:01.718226 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:01.718195    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ca6b5505-f5fc4bf45-9v25p"
Nov 01 04:12:01.803702 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:01.803667    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1048d330-ed78-4ab6-ad60-539e0637dc54-kube-api-access-f7ftt\" (UniqueName: \"kubernetes.io/projected/1048d330-ed78-4ab6-ad60-539e0637dc54-kube-api-access-f7ftt\") pod \"1048d330-ed78-4ab6-ad60-539e0637dc54\" (UID: \"1048d330-ed78-4ab6-ad60-539e0637dc54\") "
Nov 01 04:12:01.803872 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:01.803837    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1048d330-ed78-4ab6-ad60-539e0637dc54-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1048d330-ed78-4ab6-ad60-539e0637dc54-plugins-config-volume\") pod \"1048d330-ed78-4ab6-ad60-539e0637dc54\" (UID: \"1048d330-ed78-4ab6-ad60-539e0637dc54\") "
Nov 01 04:12:01.804235 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:01.804209    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1048d330-ed78-4ab6-ad60-539e0637dc54-plugins-config-volume" pod "1048d330-ed78-4ab6-ad60-539e0637dc54" (UID: "1048d330-ed78-4ab6-ad60-539e0637dc54"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:12:01.805855 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:01.805830    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1048d330-ed78-4ab6-ad60-539e0637dc54-kube-api-access-f7ftt" pod "1048d330-ed78-4ab6-ad60-539e0637dc54" (UID: "1048d330-ed78-4ab6-ad60-539e0637dc54"). InnerVolumeSpecName "kube-api-access-f7ftt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:12:01.824185 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:01.824167    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ca6b5505-6cf77bdd9f-qcs27"
Nov 01 04:12:01.905430 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:01.905357    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97531ac4-7bfb-4e7a-bfae-eec9e76043c4-kube-api-access-qqt9t\" (UniqueName: \"kubernetes.io/projected/97531ac4-7bfb-4e7a-bfae-eec9e76043c4-kube-api-access-qqt9t\") pod \"97531ac4-7bfb-4e7a-bfae-eec9e76043c4\" (UID: \"97531ac4-7bfb-4e7a-bfae-eec9e76043c4\") "
Nov 01 04:12:01.905430 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:01.905420    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97531ac4-7bfb-4e7a-bfae-eec9e76043c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97531ac4-7bfb-4e7a-bfae-eec9e76043c4-plugins-config-volume\") pod \"97531ac4-7bfb-4e7a-bfae-eec9e76043c4\" (UID: \"97531ac4-7bfb-4e7a-bfae-eec9e76043c4\") "
Nov 01 04:12:01.905851 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:01.905828    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7ftt\" (UniqueName: \"kubernetes.io/projected/1048d330-ed78-4ab6-ad60-539e0637dc54-kube-api-access-f7ftt\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:12:01.905925 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:01.905856    4930 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1048d330-ed78-4ab6-ad60-539e0637dc54-plugins-config-volume\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:12:01.905925 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:01.905898    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97531ac4-7bfb-4e7a-bfae-eec9e76043c4-plugins-config-volume" pod "97531ac4-7bfb-4e7a-bfae-eec9e76043c4" (UID: "97531ac4-7bfb-4e7a-bfae-eec9e76043c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:12:01.907634 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:01.907608    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97531ac4-7bfb-4e7a-bfae-eec9e76043c4-kube-api-access-qqt9t" pod "97531ac4-7bfb-4e7a-bfae-eec9e76043c4" (UID: "97531ac4-7bfb-4e7a-bfae-eec9e76043c4"). InnerVolumeSpecName "kube-api-access-qqt9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:12:02.006686 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:02.006650    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqt9t\" (UniqueName: \"kubernetes.io/projected/97531ac4-7bfb-4e7a-bfae-eec9e76043c4-kube-api-access-qqt9t\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:12:02.006866 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:02.006701    4930 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97531ac4-7bfb-4e7a-bfae-eec9e76043c4-plugins-config-volume\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:12:02.642164 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:02.642136    4930 generic.go:356] "Generic (PLEG): container finished" podID="97531ac4-7bfb-4e7a-bfae-eec9e76043c4" containerID="bf652bed95876a83dceea357534633c56a1ca893173790df06c5585ffd4d0d7b" exitCode=0
Nov 01 04:12:02.642356 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:02.642172    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ca6b5505-6cf77bdd9f-qcs27"
Nov 01 04:12:02.642356 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:02.642222    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ca6b5505-6cf77bdd9f-qcs27" event={"ID":"97531ac4-7bfb-4e7a-bfae-eec9e76043c4","Type":"ContainerDied","Data":"bf652bed95876a83dceea357534633c56a1ca893173790df06c5585ffd4d0d7b"}
Nov 01 04:12:02.642356 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:02.642269    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ca6b5505-6cf77bdd9f-qcs27" event={"ID":"97531ac4-7bfb-4e7a-bfae-eec9e76043c4","Type":"ContainerDied","Data":"7cf5673c91d358cdf4759b9a62762099d9ceabd7c03800ae04a202add3074f11"}
Nov 01 04:12:02.642356 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:02.642296    4930 scope.go:122] "RemoveContainer" containerID="bf652bed95876a83dceea357534633c56a1ca893173790df06c5585ffd4d0d7b"
Nov 01 04:12:02.643825 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:02.643640    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ca6b5505-f5fc4bf45-9v25p" event={"ID":"1048d330-ed78-4ab6-ad60-539e0637dc54","Type":"ContainerDied","Data":"ecd98ffd540e847522f2d5edf4f3b57060dec655ea9ff147ea18c48984a52b45"}
Nov 01 04:12:02.643825 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:02.643698    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ca6b5505-f5fc4bf45-9v25p"
Nov 01 04:12:02.652892 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:02.652872    4930 scope.go:122] "RemoveContainer" containerID="bf652bed95876a83dceea357534633c56a1ca893173790df06c5585ffd4d0d7b"
Nov 01 04:12:02.653205 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:12:02.653178    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf652bed95876a83dceea357534633c56a1ca893173790df06c5585ffd4d0d7b\": container with ID starting with bf652bed95876a83dceea357534633c56a1ca893173790df06c5585ffd4d0d7b not found: ID does not exist" containerID="bf652bed95876a83dceea357534633c56a1ca893173790df06c5585ffd4d0d7b"
Nov 01 04:12:02.653302 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:02.653216    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf652bed95876a83dceea357534633c56a1ca893173790df06c5585ffd4d0d7b"} err="failed to get container status \"bf652bed95876a83dceea357534633c56a1ca893173790df06c5585ffd4d0d7b\": rpc error: code = NotFound desc = could not find container \"bf652bed95876a83dceea357534633c56a1ca893173790df06c5585ffd4d0d7b\": container with ID starting with bf652bed95876a83dceea357534633c56a1ca893173790df06c5585ffd4d0d7b not found: ID does not exist"
Nov 01 04:12:02.653302 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:02.653235    4930 scope.go:122] "RemoveContainer" containerID="c4c948c4b468041b076f1461f3ca829f4d3ad209e97da253991536a17cdc4eea"
Nov 01 04:12:02.820402 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:02.820336    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ca6b5505-6cf77bdd9f-qcs27"]
Nov 01 04:12:02.869758 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:02.869731    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ca6b5505-6cf77bdd9f-qcs27"]
Nov 01 04:12:02.933035 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:02.933006    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ca6b5505-f5fc4bf45-9v25p"]
Nov 01 04:12:02.947863 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:02.947841    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ca6b5505-f5fc4bf45-9v25p"]
Nov 01 04:12:03.621019 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:03.620987    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1048d330-ed78-4ab6-ad60-539e0637dc54" path="/var/lib/kubelet/pods/1048d330-ed78-4ab6-ad60-539e0637dc54/volumes"
Nov 01 04:12:03.621391 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:03.621378    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97531ac4-7bfb-4e7a-bfae-eec9e76043c4" path="/var/lib/kubelet/pods/97531ac4-7bfb-4e7a-bfae-eec9e76043c4/volumes"
Nov 01 04:12:06.125783 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:06.125746    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:12:06.134637 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:06.134603    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"]
Nov 01 04:12:06.134840 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:06.134820    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5" podUID="243fb857-1985-4313-9d36-1cc66cb6e503" containerName="istio-proxy" containerID="cri-o://34fd0e7729096ad28394e67daea9e810234dbbc95f64b026e7eaa2eb8fbe55b5" gracePeriod=30
Nov 01 04:12:07.388626 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:07.388586    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:12:07.936066 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:07.936032    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"]
Nov 01 04:12:07.936777 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:07.936750    4930 cpu_manager.go:414] "containerMap: removing container" podUID="1048d330-ed78-4ab6-ad60-539e0637dc54" containerName="payload-processing"
Nov 01 04:12:07.936777 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:07.936775    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1048d330-ed78-4ab6-ad60-539e0637dc54" containerName="payload-processing"
Nov 01 04:12:07.936992 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:07.936792    4930 cpu_manager.go:414] "containerMap: removing container" podUID="97531ac4-7bfb-4e7a-bfae-eec9e76043c4" containerName="payload-pre-processing"
Nov 01 04:12:07.936992 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:07.936800    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97531ac4-7bfb-4e7a-bfae-eec9e76043c4" containerName="payload-pre-processing"
Nov 01 04:12:07.936992 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:07.936814    4930 cpu_manager.go:414] "containerMap: removing container" podUID="a4abebf5-8329-433f-b323-65f9eb4aafb6" containerName="revoke-keys"
Nov 01 04:12:07.936992 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:07.936820    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4abebf5-8329-433f-b323-65f9eb4aafb6" containerName="revoke-keys"
Nov 01 04:12:07.936992 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:07.936896    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4abebf5-8329-433f-b323-65f9eb4aafb6" containerName="revoke-keys"
Nov 01 04:12:07.936992 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:07.936911    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="1048d330-ed78-4ab6-ad60-539e0637dc54" containerName="payload-processing"
Nov 01 04:12:07.936992 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:07.936922    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="97531ac4-7bfb-4e7a-bfae-eec9e76043c4" containerName="payload-pre-processing"
Nov 01 04:12:07.941886 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:07.941865    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"
Nov 01 04:12:07.995578 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:07.994845    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-84f12f-openshift-default-dockercfg-56lqw\""
Nov 01 04:12:08.034643 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:08.034602    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:12:08.048346 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:08.048310    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"]
Nov 01 04:12:08.064615 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:08.064585    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/259da31c-7d23-434a-866a-e94d82ee82bd-istiod-ca-cert\") pod \"e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684\" (UID: \"259da31c-7d23-434a-866a-e94d82ee82bd\") " pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"
Nov 01 04:12:08.064799 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:08.064645    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/259da31c-7d23-434a-866a-e94d82ee82bd-istio-data\") pod \"e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684\" (UID: \"259da31c-7d23-434a-866a-e94d82ee82bd\") " pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"
Nov 01 04:12:08.064799 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:08.064673    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/259da31c-7d23-434a-866a-e94d82ee82bd-istio-podinfo\") pod \"e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684\" (UID: \"259da31c-7d23-434a-866a-e94d82ee82bd\") " pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"
Nov 01 04:12:08.064911 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:08.064836    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/259da31c-7d23-434a-866a-e94d82ee82bd-istio-token\") pod \"e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684\" (UID: \"259da31c-7d23-434a-866a-e94d82ee82bd\") " pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"
Nov 01 04:12:08.064911 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:08.064899    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/259da31c-7d23-434a-866a-e94d82ee82bd-workload-socket\") pod \"e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684\" (UID: \"259da31c-7d23-434a-866a-e94d82ee82bd\") " pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"
Nov 01 04:12:08.064985 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:08.064966    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/259da31c-7d23-434a-866a-e94d82ee82bd-istio-envoy\") pod \"e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684\" (UID: \"259da31c-7d23-434a-866a-e94d82ee82bd\") " pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"
Nov 01 04:12:08.065020 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:08.065005    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89mnc\" (UniqueName: \"kubernetes.io/projected/259da31c-7d23-434a-866a-e94d82ee82bd-kube-api-access-89mnc\") pod \"e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684\" (UID: \"259da31c-7d23-434a-866a-e94d82ee82bd\") " pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"
Nov 01 04:12:08.065106 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:08.065063    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/259da31c-7d23-434a-866a-e94d82ee82bd-credential-socket\") pod \"e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684\" (UID: \"259da31c-7d23-434a-866a-e94d82ee82bd\") " pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"
Nov 01 04:12:08.065190 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:08.065144    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/259da31c-7d23-434a-866a-e94d82ee82bd-workload-certs\") pod \"e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684\" (UID: \"259da31c-7d23-434a-866a-e94d82ee82bd\") " pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"
Nov 01 04:12:08.165941 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:08.165908    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/259da31c-7d23-434a-866a-e94d82ee82bd-workload-socket\") pod \"e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684\" (UID: \"259da31c-7d23-434a-866a-e94d82ee82bd\") " pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"
Nov 01 04:12:08.165941 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:08.165954    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/259da31c-7d23-434a-866a-e94d82ee82bd-istio-envoy\") pod \"e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684\" (UID: \"259da31c-7d23-434a-866a-e94d82ee82bd\") " pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"
Nov 01 04:12:08.166576 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:08.165976    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89mnc\" (UniqueName: \"kubernetes.io/projected/259da31c-7d23-434a-866a-e94d82ee82bd-kube-api-access-89mnc\") pod \"e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684\" (UID: \"259da31c-7d23-434a-866a-e94d82ee82bd\") " pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"
Nov 01 04:12:08.166576 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:08.166032    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/259da31c-7d23-434a-866a-e94d82ee82bd-credential-socket\") pod \"e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684\" (UID: \"259da31c-7d23-434a-866a-e94d82ee82bd\") " pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"
Nov 01 04:12:08.166576 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:08.166109    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/259da31c-7d23-434a-866a-e94d82ee82bd-workload-certs\") pod \"e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684\" (UID: \"259da31c-7d23-434a-866a-e94d82ee82bd\") " pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"
Nov 01 04:12:08.166576 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:08.166163    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/259da31c-7d23-434a-866a-e94d82ee82bd-istiod-ca-cert\") pod \"e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684\" (UID: \"259da31c-7d23-434a-866a-e94d82ee82bd\") " pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"
Nov 01 04:12:08.166576 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:08.166212    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/259da31c-7d23-434a-866a-e94d82ee82bd-istio-data\") pod \"e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684\" (UID: \"259da31c-7d23-434a-866a-e94d82ee82bd\") " pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"
Nov 01 04:12:08.166576 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:08.166235    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/259da31c-7d23-434a-866a-e94d82ee82bd-istio-podinfo\") pod \"e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684\" (UID: \"259da31c-7d23-434a-866a-e94d82ee82bd\") " pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"
Nov 01 04:12:08.166576 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:08.166278    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/259da31c-7d23-434a-866a-e94d82ee82bd-istio-token\") pod \"e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684\" (UID: \"259da31c-7d23-434a-866a-e94d82ee82bd\") " pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"
Nov 01 04:12:08.166576 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:08.166380    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/259da31c-7d23-434a-866a-e94d82ee82bd-workload-socket\") pod \"e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684\" (UID: \"259da31c-7d23-434a-866a-e94d82ee82bd\") " pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"
Nov 01 04:12:08.167113 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:08.166765    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/259da31c-7d23-434a-866a-e94d82ee82bd-credential-socket\") pod \"e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684\" (UID: \"259da31c-7d23-434a-866a-e94d82ee82bd\") " pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"
Nov 01 04:12:08.167113 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:08.166784    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/259da31c-7d23-434a-866a-e94d82ee82bd-workload-certs\") pod \"e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684\" (UID: \"259da31c-7d23-434a-866a-e94d82ee82bd\") " pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"
Nov 01 04:12:08.167113 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:08.167040    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/259da31c-7d23-434a-866a-e94d82ee82bd-istio-data\") pod \"e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684\" (UID: \"259da31c-7d23-434a-866a-e94d82ee82bd\") " pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"
Nov 01 04:12:08.167113 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:08.167097    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/259da31c-7d23-434a-866a-e94d82ee82bd-istiod-ca-cert\") pod \"e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684\" (UID: \"259da31c-7d23-434a-866a-e94d82ee82bd\") " pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"
Nov 01 04:12:08.168764 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:08.168738    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/259da31c-7d23-434a-866a-e94d82ee82bd-istio-envoy\") pod \"e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684\" (UID: \"259da31c-7d23-434a-866a-e94d82ee82bd\") " pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"
Nov 01 04:12:08.169375 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:08.169357    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/259da31c-7d23-434a-866a-e94d82ee82bd-istio-podinfo\") pod \"e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684\" (UID: \"259da31c-7d23-434a-866a-e94d82ee82bd\") " pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"
Nov 01 04:12:08.230502 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:08.230405    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89mnc\" (UniqueName: \"kubernetes.io/projected/259da31c-7d23-434a-866a-e94d82ee82bd-kube-api-access-89mnc\") pod \"e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684\" (UID: \"259da31c-7d23-434a-866a-e94d82ee82bd\") " pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"
Nov 01 04:12:08.232468 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:08.232437    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/259da31c-7d23-434a-866a-e94d82ee82bd-istio-token\") pod \"e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684\" (UID: \"259da31c-7d23-434a-866a-e94d82ee82bd\") " pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"
Nov 01 04:12:08.242907 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:08.242876    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:12:08.255802 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:08.255775    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"
Nov 01 04:12:08.687043 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:08.687009    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"]
Nov 01 04:12:08.688894 ip-10-0-141-122 kubenswrapper[4930]: W1101 04:12:08.688869    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod259da31c_7d23_434a_866a_e94d82ee82bd.slice/crio-bc35cc87b759f93dc5d2d36b257cc7a99e48a2b2281e30262aac0d5014b2da8b WatchSource:0}: Error finding container bc35cc87b759f93dc5d2d36b257cc7a99e48a2b2281e30262aac0d5014b2da8b: Status 404 returned error can't find the container with id bc35cc87b759f93dc5d2d36b257cc7a99e48a2b2281e30262aac0d5014b2da8b
Nov 01 04:12:08.690939 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:08.690904    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:12:08.691034 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:08.690968    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:12:08.691034 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:08.690999    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:12:09.685132 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:09.685093    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684" event={"ID":"259da31c-7d23-434a-866a-e94d82ee82bd","Type":"ContainerStarted","Data":"375fdeb4241dfdf2217ac650b7b4fd283bf93522bf0a6655f3f001b5401dd35a"}
Nov 01 04:12:09.685326 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:09.685150    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684" event={"ID":"259da31c-7d23-434a-866a-e94d82ee82bd","Type":"ContainerStarted","Data":"bc35cc87b759f93dc5d2d36b257cc7a99e48a2b2281e30262aac0d5014b2da8b"}
Nov 01 04:12:09.903470 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:09.903417    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684" podStartSLOduration=2.90340445 podStartE2EDuration="2.90340445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:12:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:12:09.853664674 +0000 UTC m=+3934.376671262" watchObservedRunningTime="2026-11-01 04:12:09.90340445 +0000 UTC m=+3934.426411113"
Nov 01 04:12:10.256133 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:10.256097    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"
Nov 01 04:12:10.261050 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:10.261007    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"
Nov 01 04:12:10.692581 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:10.692093    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"
Nov 01 04:12:10.693017 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:10.692991    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"
Nov 01 04:12:11.230895 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:11.230854    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:12:11.393409 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:11.393385    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"
Nov 01 04:12:11.499466 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:11.499396    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/243fb857-1985-4313-9d36-1cc66cb6e503-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/243fb857-1985-4313-9d36-1cc66cb6e503-istiod-ca-cert\") pod \"243fb857-1985-4313-9d36-1cc66cb6e503\" (UID: \"243fb857-1985-4313-9d36-1cc66cb6e503\") "
Nov 01 04:12:11.499466 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:11.499440    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/243fb857-1985-4313-9d36-1cc66cb6e503-istio-token\" (UniqueName: \"kubernetes.io/projected/243fb857-1985-4313-9d36-1cc66cb6e503-istio-token\") pod \"243fb857-1985-4313-9d36-1cc66cb6e503\" (UID: \"243fb857-1985-4313-9d36-1cc66cb6e503\") "
Nov 01 04:12:11.499719 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:11.499517    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/243fb857-1985-4313-9d36-1cc66cb6e503-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/243fb857-1985-4313-9d36-1cc66cb6e503-istio-podinfo\") pod \"243fb857-1985-4313-9d36-1cc66cb6e503\" (UID: \"243fb857-1985-4313-9d36-1cc66cb6e503\") "
Nov 01 04:12:11.499719 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:11.499549    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/243fb857-1985-4313-9d36-1cc66cb6e503-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/243fb857-1985-4313-9d36-1cc66cb6e503-workload-certs\") pod \"243fb857-1985-4313-9d36-1cc66cb6e503\" (UID: \"243fb857-1985-4313-9d36-1cc66cb6e503\") "
Nov 01 04:12:11.499719 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:11.499608    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/243fb857-1985-4313-9d36-1cc66cb6e503-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/243fb857-1985-4313-9d36-1cc66cb6e503-workload-socket\") pod \"243fb857-1985-4313-9d36-1cc66cb6e503\" (UID: \"243fb857-1985-4313-9d36-1cc66cb6e503\") "
Nov 01 04:12:11.499719 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:11.499655    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/243fb857-1985-4313-9d36-1cc66cb6e503-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/243fb857-1985-4313-9d36-1cc66cb6e503-istio-envoy\") pod \"243fb857-1985-4313-9d36-1cc66cb6e503\" (UID: \"243fb857-1985-4313-9d36-1cc66cb6e503\") "
Nov 01 04:12:11.499719 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:11.499717    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/243fb857-1985-4313-9d36-1cc66cb6e503-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/243fb857-1985-4313-9d36-1cc66cb6e503-istio-data\") pod \"243fb857-1985-4313-9d36-1cc66cb6e503\" (UID: \"243fb857-1985-4313-9d36-1cc66cb6e503\") "
Nov 01 04:12:11.500005 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:11.499774    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/243fb857-1985-4313-9d36-1cc66cb6e503-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/243fb857-1985-4313-9d36-1cc66cb6e503-credential-socket\") pod \"243fb857-1985-4313-9d36-1cc66cb6e503\" (UID: \"243fb857-1985-4313-9d36-1cc66cb6e503\") "
Nov 01 04:12:11.500005 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:11.499808    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/243fb857-1985-4313-9d36-1cc66cb6e503-kube-api-access-xv6lp\" (UniqueName: \"kubernetes.io/projected/243fb857-1985-4313-9d36-1cc66cb6e503-kube-api-access-xv6lp\") pod \"243fb857-1985-4313-9d36-1cc66cb6e503\" (UID: \"243fb857-1985-4313-9d36-1cc66cb6e503\") "
Nov 01 04:12:11.500005 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:11.499859    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/243fb857-1985-4313-9d36-1cc66cb6e503-workload-certs" pod "243fb857-1985-4313-9d36-1cc66cb6e503" (UID: "243fb857-1985-4313-9d36-1cc66cb6e503"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:12:11.500005 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:11.499890    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/243fb857-1985-4313-9d36-1cc66cb6e503-workload-socket" pod "243fb857-1985-4313-9d36-1cc66cb6e503" (UID: "243fb857-1985-4313-9d36-1cc66cb6e503"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:12:11.500005 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:11.499912    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/243fb857-1985-4313-9d36-1cc66cb6e503-istiod-ca-cert" pod "243fb857-1985-4313-9d36-1cc66cb6e503" (UID: "243fb857-1985-4313-9d36-1cc66cb6e503"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:12:11.500256 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:11.500082    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/243fb857-1985-4313-9d36-1cc66cb6e503-istio-data" pod "243fb857-1985-4313-9d36-1cc66cb6e503" (UID: "243fb857-1985-4313-9d36-1cc66cb6e503"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:12:11.500256 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:11.500096    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/243fb857-1985-4313-9d36-1cc66cb6e503-credential-socket" pod "243fb857-1985-4313-9d36-1cc66cb6e503" (UID: "243fb857-1985-4313-9d36-1cc66cb6e503"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:12:11.500256 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:11.500235    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/243fb857-1985-4313-9d36-1cc66cb6e503-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:12:11.500256 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:11.500253    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/243fb857-1985-4313-9d36-1cc66cb6e503-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:12:11.500459 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:11.500263    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/243fb857-1985-4313-9d36-1cc66cb6e503-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:12:11.500459 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:11.500273    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/243fb857-1985-4313-9d36-1cc66cb6e503-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:12:11.500459 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:11.500283    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/243fb857-1985-4313-9d36-1cc66cb6e503-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:12:11.501896 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:11.501868    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/243fb857-1985-4313-9d36-1cc66cb6e503-istio-podinfo" pod "243fb857-1985-4313-9d36-1cc66cb6e503" (UID: "243fb857-1985-4313-9d36-1cc66cb6e503"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:12:11.502032 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:11.501983    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/243fb857-1985-4313-9d36-1cc66cb6e503-kube-api-access-xv6lp" pod "243fb857-1985-4313-9d36-1cc66cb6e503" (UID: "243fb857-1985-4313-9d36-1cc66cb6e503"). InnerVolumeSpecName "kube-api-access-xv6lp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:12:11.502122 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:11.502104    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/243fb857-1985-4313-9d36-1cc66cb6e503-istio-token" pod "243fb857-1985-4313-9d36-1cc66cb6e503" (UID: "243fb857-1985-4313-9d36-1cc66cb6e503"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:12:11.502194 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:11.502173    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/243fb857-1985-4313-9d36-1cc66cb6e503-istio-envoy" pod "243fb857-1985-4313-9d36-1cc66cb6e503" (UID: "243fb857-1985-4313-9d36-1cc66cb6e503"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:12:11.600945 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:11.600914    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv6lp\" (UniqueName: \"kubernetes.io/projected/243fb857-1985-4313-9d36-1cc66cb6e503-kube-api-access-xv6lp\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:12:11.600945 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:11.600942    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/243fb857-1985-4313-9d36-1cc66cb6e503-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:12:11.601122 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:11.600953    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/243fb857-1985-4313-9d36-1cc66cb6e503-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:12:11.601122 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:11.600962    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/243fb857-1985-4313-9d36-1cc66cb6e503-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:12:11.698918 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:11.698889    4930 generic.go:356] "Generic (PLEG): container finished" podID="243fb857-1985-4313-9d36-1cc66cb6e503" containerID="34fd0e7729096ad28394e67daea9e810234dbbc95f64b026e7eaa2eb8fbe55b5" exitCode=0
Nov 01 04:12:11.699064 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:11.698946    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"
Nov 01 04:12:11.699064 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:11.698976    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5" event={"ID":"243fb857-1985-4313-9d36-1cc66cb6e503","Type":"ContainerDied","Data":"34fd0e7729096ad28394e67daea9e810234dbbc95f64b026e7eaa2eb8fbe55b5"}
Nov 01 04:12:11.699064 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:11.699037    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5" event={"ID":"243fb857-1985-4313-9d36-1cc66cb6e503","Type":"ContainerDied","Data":"3180f93a751e5b29157bc4089ab20f394e56854c7324f7bff1c9e9896cfd502a"}
Nov 01 04:12:11.699260 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:11.699065    4930 scope.go:122] "RemoveContainer" containerID="34fd0e7729096ad28394e67daea9e810234dbbc95f64b026e7eaa2eb8fbe55b5"
Nov 01 04:12:11.708509 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:11.708492    4930 scope.go:122] "RemoveContainer" containerID="9eccbec3b560c1dba54f4bf5513d6e12e7f236b090d392b5620ab81ffc803b81"
Nov 01 04:12:11.717091 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:11.717072    4930 scope.go:122] "RemoveContainer" containerID="34fd0e7729096ad28394e67daea9e810234dbbc95f64b026e7eaa2eb8fbe55b5"
Nov 01 04:12:11.717394 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:12:11.717365    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34fd0e7729096ad28394e67daea9e810234dbbc95f64b026e7eaa2eb8fbe55b5\": container with ID starting with 34fd0e7729096ad28394e67daea9e810234dbbc95f64b026e7eaa2eb8fbe55b5 not found: ID does not exist" containerID="34fd0e7729096ad28394e67daea9e810234dbbc95f64b026e7eaa2eb8fbe55b5"
Nov 01 04:12:11.717465 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:11.717402    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34fd0e7729096ad28394e67daea9e810234dbbc95f64b026e7eaa2eb8fbe55b5"} err="failed to get container status \"34fd0e7729096ad28394e67daea9e810234dbbc95f64b026e7eaa2eb8fbe55b5\": rpc error: code = NotFound desc = could not find container \"34fd0e7729096ad28394e67daea9e810234dbbc95f64b026e7eaa2eb8fbe55b5\": container with ID starting with 34fd0e7729096ad28394e67daea9e810234dbbc95f64b026e7eaa2eb8fbe55b5 not found: ID does not exist"
Nov 01 04:12:11.717465 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:11.717421    4930 scope.go:122] "RemoveContainer" containerID="9eccbec3b560c1dba54f4bf5513d6e12e7f236b090d392b5620ab81ffc803b81"
Nov 01 04:12:11.717695 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:12:11.717675    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9eccbec3b560c1dba54f4bf5513d6e12e7f236b090d392b5620ab81ffc803b81\": container with ID starting with 9eccbec3b560c1dba54f4bf5513d6e12e7f236b090d392b5620ab81ffc803b81 not found: ID does not exist" containerID="9eccbec3b560c1dba54f4bf5513d6e12e7f236b090d392b5620ab81ffc803b81"
Nov 01 04:12:11.717741 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:11.717700    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eccbec3b560c1dba54f4bf5513d6e12e7f236b090d392b5620ab81ffc803b81"} err="failed to get container status \"9eccbec3b560c1dba54f4bf5513d6e12e7f236b090d392b5620ab81ffc803b81\": rpc error: code = NotFound desc = could not find container \"9eccbec3b560c1dba54f4bf5513d6e12e7f236b090d392b5620ab81ffc803b81\": container with ID starting with 9eccbec3b560c1dba54f4bf5513d6e12e7f236b090d392b5620ab81ffc803b81 not found: ID does not exist"
Nov 01 04:12:11.863429 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:11.863400    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"]
Nov 01 04:12:11.868253 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:11.868224    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ca6b5505-openshift-default-769b66775-2zgl5"]
Nov 01 04:12:12.628358 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:12.628324    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:12:12.797067 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:12.797036    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:12:13.011108 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:13.011015    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:12:13.619093 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:13.619059    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="243fb857-1985-4313-9d36-1cc66cb6e503" path="/var/lib/kubelet/pods/243fb857-1985-4313-9d36-1cc66cb6e503/volumes"
Nov 01 04:12:15.909367 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:15.909331    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:12:16.083234 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:16.083194    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:12:16.321602 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:16.321514    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-84f12f-65bd455fc9-kq25h"]
Nov 01 04:12:16.321950 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:16.321935    4930 cpu_manager.go:414] "containerMap: removing container" podUID="243fb857-1985-4313-9d36-1cc66cb6e503" containerName="istio-proxy"
Nov 01 04:12:16.321991 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:16.321952    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="243fb857-1985-4313-9d36-1cc66cb6e503" containerName="istio-proxy"
Nov 01 04:12:16.321991 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:16.321974    4930 cpu_manager.go:414] "containerMap: removing container" podUID="243fb857-1985-4313-9d36-1cc66cb6e503" containerName="istio-proxy"
Nov 01 04:12:16.321991 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:16.321980    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="243fb857-1985-4313-9d36-1cc66cb6e503" containerName="istio-proxy"
Nov 01 04:12:16.322094 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:16.322063    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="243fb857-1985-4313-9d36-1cc66cb6e503" containerName="istio-proxy"
Nov 01 04:12:16.325332 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:16.325315    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-84f12f-65bd455fc9-kq25h"
Nov 01 04:12:16.379917 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:16.379895    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-84f12f-dockercfg-mj2zn\""
Nov 01 04:12:16.381871 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:16.381844    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-84f12f-65bd455fc9-kq25h"]
Nov 01 04:12:16.395621 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:16.395600    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-84f12f\""
Nov 01 04:12:16.443664 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:16.443630    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e521d66-d7d5-497c-9c34-4c2ce9e821c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-84f12f-65bd455fc9-kq25h\" (UID: \"7e521d66-d7d5-497c-9c34-4c2ce9e821c7\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-84f12f-65bd455fc9-kq25h"
Nov 01 04:12:16.443664 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:16.443663    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd9px\" (UniqueName: \"kubernetes.io/projected/7e521d66-d7d5-497c-9c34-4c2ce9e821c7-kube-api-access-gd9px\") pod \"payload-pre-processing-e2e-authcfg-http-84f12f-65bd455fc9-kq25h\" (UID: \"7e521d66-d7d5-497c-9c34-4c2ce9e821c7\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-84f12f-65bd455fc9-kq25h"
Nov 01 04:12:16.505583 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:16.504034    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-84f12f-855d9644d5-jd96c"]
Nov 01 04:12:16.505583 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:16.505301    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="243fb857-1985-4313-9d36-1cc66cb6e503" containerName="istio-proxy"
Nov 01 04:12:16.510690 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:16.510659    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-84f12f-855d9644d5-jd96c"
Nov 01 04:12:16.544203 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:16.544168    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-84f12f-855d9644d5-jd96c"]
Nov 01 04:12:16.544360 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:16.544339    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e521d66-d7d5-497c-9c34-4c2ce9e821c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-84f12f-65bd455fc9-kq25h\" (UID: \"7e521d66-d7d5-497c-9c34-4c2ce9e821c7\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-84f12f-65bd455fc9-kq25h"
Nov 01 04:12:16.544428 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:16.544367    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd9px\" (UniqueName: \"kubernetes.io/projected/7e521d66-d7d5-497c-9c34-4c2ce9e821c7-kube-api-access-gd9px\") pod \"payload-pre-processing-e2e-authcfg-http-84f12f-65bd455fc9-kq25h\" (UID: \"7e521d66-d7d5-497c-9c34-4c2ce9e821c7\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-84f12f-65bd455fc9-kq25h"
Nov 01 04:12:16.545155 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:16.545134    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e521d66-d7d5-497c-9c34-4c2ce9e821c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-84f12f-65bd455fc9-kq25h\" (UID: \"7e521d66-d7d5-497c-9c34-4c2ce9e821c7\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-84f12f-65bd455fc9-kq25h"
Nov 01 04:12:16.605638 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:16.605610    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd9px\" (UniqueName: \"kubernetes.io/projected/7e521d66-d7d5-497c-9c34-4c2ce9e821c7-kube-api-access-gd9px\") pod \"payload-pre-processing-e2e-authcfg-http-84f12f-65bd455fc9-kq25h\" (UID: \"7e521d66-d7d5-497c-9c34-4c2ce9e821c7\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-84f12f-65bd455fc9-kq25h"
Nov 01 04:12:16.638520 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:16.638493    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-84f12f-65bd455fc9-kq25h"
Nov 01 04:12:16.645802 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:16.645783    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6712d7f0-b182-4d1f-9d0e-e92e55282f66-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-84f12f-855d9644d5-jd96c\" (UID: \"6712d7f0-b182-4d1f-9d0e-e92e55282f66\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-84f12f-855d9644d5-jd96c"
Nov 01 04:12:16.645885 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:16.645833    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxmk8\" (UniqueName: \"kubernetes.io/projected/6712d7f0-b182-4d1f-9d0e-e92e55282f66-kube-api-access-pxmk8\") pod \"payload-processing-e2e-authcfg-http-84f12f-855d9644d5-jd96c\" (UID: \"6712d7f0-b182-4d1f-9d0e-e92e55282f66\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-84f12f-855d9644d5-jd96c"
Nov 01 04:12:16.747116 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:16.747084    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6712d7f0-b182-4d1f-9d0e-e92e55282f66-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-84f12f-855d9644d5-jd96c\" (UID: \"6712d7f0-b182-4d1f-9d0e-e92e55282f66\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-84f12f-855d9644d5-jd96c"
Nov 01 04:12:16.747276 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:16.747138    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxmk8\" (UniqueName: \"kubernetes.io/projected/6712d7f0-b182-4d1f-9d0e-e92e55282f66-kube-api-access-pxmk8\") pod \"payload-processing-e2e-authcfg-http-84f12f-855d9644d5-jd96c\" (UID: \"6712d7f0-b182-4d1f-9d0e-e92e55282f66\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-84f12f-855d9644d5-jd96c"
Nov 01 04:12:16.747841 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:16.747817    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6712d7f0-b182-4d1f-9d0e-e92e55282f66-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-84f12f-855d9644d5-jd96c\" (UID: \"6712d7f0-b182-4d1f-9d0e-e92e55282f66\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-84f12f-855d9644d5-jd96c"
Nov 01 04:12:16.809587 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:16.809539    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxmk8\" (UniqueName: \"kubernetes.io/projected/6712d7f0-b182-4d1f-9d0e-e92e55282f66-kube-api-access-pxmk8\") pod \"payload-processing-e2e-authcfg-http-84f12f-855d9644d5-jd96c\" (UID: \"6712d7f0-b182-4d1f-9d0e-e92e55282f66\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-84f12f-855d9644d5-jd96c"
Nov 01 04:12:16.826165 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:16.826142    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-84f12f-855d9644d5-jd96c"
Nov 01 04:12:17.012013 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:17.011986    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-84f12f-65bd455fc9-kq25h"]
Nov 01 04:12:17.013688 ip-10-0-141-122 kubenswrapper[4930]: W1101 04:12:17.013657    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e521d66_d7d5_497c_9c34_4c2ce9e821c7.slice/crio-ff76d789719b96262a9d02d7853a26688a09be942ce36fee1d679c0257d7c89f WatchSource:0}: Error finding container ff76d789719b96262a9d02d7853a26688a09be942ce36fee1d679c0257d7c89f: Status 404 returned error can't find the container with id ff76d789719b96262a9d02d7853a26688a09be942ce36fee1d679c0257d7c89f
Nov 01 04:12:17.204980 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:17.204949    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-84f12f-855d9644d5-jd96c"]
Nov 01 04:12:17.205946 ip-10-0-141-122 kubenswrapper[4930]: W1101 04:12:17.205917    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6712d7f0_b182_4d1f_9d0e_e92e55282f66.slice/crio-369b1faa00dd0f37192bb3988da38c403cedbb0044f059628e75c7eca3a2ad25 WatchSource:0}: Error finding container 369b1faa00dd0f37192bb3988da38c403cedbb0044f059628e75c7eca3a2ad25: Status 404 returned error can't find the container with id 369b1faa00dd0f37192bb3988da38c403cedbb0044f059628e75c7eca3a2ad25
Nov 01 04:12:17.735344 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:17.735305    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-84f12f-65bd455fc9-kq25h" event={"ID":"7e521d66-d7d5-497c-9c34-4c2ce9e821c7","Type":"ContainerStarted","Data":"9f5de54097e67638204e7d343462a90c4f5f6903a30de60a126214772895b116"}
Nov 01 04:12:17.735528 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:17.735454    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-84f12f-65bd455fc9-kq25h"
Nov 01 04:12:17.735528 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:17.735512    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-84f12f-65bd455fc9-kq25h" event={"ID":"7e521d66-d7d5-497c-9c34-4c2ce9e821c7","Type":"ContainerStarted","Data":"ff76d789719b96262a9d02d7853a26688a09be942ce36fee1d679c0257d7c89f"}
Nov 01 04:12:17.736948 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:17.736926    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-84f12f-855d9644d5-jd96c" event={"ID":"6712d7f0-b182-4d1f-9d0e-e92e55282f66","Type":"ContainerStarted","Data":"7eaa9e8fb730e9c0fb312910ff57226f3b54bd367fc347af12199a10a797a185"}
Nov 01 04:12:17.737057 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:17.736969    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-84f12f-855d9644d5-jd96c" event={"ID":"6712d7f0-b182-4d1f-9d0e-e92e55282f66","Type":"ContainerStarted","Data":"369b1faa00dd0f37192bb3988da38c403cedbb0044f059628e75c7eca3a2ad25"}
Nov 01 04:12:17.737057 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:17.736997    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-84f12f-855d9644d5-jd96c"
Nov 01 04:12:17.850288 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:17.850240    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-84f12f-65bd455fc9-kq25h" podStartSLOduration=1.85022826 podStartE2EDuration="1.85022826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:12:17.849895532 +0000 UTC m=+3942.372902120" watchObservedRunningTime="2026-11-01 04:12:17.85022826 +0000 UTC m=+3942.373234845"
Nov 01 04:12:17.969767 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:17.969698    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-84f12f-855d9644d5-jd96c" podStartSLOduration=1.9696789479999999 podStartE2EDuration="1.969678948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:12:17.964133547 +0000 UTC m=+3942.487140136" watchObservedRunningTime="2026-11-01 04:12:17.969678948 +0000 UTC m=+3942.492685538"
Nov 01 04:12:18.812436 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:18.812399    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:12:19.004779 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:19.004733    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:12:19.122647 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:12:19.122606    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 04:12:19.162953 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:19.162922    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:12:23.884498 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:23.884453    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:12:28.746061 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:28.746030    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-84f12f-855d9644d5-jd96c"
Nov 01 04:12:28.746462 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:28.746245    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-84f12f-65bd455fc9-kq25h"
Nov 01 04:12:28.816084 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:28.816050    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:12:33.673366 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:33.673327    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:12:36.851999 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:36.851968    4930 generic.go:356] "Generic (PLEG): container finished" podID="259da31c-7d23-434a-866a-e94d82ee82bd" containerID="375fdeb4241dfdf2217ac650b7b4fd283bf93522bf0a6655f3f001b5401dd35a" exitCode=0
Nov 01 04:12:36.852433 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:36.852020    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684" event={"ID":"259da31c-7d23-434a-866a-e94d82ee82bd","Type":"ContainerDied","Data":"375fdeb4241dfdf2217ac650b7b4fd283bf93522bf0a6655f3f001b5401dd35a"}
Nov 01 04:12:36.852433 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:36.852362    4930 scope.go:122] "RemoveContainer" containerID="375fdeb4241dfdf2217ac650b7b4fd283bf93522bf0a6655f3f001b5401dd35a"
Nov 01 04:12:36.853639 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:36.853609    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:12:36.853708 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:36.853682    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:12:36.853747 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:36.853718    4930 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 04:12:37.585618 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:37.585548    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:12:37.862039 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:37.861939    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684" event={"ID":"259da31c-7d23-434a-866a-e94d82ee82bd","Type":"ContainerStarted","Data":"de8665eea0b92f61b5644a4580651a9f78630449efe61a17584e7ff1067a720d"}
Nov 01 04:12:38.256677 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:38.256598    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"
Nov 01 04:12:38.257832 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:38.257804    4930 patch_prober.go:29] interesting pod/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body=
Nov 01 04:12:38.257967 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:38.257855    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684" podUID="259da31c-7d23-434a-866a-e94d82ee82bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused"
Nov 01 04:12:38.805049 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:38.805011    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:12:39.256835 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:39.256796    4930 patch_prober.go:29] interesting pod/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body=
Nov 01 04:12:39.257354 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:39.256850    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684" podUID="259da31c-7d23-434a-866a-e94d82ee82bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused"
Nov 01 04:12:40.256639 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:40.256606    4930 patch_prober.go:29] interesting pod/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body=
Nov 01 04:12:40.256819 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:40.256658    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684" podUID="259da31c-7d23-434a-866a-e94d82ee82bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused"
Nov 01 04:12:40.692026 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:40.691983    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"
Nov 01 04:12:41.257064 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:41.257027    4930 patch_prober.go:29] interesting pod/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body=
Nov 01 04:12:41.257262 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:41.257104    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684" podUID="259da31c-7d23-434a-866a-e94d82ee82bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused"
Nov 01 04:12:41.292290 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:41.292252    4930 scope.go:122] "RemoveContainer" containerID="b924be943a87f5481cbab24e4532389636f19ddc94c333d5e1bfe6141d54e030"
Nov 01 04:12:41.302831 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:41.302809    4930 scope.go:122] "RemoveContainer" containerID="35cfee4777a6a1cdd84299451097e79bdeada1cf6d7c838523000edd77afaa0a"
Nov 01 04:12:42.257059 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:42.257023    4930 patch_prober.go:29] interesting pod/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body=
Nov 01 04:12:42.257470 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:42.257087    4930 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684" podUID="259da31c-7d23-434a-866a-e94d82ee82bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused"
Nov 01 04:12:43.261218 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:43.261188    4930 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"
Nov 01 04:12:43.262217 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:43.262197    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"
Nov 01 04:12:43.608930 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:43.608895    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:12:48.469888 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:48.469852    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:12:49.121058 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:12:49.121018    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 04:12:49.435982 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:49.435901    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:12:53.282907 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:53.282868    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:12:58.182420 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:58.182368    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:12:59.182569 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:12:59.182532    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:13:02.984677 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:13:02.984640    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:13:07.776375 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:13:07.776334    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:13:12.581213 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:13:12.581169    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:13:17.388830 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:13:17.388746    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:13:19.121719 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:13:19.121684    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 04:13:22.271743 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:13:22.271700    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:13:26.978397 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:13:26.978353    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:13:31.769567 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:13:31.769533    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:13:36.713062 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:13:36.713027    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:13:41.505650 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:13:41.505612    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:13:46.375514 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:13:46.375482    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:13:49.121198 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:13:49.121161    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 04:13:49.780163 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:13:49.780128    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:13:51.196677 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:13:51.196642    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:13:55.998524 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:13:55.998485    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:13:58.265540 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:13:58.265502    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-84f12f-t65sp"]
Nov 01 04:13:58.269234 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:13:58.269213    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-84f12f-t65sp"
Nov 01 04:13:58.321333 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:13:58.321309    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 04:13:58.370943 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:13:58.370915    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:13:58.370943 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:13:58.370911    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 04:13:58.371150 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:13:58.371022    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7br8\""
Nov 01 04:13:58.373442 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:13:58.373408    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-84f12f-t65sp"]
Nov 01 04:13:58.446304 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:13:58.446276    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b56249c-3ed6-4016-9daf-7f6942f39355-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-84f12f-t65sp\" (UID: \"4b56249c-3ed6-4016-9daf-7f6942f39355\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-84f12f-t65sp"
Nov 01 04:13:58.446454 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:13:58.446314    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89rkh\" (UniqueName: \"kubernetes.io/projected/4b56249c-3ed6-4016-9daf-7f6942f39355-kube-api-access-89rkh\") pod \"maas-api-revoke-keys-e2e-authcfg-http-84f12f-t65sp\" (UID: \"4b56249c-3ed6-4016-9daf-7f6942f39355\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-84f12f-t65sp"
Nov 01 04:13:58.446522 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:13:58.446460    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b56249c-3ed6-4016-9daf-7f6942f39355-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-84f12f-t65sp\" (UID: \"4b56249c-3ed6-4016-9daf-7f6942f39355\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-84f12f-t65sp"
Nov 01 04:13:58.547696 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:13:58.547612    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b56249c-3ed6-4016-9daf-7f6942f39355-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-84f12f-t65sp\" (UID: \"4b56249c-3ed6-4016-9daf-7f6942f39355\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-84f12f-t65sp"
Nov 01 04:13:58.547864 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:13:58.547720    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b56249c-3ed6-4016-9daf-7f6942f39355-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-84f12f-t65sp\" (UID: \"4b56249c-3ed6-4016-9daf-7f6942f39355\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-84f12f-t65sp"
Nov 01 04:13:58.547864 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:13:58.547763    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89rkh\" (UniqueName: \"kubernetes.io/projected/4b56249c-3ed6-4016-9daf-7f6942f39355-kube-api-access-89rkh\") pod \"maas-api-revoke-keys-e2e-authcfg-http-84f12f-t65sp\" (UID: \"4b56249c-3ed6-4016-9daf-7f6942f39355\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-84f12f-t65sp"
Nov 01 04:13:58.548422 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:13:58.548396    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b56249c-3ed6-4016-9daf-7f6942f39355-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-84f12f-t65sp\" (UID: \"4b56249c-3ed6-4016-9daf-7f6942f39355\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-84f12f-t65sp"
Nov 01 04:13:58.550044 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:13:58.550023    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b56249c-3ed6-4016-9daf-7f6942f39355-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-84f12f-t65sp\" (UID: \"4b56249c-3ed6-4016-9daf-7f6942f39355\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-84f12f-t65sp"
Nov 01 04:13:58.608012 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:13:58.607983    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89rkh\" (UniqueName: \"kubernetes.io/projected/4b56249c-3ed6-4016-9daf-7f6942f39355-kube-api-access-89rkh\") pod \"maas-api-revoke-keys-e2e-authcfg-http-84f12f-t65sp\" (UID: \"4b56249c-3ed6-4016-9daf-7f6942f39355\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-84f12f-t65sp"
Nov 01 04:13:58.882062 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:13:58.882034    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-84f12f-t65sp"
Nov 01 04:13:59.253535 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:13:59.253503    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-84f12f-t65sp"]
Nov 01 04:13:59.255581 ip-10-0-141-122 kubenswrapper[4930]: W1101 04:13:59.255523    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b56249c_3ed6_4016_9daf_7f6942f39355.slice/crio-64931decb962c1b7c5aee645a38a30d77204db8a4ae2bf5b706bff1cc11e1c46 WatchSource:0}: Error finding container 64931decb962c1b7c5aee645a38a30d77204db8a4ae2bf5b706bff1cc11e1c46: Status 404 returned error can't find the container with id 64931decb962c1b7c5aee645a38a30d77204db8a4ae2bf5b706bff1cc11e1c46
Nov 01 04:13:59.351626 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:13:59.351597    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-84f12f-t65sp" event={"ID":"4b56249c-3ed6-4016-9daf-7f6942f39355","Type":"ContainerStarted","Data":"64931decb962c1b7c5aee645a38a30d77204db8a4ae2bf5b706bff1cc11e1c46"}
Nov 01 04:13:59.513760 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:13:59.513691    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:14:00.359329 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:00.359297    4930 generic.go:356] "Generic (PLEG): container finished" podID="4b56249c-3ed6-4016-9daf-7f6942f39355" containerID="8bd39c4a9520add887b2ac1d16e6f70579e410b213bacdb9961eade7e87d353c" exitCode=0
Nov 01 04:14:00.359781 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:00.359389    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-84f12f-t65sp" event={"ID":"4b56249c-3ed6-4016-9daf-7f6942f39355","Type":"ContainerDied","Data":"8bd39c4a9520add887b2ac1d16e6f70579e410b213bacdb9961eade7e87d353c"}
Nov 01 04:14:00.917003 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:00.916962    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:14:01.509897 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:01.509874    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-84f12f-t65sp"
Nov 01 04:14:01.577701 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:01.577670    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b56249c-3ed6-4016-9daf-7f6942f39355-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b56249c-3ed6-4016-9daf-7f6942f39355-maas-api-xks-ca\") pod \"4b56249c-3ed6-4016-9daf-7f6942f39355\" (UID: \"4b56249c-3ed6-4016-9daf-7f6942f39355\") "
Nov 01 04:14:01.577906 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:01.577727    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b56249c-3ed6-4016-9daf-7f6942f39355-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b56249c-3ed6-4016-9daf-7f6942f39355-maas-api-openshift-ca\") pod \"4b56249c-3ed6-4016-9daf-7f6942f39355\" (UID: \"4b56249c-3ed6-4016-9daf-7f6942f39355\") "
Nov 01 04:14:01.577906 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:01.577896    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b56249c-3ed6-4016-9daf-7f6942f39355-kube-api-access-89rkh\" (UniqueName: \"kubernetes.io/projected/4b56249c-3ed6-4016-9daf-7f6942f39355-kube-api-access-89rkh\") pod \"4b56249c-3ed6-4016-9daf-7f6942f39355\" (UID: \"4b56249c-3ed6-4016-9daf-7f6942f39355\") "
Nov 01 04:14:01.578173 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:01.578136    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b56249c-3ed6-4016-9daf-7f6942f39355-maas-api-openshift-ca" pod "4b56249c-3ed6-4016-9daf-7f6942f39355" (UID: "4b56249c-3ed6-4016-9daf-7f6942f39355"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:14:01.579996 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:01.579967    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b56249c-3ed6-4016-9daf-7f6942f39355-maas-api-xks-ca" pod "4b56249c-3ed6-4016-9daf-7f6942f39355" (UID: "4b56249c-3ed6-4016-9daf-7f6942f39355"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:14:01.580152 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:01.580131    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b56249c-3ed6-4016-9daf-7f6942f39355-kube-api-access-89rkh" pod "4b56249c-3ed6-4016-9daf-7f6942f39355" (UID: "4b56249c-3ed6-4016-9daf-7f6942f39355"). InnerVolumeSpecName "kube-api-access-89rkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:14:01.678955 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:01.678921    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89rkh\" (UniqueName: \"kubernetes.io/projected/4b56249c-3ed6-4016-9daf-7f6942f39355-kube-api-access-89rkh\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:14:01.678955 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:01.678947    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b56249c-3ed6-4016-9daf-7f6942f39355-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:14:01.678955 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:01.678957    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b56249c-3ed6-4016-9daf-7f6942f39355-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:14:02.375421 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:02.375387    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-84f12f-t65sp"
Nov 01 04:14:02.375596 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:02.375420    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-84f12f-t65sp" event={"ID":"4b56249c-3ed6-4016-9daf-7f6942f39355","Type":"ContainerDied","Data":"64931decb962c1b7c5aee645a38a30d77204db8a4ae2bf5b706bff1cc11e1c46"}
Nov 01 04:14:02.375596 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:02.375477    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64931decb962c1b7c5aee645a38a30d77204db8a4ae2bf5b706bff1cc11e1c46"
Nov 01 04:14:05.781879 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:05.781842    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:14:09.801751 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:09.801716    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-84f12f-855d9644d5-jd96c"]
Nov 01 04:14:09.907315 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:09.907284    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-84f12f-65bd455fc9-kq25h"]
Nov 01 04:14:10.172068 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:10.172021    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:14:10.324995 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:10.324958    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:14:10.582953 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:10.582863    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:14:12.392027 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:12.391991    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:14:12.594451 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:12.594398    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:14:14.802573 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:14.802520    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-84f12f-855d9644d5-jd96c" podUID="6712d7f0-b182-4d1f-9d0e-e92e55282f66" containerName="payload-processing" containerID="cri-o://7eaa9e8fb730e9c0fb312910ff57226f3b54bd367fc347af12199a10a797a185" gracePeriod=25
Nov 01 04:14:14.908419 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:14.908382    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-84f12f-65bd455fc9-kq25h" podUID="7e521d66-d7d5-497c-9c34-4c2ce9e821c7" containerName="payload-pre-processing" containerID="cri-o://9f5de54097e67638204e7d343462a90c4f5f6903a30de60a126214772895b116" gracePeriod=25
Nov 01 04:14:15.064365 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:15.064300    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-84f12f-855d9644d5-jd96c"
Nov 01 04:14:15.110255 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:15.110220    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6712d7f0-b182-4d1f-9d0e-e92e55282f66-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6712d7f0-b182-4d1f-9d0e-e92e55282f66-plugins-config-volume\") pod \"6712d7f0-b182-4d1f-9d0e-e92e55282f66\" (UID: \"6712d7f0-b182-4d1f-9d0e-e92e55282f66\") "
Nov 01 04:14:15.110475 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:15.110456    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6712d7f0-b182-4d1f-9d0e-e92e55282f66-kube-api-access-pxmk8\" (UniqueName: \"kubernetes.io/projected/6712d7f0-b182-4d1f-9d0e-e92e55282f66-kube-api-access-pxmk8\") pod \"6712d7f0-b182-4d1f-9d0e-e92e55282f66\" (UID: \"6712d7f0-b182-4d1f-9d0e-e92e55282f66\") "
Nov 01 04:14:15.110638 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:15.110613    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6712d7f0-b182-4d1f-9d0e-e92e55282f66-plugins-config-volume" pod "6712d7f0-b182-4d1f-9d0e-e92e55282f66" (UID: "6712d7f0-b182-4d1f-9d0e-e92e55282f66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:14:15.110838 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:15.110816    4930 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6712d7f0-b182-4d1f-9d0e-e92e55282f66-plugins-config-volume\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:14:15.112458 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:15.112431    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6712d7f0-b182-4d1f-9d0e-e92e55282f66-kube-api-access-pxmk8" pod "6712d7f0-b182-4d1f-9d0e-e92e55282f66" (UID: "6712d7f0-b182-4d1f-9d0e-e92e55282f66"). InnerVolumeSpecName "kube-api-access-pxmk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:14:15.159915 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:15.159894    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-84f12f-65bd455fc9-kq25h"
Nov 01 04:14:15.212072 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:15.212051    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e521d66-d7d5-497c-9c34-4c2ce9e821c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e521d66-d7d5-497c-9c34-4c2ce9e821c7-plugins-config-volume\") pod \"7e521d66-d7d5-497c-9c34-4c2ce9e821c7\" (UID: \"7e521d66-d7d5-497c-9c34-4c2ce9e821c7\") "
Nov 01 04:14:15.212197 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:15.212143    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e521d66-d7d5-497c-9c34-4c2ce9e821c7-kube-api-access-gd9px\" (UniqueName: \"kubernetes.io/projected/7e521d66-d7d5-497c-9c34-4c2ce9e821c7-kube-api-access-gd9px\") pod \"7e521d66-d7d5-497c-9c34-4c2ce9e821c7\" (UID: \"7e521d66-d7d5-497c-9c34-4c2ce9e821c7\") "
Nov 01 04:14:15.212409 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:15.212385    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e521d66-d7d5-497c-9c34-4c2ce9e821c7-plugins-config-volume" pod "7e521d66-d7d5-497c-9c34-4c2ce9e821c7" (UID: "7e521d66-d7d5-497c-9c34-4c2ce9e821c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:14:15.212500 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:15.212394    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxmk8\" (UniqueName: \"kubernetes.io/projected/6712d7f0-b182-4d1f-9d0e-e92e55282f66-kube-api-access-pxmk8\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:14:15.214132 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:15.214111    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e521d66-d7d5-497c-9c34-4c2ce9e821c7-kube-api-access-gd9px" pod "7e521d66-d7d5-497c-9c34-4c2ce9e821c7" (UID: "7e521d66-d7d5-497c-9c34-4c2ce9e821c7"). InnerVolumeSpecName "kube-api-access-gd9px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:14:15.312877 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:15.312848    4930 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e521d66-d7d5-497c-9c34-4c2ce9e821c7-plugins-config-volume\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:14:15.312877 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:15.312876    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd9px\" (UniqueName: \"kubernetes.io/projected/7e521d66-d7d5-497c-9c34-4c2ce9e821c7-kube-api-access-gd9px\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:14:15.450155 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:15.450125    4930 generic.go:356] "Generic (PLEG): container finished" podID="7e521d66-d7d5-497c-9c34-4c2ce9e821c7" containerID="9f5de54097e67638204e7d343462a90c4f5f6903a30de60a126214772895b116" exitCode=0
Nov 01 04:14:15.450310 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:15.450183    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-84f12f-65bd455fc9-kq25h"
Nov 01 04:14:15.450310 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:15.450207    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-84f12f-65bd455fc9-kq25h" event={"ID":"7e521d66-d7d5-497c-9c34-4c2ce9e821c7","Type":"ContainerDied","Data":"9f5de54097e67638204e7d343462a90c4f5f6903a30de60a126214772895b116"}
Nov 01 04:14:15.450310 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:15.450291    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-84f12f-65bd455fc9-kq25h" event={"ID":"7e521d66-d7d5-497c-9c34-4c2ce9e821c7","Type":"ContainerDied","Data":"ff76d789719b96262a9d02d7853a26688a09be942ce36fee1d679c0257d7c89f"}
Nov 01 04:14:15.450465 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:15.450325    4930 scope.go:122] "RemoveContainer" containerID="9f5de54097e67638204e7d343462a90c4f5f6903a30de60a126214772895b116"
Nov 01 04:14:15.451809 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:15.451775    4930 generic.go:356] "Generic (PLEG): container finished" podID="6712d7f0-b182-4d1f-9d0e-e92e55282f66" containerID="7eaa9e8fb730e9c0fb312910ff57226f3b54bd367fc347af12199a10a797a185" exitCode=0
Nov 01 04:14:15.451929 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:15.451835    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-84f12f-855d9644d5-jd96c"
Nov 01 04:14:15.451929 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:15.451851    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-84f12f-855d9644d5-jd96c" event={"ID":"6712d7f0-b182-4d1f-9d0e-e92e55282f66","Type":"ContainerDied","Data":"7eaa9e8fb730e9c0fb312910ff57226f3b54bd367fc347af12199a10a797a185"}
Nov 01 04:14:15.451929 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:15.451906    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-84f12f-855d9644d5-jd96c" event={"ID":"6712d7f0-b182-4d1f-9d0e-e92e55282f66","Type":"ContainerDied","Data":"369b1faa00dd0f37192bb3988da38c403cedbb0044f059628e75c7eca3a2ad25"}
Nov 01 04:14:15.461067 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:15.461050    4930 scope.go:122] "RemoveContainer" containerID="9f5de54097e67638204e7d343462a90c4f5f6903a30de60a126214772895b116"
Nov 01 04:14:15.461422 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:14:15.461399    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f5de54097e67638204e7d343462a90c4f5f6903a30de60a126214772895b116\": container with ID starting with 9f5de54097e67638204e7d343462a90c4f5f6903a30de60a126214772895b116 not found: ID does not exist" containerID="9f5de54097e67638204e7d343462a90c4f5f6903a30de60a126214772895b116"
Nov 01 04:14:15.461514 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:15.461427    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f5de54097e67638204e7d343462a90c4f5f6903a30de60a126214772895b116"} err="failed to get container status \"9f5de54097e67638204e7d343462a90c4f5f6903a30de60a126214772895b116\": rpc error: code = NotFound desc = could not find container \"9f5de54097e67638204e7d343462a90c4f5f6903a30de60a126214772895b116\": container with ID starting with 9f5de54097e67638204e7d343462a90c4f5f6903a30de60a126214772895b116 not found: ID does not exist"
Nov 01 04:14:15.461514 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:15.461443    4930 scope.go:122] "RemoveContainer" containerID="7eaa9e8fb730e9c0fb312910ff57226f3b54bd367fc347af12199a10a797a185"
Nov 01 04:14:15.469986 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:15.469967    4930 scope.go:122] "RemoveContainer" containerID="7eaa9e8fb730e9c0fb312910ff57226f3b54bd367fc347af12199a10a797a185"
Nov 01 04:14:15.470257 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:14:15.470237    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eaa9e8fb730e9c0fb312910ff57226f3b54bd367fc347af12199a10a797a185\": container with ID starting with 7eaa9e8fb730e9c0fb312910ff57226f3b54bd367fc347af12199a10a797a185 not found: ID does not exist" containerID="7eaa9e8fb730e9c0fb312910ff57226f3b54bd367fc347af12199a10a797a185"
Nov 01 04:14:15.470321 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:15.470263    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eaa9e8fb730e9c0fb312910ff57226f3b54bd367fc347af12199a10a797a185"} err="failed to get container status \"7eaa9e8fb730e9c0fb312910ff57226f3b54bd367fc347af12199a10a797a185\": rpc error: code = NotFound desc = could not find container \"7eaa9e8fb730e9c0fb312910ff57226f3b54bd367fc347af12199a10a797a185\": container with ID starting with 7eaa9e8fb730e9c0fb312910ff57226f3b54bd367fc347af12199a10a797a185 not found: ID does not exist"
Nov 01 04:14:15.619119 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:15.619095    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-84f12f-855d9644d5-jd96c"]
Nov 01 04:14:15.619266 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:15.619128    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-84f12f-855d9644d5-jd96c"]
Nov 01 04:14:15.725214 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:15.725148    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-84f12f-65bd455fc9-kq25h"]
Nov 01 04:14:15.729884 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:15.729862    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-84f12f-65bd455fc9-kq25h"]
Nov 01 04:14:17.619571 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:17.619528    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6712d7f0-b182-4d1f-9d0e-e92e55282f66" path="/var/lib/kubelet/pods/6712d7f0-b182-4d1f-9d0e-e92e55282f66/volumes"
Nov 01 04:14:17.620166 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:17.620144    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e521d66-d7d5-497c-9c34-4c2ce9e821c7" path="/var/lib/kubelet/pods/7e521d66-d7d5-497c-9c34-4c2ce9e821c7/volumes"
Nov 01 04:14:19.121532 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:14:19.121501    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 04:14:19.856037 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:19.855992    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:14:21.367949 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:21.367910    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:14:21.641957 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:21.641866    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:14:21.981157 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:21.981078    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-84f12f-t65sp"]
Nov 01 04:14:22.030484 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:22.030452    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-84f12f-t65sp"]
Nov 01 04:14:22.425602 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:22.425566    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:14:23.619324 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:23.619292    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b56249c-3ed6-4016-9daf-7f6942f39355" path="/var/lib/kubelet/pods/4b56249c-3ed6-4016-9daf-7f6942f39355/volumes"
Nov 01 04:14:28.802203 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:28.802167    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:14:28.947247 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:28.947208    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:14:29.137547 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:29.137511    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"]
Nov 01 04:14:29.137821 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:29.137795    4930 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684" podUID="259da31c-7d23-434a-866a-e94d82ee82bd" containerName="istio-proxy" containerID="cri-o://de8665eea0b92f61b5644a4580651a9f78630449efe61a17584e7ff1067a720d" gracePeriod=30
Nov 01 04:14:29.152152 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:29.152112    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:14:34.393505 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:34.393483    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"
Nov 01 04:14:34.496432 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:34.496369    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/259da31c-7d23-434a-866a-e94d82ee82bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/259da31c-7d23-434a-866a-e94d82ee82bd-istio-envoy\") pod \"259da31c-7d23-434a-866a-e94d82ee82bd\" (UID: \"259da31c-7d23-434a-866a-e94d82ee82bd\") "
Nov 01 04:14:34.496432 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:34.496403    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/259da31c-7d23-434a-866a-e94d82ee82bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/259da31c-7d23-434a-866a-e94d82ee82bd-istio-data\") pod \"259da31c-7d23-434a-866a-e94d82ee82bd\" (UID: \"259da31c-7d23-434a-866a-e94d82ee82bd\") "
Nov 01 04:14:34.496432 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:34.496423    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/259da31c-7d23-434a-866a-e94d82ee82bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/259da31c-7d23-434a-866a-e94d82ee82bd-istio-podinfo\") pod \"259da31c-7d23-434a-866a-e94d82ee82bd\" (UID: \"259da31c-7d23-434a-866a-e94d82ee82bd\") "
Nov 01 04:14:34.496732 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:34.496455    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/259da31c-7d23-434a-866a-e94d82ee82bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/259da31c-7d23-434a-866a-e94d82ee82bd-workload-certs\") pod \"259da31c-7d23-434a-866a-e94d82ee82bd\" (UID: \"259da31c-7d23-434a-866a-e94d82ee82bd\") "
Nov 01 04:14:34.496732 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:34.496478    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/259da31c-7d23-434a-866a-e94d82ee82bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/259da31c-7d23-434a-866a-e94d82ee82bd-workload-socket\") pod \"259da31c-7d23-434a-866a-e94d82ee82bd\" (UID: \"259da31c-7d23-434a-866a-e94d82ee82bd\") "
Nov 01 04:14:34.496732 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:34.496527    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/259da31c-7d23-434a-866a-e94d82ee82bd-kube-api-access-89mnc\" (UniqueName: \"kubernetes.io/projected/259da31c-7d23-434a-866a-e94d82ee82bd-kube-api-access-89mnc\") pod \"259da31c-7d23-434a-866a-e94d82ee82bd\" (UID: \"259da31c-7d23-434a-866a-e94d82ee82bd\") "
Nov 01 04:14:34.496732 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:34.496592    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/259da31c-7d23-434a-866a-e94d82ee82bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/259da31c-7d23-434a-866a-e94d82ee82bd-credential-socket\") pod \"259da31c-7d23-434a-866a-e94d82ee82bd\" (UID: \"259da31c-7d23-434a-866a-e94d82ee82bd\") "
Nov 01 04:14:34.496732 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:34.496619    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/259da31c-7d23-434a-866a-e94d82ee82bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/259da31c-7d23-434a-866a-e94d82ee82bd-istiod-ca-cert\") pod \"259da31c-7d23-434a-866a-e94d82ee82bd\" (UID: \"259da31c-7d23-434a-866a-e94d82ee82bd\") "
Nov 01 04:14:34.496732 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:34.496701    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/259da31c-7d23-434a-866a-e94d82ee82bd-istio-data" pod "259da31c-7d23-434a-866a-e94d82ee82bd" (UID: "259da31c-7d23-434a-866a-e94d82ee82bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:14:34.496732 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:34.496706    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/259da31c-7d23-434a-866a-e94d82ee82bd-istio-token\" (UniqueName: \"kubernetes.io/projected/259da31c-7d23-434a-866a-e94d82ee82bd-istio-token\") pod \"259da31c-7d23-434a-866a-e94d82ee82bd\" (UID: \"259da31c-7d23-434a-866a-e94d82ee82bd\") "
Nov 01 04:14:34.497138 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:34.496883    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/259da31c-7d23-434a-866a-e94d82ee82bd-workload-socket" pod "259da31c-7d23-434a-866a-e94d82ee82bd" (UID: "259da31c-7d23-434a-866a-e94d82ee82bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:14:34.497138 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:34.496920    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/259da31c-7d23-434a-866a-e94d82ee82bd-credential-socket" pod "259da31c-7d23-434a-866a-e94d82ee82bd" (UID: "259da31c-7d23-434a-866a-e94d82ee82bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:14:34.497138 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:34.496955    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/259da31c-7d23-434a-866a-e94d82ee82bd-workload-certs" pod "259da31c-7d23-434a-866a-e94d82ee82bd" (UID: "259da31c-7d23-434a-866a-e94d82ee82bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:14:34.497310 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:34.497177    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/259da31c-7d23-434a-866a-e94d82ee82bd-istiod-ca-cert" pod "259da31c-7d23-434a-866a-e94d82ee82bd" (UID: "259da31c-7d23-434a-866a-e94d82ee82bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:14:34.497374 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:34.497363    4930 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/259da31c-7d23-434a-866a-e94d82ee82bd-istio-data\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:14:34.497441 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:34.497380    4930 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/259da31c-7d23-434a-866a-e94d82ee82bd-workload-certs\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:14:34.497441 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:34.497391    4930 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/259da31c-7d23-434a-866a-e94d82ee82bd-workload-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:14:34.497441 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:34.497400    4930 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/259da31c-7d23-434a-866a-e94d82ee82bd-credential-socket\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:14:34.497441 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:34.497409    4930 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/259da31c-7d23-434a-866a-e94d82ee82bd-istiod-ca-cert\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:14:34.498794 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:34.498766    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/259da31c-7d23-434a-866a-e94d82ee82bd-istio-podinfo" pod "259da31c-7d23-434a-866a-e94d82ee82bd" (UID: "259da31c-7d23-434a-866a-e94d82ee82bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 04:14:34.498920 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:34.498813    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/259da31c-7d23-434a-866a-e94d82ee82bd-istio-token" pod "259da31c-7d23-434a-866a-e94d82ee82bd" (UID: "259da31c-7d23-434a-866a-e94d82ee82bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:14:34.499031 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:34.499011    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/259da31c-7d23-434a-866a-e94d82ee82bd-istio-envoy" pod "259da31c-7d23-434a-866a-e94d82ee82bd" (UID: "259da31c-7d23-434a-866a-e94d82ee82bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 04:14:34.499080 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:34.499060    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/259da31c-7d23-434a-866a-e94d82ee82bd-kube-api-access-89mnc" pod "259da31c-7d23-434a-866a-e94d82ee82bd" (UID: "259da31c-7d23-434a-866a-e94d82ee82bd"). InnerVolumeSpecName "kube-api-access-89mnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:14:34.563673 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:34.563646    4930 generic.go:356] "Generic (PLEG): container finished" podID="259da31c-7d23-434a-866a-e94d82ee82bd" containerID="de8665eea0b92f61b5644a4580651a9f78630449efe61a17584e7ff1067a720d" exitCode=0
Nov 01 04:14:34.563827 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:34.563761    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684" event={"ID":"259da31c-7d23-434a-866a-e94d82ee82bd","Type":"ContainerDied","Data":"de8665eea0b92f61b5644a4580651a9f78630449efe61a17584e7ff1067a720d"}
Nov 01 04:14:34.563827 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:34.563814    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684" event={"ID":"259da31c-7d23-434a-866a-e94d82ee82bd","Type":"ContainerDied","Data":"bc35cc87b759f93dc5d2d36b257cc7a99e48a2b2281e30262aac0d5014b2da8b"}
Nov 01 04:14:34.564015 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:34.563843    4930 scope.go:122] "RemoveContainer" containerID="de8665eea0b92f61b5644a4580651a9f78630449efe61a17584e7ff1067a720d"
Nov 01 04:14:34.564077 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:34.564022    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"
Nov 01 04:14:34.576985 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:34.576966    4930 scope.go:122] "RemoveContainer" containerID="375fdeb4241dfdf2217ac650b7b4fd283bf93522bf0a6655f3f001b5401dd35a"
Nov 01 04:14:34.585498 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:34.585477    4930 scope.go:122] "RemoveContainer" containerID="de8665eea0b92f61b5644a4580651a9f78630449efe61a17584e7ff1067a720d"
Nov 01 04:14:34.585806 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:14:34.585782    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de8665eea0b92f61b5644a4580651a9f78630449efe61a17584e7ff1067a720d\": container with ID starting with de8665eea0b92f61b5644a4580651a9f78630449efe61a17584e7ff1067a720d not found: ID does not exist" containerID="de8665eea0b92f61b5644a4580651a9f78630449efe61a17584e7ff1067a720d"
Nov 01 04:14:34.585895 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:34.585813    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de8665eea0b92f61b5644a4580651a9f78630449efe61a17584e7ff1067a720d"} err="failed to get container status \"de8665eea0b92f61b5644a4580651a9f78630449efe61a17584e7ff1067a720d\": rpc error: code = NotFound desc = could not find container \"de8665eea0b92f61b5644a4580651a9f78630449efe61a17584e7ff1067a720d\": container with ID starting with de8665eea0b92f61b5644a4580651a9f78630449efe61a17584e7ff1067a720d not found: ID does not exist"
Nov 01 04:14:34.585895 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:34.585831    4930 scope.go:122] "RemoveContainer" containerID="375fdeb4241dfdf2217ac650b7b4fd283bf93522bf0a6655f3f001b5401dd35a"
Nov 01 04:14:34.586038 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:14:34.586021    4930 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"375fdeb4241dfdf2217ac650b7b4fd283bf93522bf0a6655f3f001b5401dd35a\": container with ID starting with 375fdeb4241dfdf2217ac650b7b4fd283bf93522bf0a6655f3f001b5401dd35a not found: ID does not exist" containerID="375fdeb4241dfdf2217ac650b7b4fd283bf93522bf0a6655f3f001b5401dd35a"
Nov 01 04:14:34.586105 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:34.586041    4930 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"375fdeb4241dfdf2217ac650b7b4fd283bf93522bf0a6655f3f001b5401dd35a"} err="failed to get container status \"375fdeb4241dfdf2217ac650b7b4fd283bf93522bf0a6655f3f001b5401dd35a\": rpc error: code = NotFound desc = could not find container \"375fdeb4241dfdf2217ac650b7b4fd283bf93522bf0a6655f3f001b5401dd35a\": container with ID starting with 375fdeb4241dfdf2217ac650b7b4fd283bf93522bf0a6655f3f001b5401dd35a not found: ID does not exist"
Nov 01 04:14:34.598249 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:34.598212    4930 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/259da31c-7d23-434a-866a-e94d82ee82bd-istio-token\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:14:34.598249 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:34.598240    4930 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/259da31c-7d23-434a-866a-e94d82ee82bd-istio-envoy\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:14:34.598419 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:34.598253    4930 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/259da31c-7d23-434a-866a-e94d82ee82bd-istio-podinfo\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:14:34.598419 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:34.598263    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89mnc\" (UniqueName: \"kubernetes.io/projected/259da31c-7d23-434a-866a-e94d82ee82bd-kube-api-access-89mnc\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:14:34.728651 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:34.728627    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"]
Nov 01 04:14:34.733615 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:34.733592    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-84f12f-openshift-default-695575dc49-sp684"]
Nov 01 04:14:35.619451 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:35.619419    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="259da31c-7d23-434a-866a-e94d82ee82bd" path="/var/lib/kubelet/pods/259da31c-7d23-434a-866a-e94d82ee82bd/volumes"
Nov 01 04:14:41.352498 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:41.352462    4930 scope.go:122] "RemoveContainer" containerID="1e61ea38119102bd1b07eb971cc7319560708727ac1c205e8b6e1a2613e5c9ff"
Nov 01 04:14:49.134678 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:14:49.134572    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 04:14:50.187723 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:50.187687    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:14:56.845114 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:56.845080    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m7xld"]
Nov 01 04:14:56.845519 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:56.845462    4930 cpu_manager.go:414] "containerMap: removing container" podUID="259da31c-7d23-434a-866a-e94d82ee82bd" containerName="istio-proxy"
Nov 01 04:14:56.845519 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:56.845473    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="259da31c-7d23-434a-866a-e94d82ee82bd" containerName="istio-proxy"
Nov 01 04:14:56.845519 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:56.845486    4930 cpu_manager.go:414] "containerMap: removing container" podUID="7e521d66-d7d5-497c-9c34-4c2ce9e821c7" containerName="payload-pre-processing"
Nov 01 04:14:56.845519 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:56.845491    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e521d66-d7d5-497c-9c34-4c2ce9e821c7" containerName="payload-pre-processing"
Nov 01 04:14:56.845519 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:56.845502    4930 cpu_manager.go:414] "containerMap: removing container" podUID="4b56249c-3ed6-4016-9daf-7f6942f39355" containerName="revoke-keys"
Nov 01 04:14:56.845519 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:56.845507    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b56249c-3ed6-4016-9daf-7f6942f39355" containerName="revoke-keys"
Nov 01 04:14:56.845519 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:56.845525    4930 cpu_manager.go:414] "containerMap: removing container" podUID="259da31c-7d23-434a-866a-e94d82ee82bd" containerName="istio-proxy"
Nov 01 04:14:56.845817 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:56.845529    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="259da31c-7d23-434a-866a-e94d82ee82bd" containerName="istio-proxy"
Nov 01 04:14:56.845817 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:56.845536    4930 cpu_manager.go:414] "containerMap: removing container" podUID="6712d7f0-b182-4d1f-9d0e-e92e55282f66" containerName="payload-processing"
Nov 01 04:14:56.845817 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:56.845540    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6712d7f0-b182-4d1f-9d0e-e92e55282f66" containerName="payload-processing"
Nov 01 04:14:56.845817 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:56.845641    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="259da31c-7d23-434a-866a-e94d82ee82bd" containerName="istio-proxy"
Nov 01 04:14:56.845817 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:56.845650    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="6712d7f0-b182-4d1f-9d0e-e92e55282f66" containerName="payload-processing"
Nov 01 04:14:56.845817 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:56.845662    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e521d66-d7d5-497c-9c34-4c2ce9e821c7" containerName="payload-pre-processing"
Nov 01 04:14:56.845817 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:56.845669    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b56249c-3ed6-4016-9daf-7f6942f39355" containerName="revoke-keys"
Nov 01 04:14:56.848954 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:56.848936    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m7xld"
Nov 01 04:14:56.893052 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:56.893026    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgnzh\" (UniqueName: \"kubernetes.io/projected/1135cbb4-8c8e-4f6c-b081-4bdbe2617263-kube-api-access-sgnzh\") pod \"maas-api-revoke-sub-f03629db-m7xld\" (UID: \"1135cbb4-8c8e-4f6c-b081-4bdbe2617263\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m7xld"
Nov 01 04:14:56.893169 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:56.893074    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1135cbb4-8c8e-4f6c-b081-4bdbe2617263-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-m7xld\" (UID: \"1135cbb4-8c8e-4f6c-b081-4bdbe2617263\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m7xld"
Nov 01 04:14:56.893169 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:56.893152    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1135cbb4-8c8e-4f6c-b081-4bdbe2617263-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-m7xld\" (UID: \"1135cbb4-8c8e-4f6c-b081-4bdbe2617263\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m7xld"
Nov 01 04:14:56.901160 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:56.901137    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 04:14:56.950690 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:56.950657    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:14:56.950832 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:56.950696    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 04:14:56.950832 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:56.950710    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7br8\""
Nov 01 04:14:56.953261 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:56.953241    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m7xld"]
Nov 01 04:14:56.994218 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:56.994191    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1135cbb4-8c8e-4f6c-b081-4bdbe2617263-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-m7xld\" (UID: \"1135cbb4-8c8e-4f6c-b081-4bdbe2617263\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m7xld"
Nov 01 04:14:56.994343 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:56.994230    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1135cbb4-8c8e-4f6c-b081-4bdbe2617263-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-m7xld\" (UID: \"1135cbb4-8c8e-4f6c-b081-4bdbe2617263\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m7xld"
Nov 01 04:14:56.994343 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:56.994308    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgnzh\" (UniqueName: \"kubernetes.io/projected/1135cbb4-8c8e-4f6c-b081-4bdbe2617263-kube-api-access-sgnzh\") pod \"maas-api-revoke-sub-f03629db-m7xld\" (UID: \"1135cbb4-8c8e-4f6c-b081-4bdbe2617263\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m7xld"
Nov 01 04:14:56.994858 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:56.994841    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1135cbb4-8c8e-4f6c-b081-4bdbe2617263-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-m7xld\" (UID: \"1135cbb4-8c8e-4f6c-b081-4bdbe2617263\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m7xld"
Nov 01 04:14:56.996698 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:56.996675    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1135cbb4-8c8e-4f6c-b081-4bdbe2617263-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-m7xld\" (UID: \"1135cbb4-8c8e-4f6c-b081-4bdbe2617263\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m7xld"
Nov 01 04:14:57.055669 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:57.055642    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgnzh\" (UniqueName: \"kubernetes.io/projected/1135cbb4-8c8e-4f6c-b081-4bdbe2617263-kube-api-access-sgnzh\") pod \"maas-api-revoke-sub-f03629db-m7xld\" (UID: \"1135cbb4-8c8e-4f6c-b081-4bdbe2617263\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m7xld"
Nov 01 04:14:57.162175 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:57.162105    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m7xld"
Nov 01 04:14:57.533176 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:57.533151    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m7xld"]
Nov 01 04:14:57.535080 ip-10-0-141-122 kubenswrapper[4930]: W1101 04:14:57.535052    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1135cbb4_8c8e_4f6c_b081_4bdbe2617263.slice/crio-8375cb676f092d1d96285c7e2f8bab098c70b78a11eafe87c36e8ba9d4e38dc4 WatchSource:0}: Error finding container 8375cb676f092d1d96285c7e2f8bab098c70b78a11eafe87c36e8ba9d4e38dc4: Status 404 returned error can't find the container with id 8375cb676f092d1d96285c7e2f8bab098c70b78a11eafe87c36e8ba9d4e38dc4
Nov 01 04:14:57.779483 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:57.779457    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m7xld" event={"ID":"1135cbb4-8c8e-4f6c-b081-4bdbe2617263","Type":"ContainerStarted","Data":"8375cb676f092d1d96285c7e2f8bab098c70b78a11eafe87c36e8ba9d4e38dc4"}
Nov 01 04:14:58.787081 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:58.787005    4930 generic.go:356] "Generic (PLEG): container finished" podID="1135cbb4-8c8e-4f6c-b081-4bdbe2617263" containerID="a212d5b46bce1fa11cef89466fc29cae6599bcc913a49c0f7cccf62b2e6e349b" exitCode=0
Nov 01 04:14:58.787425 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:58.787093    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m7xld" event={"ID":"1135cbb4-8c8e-4f6c-b081-4bdbe2617263","Type":"ContainerDied","Data":"a212d5b46bce1fa11cef89466fc29cae6599bcc913a49c0f7cccf62b2e6e349b"}
Nov 01 04:14:59.847058 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:59.846932    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:14:59.928613 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:14:59.928540    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m7xld"
Nov 01 04:15:00.024388 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:00.024361    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1135cbb4-8c8e-4f6c-b081-4bdbe2617263-kube-api-access-sgnzh\" (UniqueName: \"kubernetes.io/projected/1135cbb4-8c8e-4f6c-b081-4bdbe2617263-kube-api-access-sgnzh\") pod \"1135cbb4-8c8e-4f6c-b081-4bdbe2617263\" (UID: \"1135cbb4-8c8e-4f6c-b081-4bdbe2617263\") "
Nov 01 04:15:00.024533 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:00.024401    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1135cbb4-8c8e-4f6c-b081-4bdbe2617263-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1135cbb4-8c8e-4f6c-b081-4bdbe2617263-maas-api-openshift-ca\") pod \"1135cbb4-8c8e-4f6c-b081-4bdbe2617263\" (UID: \"1135cbb4-8c8e-4f6c-b081-4bdbe2617263\") "
Nov 01 04:15:00.024533 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:00.024449    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1135cbb4-8c8e-4f6c-b081-4bdbe2617263-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1135cbb4-8c8e-4f6c-b081-4bdbe2617263-maas-api-xks-ca\") pod \"1135cbb4-8c8e-4f6c-b081-4bdbe2617263\" (UID: \"1135cbb4-8c8e-4f6c-b081-4bdbe2617263\") "
Nov 01 04:15:00.024782 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:00.024761    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1135cbb4-8c8e-4f6c-b081-4bdbe2617263-maas-api-openshift-ca" pod "1135cbb4-8c8e-4f6c-b081-4bdbe2617263" (UID: "1135cbb4-8c8e-4f6c-b081-4bdbe2617263"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:15:00.026481 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:00.026454    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1135cbb4-8c8e-4f6c-b081-4bdbe2617263-kube-api-access-sgnzh" pod "1135cbb4-8c8e-4f6c-b081-4bdbe2617263" (UID: "1135cbb4-8c8e-4f6c-b081-4bdbe2617263"). InnerVolumeSpecName "kube-api-access-sgnzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:15:00.026605 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:00.026548    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1135cbb4-8c8e-4f6c-b081-4bdbe2617263-maas-api-xks-ca" pod "1135cbb4-8c8e-4f6c-b081-4bdbe2617263" (UID: "1135cbb4-8c8e-4f6c-b081-4bdbe2617263"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:15:00.125068 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:00.125039    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgnzh\" (UniqueName: \"kubernetes.io/projected/1135cbb4-8c8e-4f6c-b081-4bdbe2617263-kube-api-access-sgnzh\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:15:00.125068 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:00.125065    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1135cbb4-8c8e-4f6c-b081-4bdbe2617263-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:15:00.125237 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:00.125076    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1135cbb4-8c8e-4f6c-b081-4bdbe2617263-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:15:00.166731 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:00.166702    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891775-hwpxh"]
Nov 01 04:15:00.167149 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:00.167135    4930 cpu_manager.go:414] "containerMap: removing container" podUID="1135cbb4-8c8e-4f6c-b081-4bdbe2617263" containerName="revoke-keys"
Nov 01 04:15:00.167205 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:00.167151    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1135cbb4-8c8e-4f6c-b081-4bdbe2617263" containerName="revoke-keys"
Nov 01 04:15:00.167248 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:00.167238    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="259da31c-7d23-434a-866a-e94d82ee82bd" containerName="istio-proxy"
Nov 01 04:15:00.167283 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:00.167248    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="1135cbb4-8c8e-4f6c-b081-4bdbe2617263" containerName="revoke-keys"
Nov 01 04:15:00.170523 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:00.170505    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891775-hwpxh"
Nov 01 04:15:00.225961 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:00.225933    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eee1b089-0f8d-4d02-a454-2d0bda24f02b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891775-hwpxh\" (UID: \"eee1b089-0f8d-4d02-a454-2d0bda24f02b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891775-hwpxh"
Nov 01 04:15:00.226158 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:00.225977    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjcks\" (UniqueName: \"kubernetes.io/projected/eee1b089-0f8d-4d02-a454-2d0bda24f02b-kube-api-access-bjcks\") pod \"maas-api-key-cleanup-29891775-hwpxh\" (UID: \"eee1b089-0f8d-4d02-a454-2d0bda24f02b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891775-hwpxh"
Nov 01 04:15:00.229399 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:00.229362    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891775-hwpxh"]
Nov 01 04:15:00.327247 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:00.327221    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eee1b089-0f8d-4d02-a454-2d0bda24f02b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891775-hwpxh\" (UID: \"eee1b089-0f8d-4d02-a454-2d0bda24f02b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891775-hwpxh"
Nov 01 04:15:00.327399 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:00.327255    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjcks\" (UniqueName: \"kubernetes.io/projected/eee1b089-0f8d-4d02-a454-2d0bda24f02b-kube-api-access-bjcks\") pod \"maas-api-key-cleanup-29891775-hwpxh\" (UID: \"eee1b089-0f8d-4d02-a454-2d0bda24f02b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891775-hwpxh"
Nov 01 04:15:00.327906 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:00.327875    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eee1b089-0f8d-4d02-a454-2d0bda24f02b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891775-hwpxh\" (UID: \"eee1b089-0f8d-4d02-a454-2d0bda24f02b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891775-hwpxh"
Nov 01 04:15:00.388461 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:00.388391    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjcks\" (UniqueName: \"kubernetes.io/projected/eee1b089-0f8d-4d02-a454-2d0bda24f02b-kube-api-access-bjcks\") pod \"maas-api-key-cleanup-29891775-hwpxh\" (UID: \"eee1b089-0f8d-4d02-a454-2d0bda24f02b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891775-hwpxh"
Nov 01 04:15:00.483114 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:00.483086    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891775-hwpxh"
Nov 01 04:15:00.799480 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:00.799409    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m7xld"
Nov 01 04:15:00.799480 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:00.799429    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-m7xld" event={"ID":"1135cbb4-8c8e-4f6c-b081-4bdbe2617263","Type":"ContainerDied","Data":"8375cb676f092d1d96285c7e2f8bab098c70b78a11eafe87c36e8ba9d4e38dc4"}
Nov 01 04:15:00.799480 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:00.799477    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8375cb676f092d1d96285c7e2f8bab098c70b78a11eafe87c36e8ba9d4e38dc4"
Nov 01 04:15:00.855514 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:00.855492    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891775-hwpxh"]
Nov 01 04:15:00.857342 ip-10-0-141-122 kubenswrapper[4930]: W1101 04:15:00.857307    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeee1b089_0f8d_4d02_a454_2d0bda24f02b.slice/crio-bbb5f380cff7c4aa583aeb4aa8f1533a07cf08191eaf6c17f03d0db6dacfcde9 WatchSource:0}: Error finding container bbb5f380cff7c4aa583aeb4aa8f1533a07cf08191eaf6c17f03d0db6dacfcde9: Status 404 returned error can't find the container with id bbb5f380cff7c4aa583aeb4aa8f1533a07cf08191eaf6c17f03d0db6dacfcde9
Nov 01 04:15:01.806052 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:01.806018    4930 generic.go:356] "Generic (PLEG): container finished" podID="eee1b089-0f8d-4d02-a454-2d0bda24f02b" containerID="3f398b15640689dc7b32de402ef3582a8a65501d2704d6153f55b91edd3a7d85" exitCode=60
Nov 01 04:15:01.806248 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:01.806105    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891775-hwpxh" event={"ID":"eee1b089-0f8d-4d02-a454-2d0bda24f02b","Type":"ContainerDied","Data":"3f398b15640689dc7b32de402ef3582a8a65501d2704d6153f55b91edd3a7d85"}
Nov 01 04:15:01.806248 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:01.806166    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891775-hwpxh" event={"ID":"eee1b089-0f8d-4d02-a454-2d0bda24f02b","Type":"ContainerStarted","Data":"bbb5f380cff7c4aa583aeb4aa8f1533a07cf08191eaf6c17f03d0db6dacfcde9"}
Nov 01 04:15:01.806346 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:01.806281    4930 scope.go:122] "RemoveContainer" containerID="3f398b15640689dc7b32de402ef3582a8a65501d2704d6153f55b91edd3a7d85"
Nov 01 04:15:02.813478 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:02.813442    4930 generic.go:356] "Generic (PLEG): container finished" podID="eee1b089-0f8d-4d02-a454-2d0bda24f02b" containerID="964963de5f140af682a1cb1cdd35214e67f2f3e34c3b1c6e62cac07838d07845" exitCode=60
Nov 01 04:15:02.813961 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:02.813519    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891775-hwpxh" event={"ID":"eee1b089-0f8d-4d02-a454-2d0bda24f02b","Type":"ContainerDied","Data":"964963de5f140af682a1cb1cdd35214e67f2f3e34c3b1c6e62cac07838d07845"}
Nov 01 04:15:02.813961 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:02.813587    4930 scope.go:122] "RemoveContainer" containerID="3f398b15640689dc7b32de402ef3582a8a65501d2704d6153f55b91edd3a7d85"
Nov 01 04:15:02.813961 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:02.813787    4930 scope.go:122] "RemoveContainer" containerID="964963de5f140af682a1cb1cdd35214e67f2f3e34c3b1c6e62cac07838d07845"
Nov 01 04:15:02.814126 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:15:02.814048    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891775-hwpxh_odh-ai-gateway-infra(eee1b089-0f8d-4d02-a454-2d0bda24f02b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891775-hwpxh" podUID="eee1b089-0f8d-4d02-a454-2d0bda24f02b"
Nov 01 04:15:03.827713 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:03.827684    4930 scope.go:122] "RemoveContainer" containerID="964963de5f140af682a1cb1cdd35214e67f2f3e34c3b1c6e62cac07838d07845"
Nov 01 04:15:03.828204 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:15:03.827962    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891775-hwpxh_odh-ai-gateway-infra(eee1b089-0f8d-4d02-a454-2d0bda24f02b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891775-hwpxh" podUID="eee1b089-0f8d-4d02-a454-2d0bda24f02b"
Nov 01 04:15:05.818480 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:05.818445    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:15:14.613427 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:14.613391    4930 scope.go:122] "RemoveContainer" containerID="964963de5f140af682a1cb1cdd35214e67f2f3e34c3b1c6e62cac07838d07845"
Nov 01 04:15:14.890035 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:14.890009    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891775-hwpxh" event={"ID":"eee1b089-0f8d-4d02-a454-2d0bda24f02b","Type":"ContainerStarted","Data":"cc5180cdea8ce8fe2bfcab1975f29e6336bcfc89a139032165235b47b41d1bde"}
Nov 01 04:15:15.006165 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:15.006105    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891775-hwpxh" podStartSLOduration=15.006088146 podStartE2EDuration="15.006088146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:15:15.004872383 +0000 UTC m=+4119.527878971" watchObservedRunningTime="2026-11-01 04:15:15.006088146 +0000 UTC m=+4119.529094781"
Nov 01 04:15:15.902216 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:15.902178    4930 generic.go:356] "Generic (PLEG): container finished" podID="eee1b089-0f8d-4d02-a454-2d0bda24f02b" containerID="cc5180cdea8ce8fe2bfcab1975f29e6336bcfc89a139032165235b47b41d1bde" exitCode=60
Nov 01 04:15:15.902862 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:15.902237    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891775-hwpxh" event={"ID":"eee1b089-0f8d-4d02-a454-2d0bda24f02b","Type":"ContainerDied","Data":"cc5180cdea8ce8fe2bfcab1975f29e6336bcfc89a139032165235b47b41d1bde"}
Nov 01 04:15:15.903014 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:15.902904    4930 scope.go:122] "RemoveContainer" containerID="964963de5f140af682a1cb1cdd35214e67f2f3e34c3b1c6e62cac07838d07845"
Nov 01 04:15:15.903293 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:15.903271    4930 scope.go:122] "RemoveContainer" containerID="cc5180cdea8ce8fe2bfcab1975f29e6336bcfc89a139032165235b47b41d1bde"
Nov 01 04:15:15.903604 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:15:15.903576    4930 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891775-hwpxh_odh-ai-gateway-infra(eee1b089-0f8d-4d02-a454-2d0bda24f02b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891775-hwpxh" podUID="eee1b089-0f8d-4d02-a454-2d0bda24f02b"
Nov 01 04:15:16.030623 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:16.029166    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891775-hwpxh"]
Nov 01 04:15:17.402068 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:17.379434    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-cf66s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 04:15:17.402068 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:17.379511    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-cf66s" podUID="07489556-954b-45d9-9ebc-53b2920d4004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:15:17.410591 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:17.406676    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 04:15:17.410591 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:17.406750    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-wmcn6" podUID="1dd1479c-0836-4b90-a7df-6f5d1a4683c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:15:17.456757 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:17.456602    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.3:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 04:15:17.456757 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:17.456688    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-xtgmg" podUID="12bbd677-485b-4653-8261-8a3399da12d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.3:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:15:17.503699 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:17.503668    4930 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7c5b899d5f-d89gf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 04:15:17.507675 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:17.507624    4930 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7c5b899d5f-d89gf" podUID="d4fa40a8-e13b-4c67-954e-d283853b8fe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 04:15:17.795757 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:17.795727    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891775-hwpxh"
Nov 01 04:15:17.843476 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:17.843446    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-n8jrg"]
Nov 01 04:15:17.844327 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:17.844307    4930 cpu_manager.go:414] "containerMap: removing container" podUID="eee1b089-0f8d-4d02-a454-2d0bda24f02b" containerName="cleanup"
Nov 01 04:15:17.844463 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:17.844436    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eee1b089-0f8d-4d02-a454-2d0bda24f02b" containerName="cleanup"
Nov 01 04:15:17.844572 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:17.844548    4930 cpu_manager.go:414] "containerMap: removing container" podUID="eee1b089-0f8d-4d02-a454-2d0bda24f02b" containerName="cleanup"
Nov 01 04:15:17.844661 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:17.844650    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eee1b089-0f8d-4d02-a454-2d0bda24f02b" containerName="cleanup"
Nov 01 04:15:17.844891 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:17.844863    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="eee1b089-0f8d-4d02-a454-2d0bda24f02b" containerName="cleanup"
Nov 01 04:15:17.844996 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:17.844985    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="eee1b089-0f8d-4d02-a454-2d0bda24f02b" containerName="cleanup"
Nov 01 04:15:17.845182 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:17.845164    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="eee1b089-0f8d-4d02-a454-2d0bda24f02b" containerName="cleanup"
Nov 01 04:15:17.848517 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:17.848492    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-n8jrg"
Nov 01 04:15:17.909758 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:17.909732    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:15:17.910704 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:17.910679    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-n8jrg"]
Nov 01 04:15:17.942776 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:17.940970    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eee1b089-0f8d-4d02-a454-2d0bda24f02b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eee1b089-0f8d-4d02-a454-2d0bda24f02b-maas-api-service-ca\") pod \"eee1b089-0f8d-4d02-a454-2d0bda24f02b\" (UID: \"eee1b089-0f8d-4d02-a454-2d0bda24f02b\") "
Nov 01 04:15:17.942776 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:17.941033    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eee1b089-0f8d-4d02-a454-2d0bda24f02b-kube-api-access-bjcks\" (UniqueName: \"kubernetes.io/projected/eee1b089-0f8d-4d02-a454-2d0bda24f02b-kube-api-access-bjcks\") pod \"eee1b089-0f8d-4d02-a454-2d0bda24f02b\" (UID: \"eee1b089-0f8d-4d02-a454-2d0bda24f02b\") "
Nov 01 04:15:17.942776 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:17.941854    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cdj2\" (UniqueName: \"kubernetes.io/projected/a4e9e2a3-c46d-492e-a157-2c7097aa1486-kube-api-access-6cdj2\") pod \"maas-api-revoke-sub-0165d7d2-n8jrg\" (UID: \"a4e9e2a3-c46d-492e-a157-2c7097aa1486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-n8jrg"
Nov 01 04:15:17.942776 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:17.941925    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4e9e2a3-c46d-492e-a157-2c7097aa1486-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-n8jrg\" (UID: \"a4e9e2a3-c46d-492e-a157-2c7097aa1486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-n8jrg"
Nov 01 04:15:17.942776 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:17.941998    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4e9e2a3-c46d-492e-a157-2c7097aa1486-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-n8jrg\" (UID: \"a4e9e2a3-c46d-492e-a157-2c7097aa1486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-n8jrg"
Nov 01 04:15:17.942776 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:17.942507    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eee1b089-0f8d-4d02-a454-2d0bda24f02b-maas-api-service-ca" pod "eee1b089-0f8d-4d02-a454-2d0bda24f02b" (UID: "eee1b089-0f8d-4d02-a454-2d0bda24f02b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:15:17.943936 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:17.943912    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eee1b089-0f8d-4d02-a454-2d0bda24f02b-kube-api-access-bjcks" pod "eee1b089-0f8d-4d02-a454-2d0bda24f02b" (UID: "eee1b089-0f8d-4d02-a454-2d0bda24f02b"). InnerVolumeSpecName "kube-api-access-bjcks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:15:17.950653 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:17.950625    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891775-hwpxh" event={"ID":"eee1b089-0f8d-4d02-a454-2d0bda24f02b","Type":"ContainerDied","Data":"bbb5f380cff7c4aa583aeb4aa8f1533a07cf08191eaf6c17f03d0db6dacfcde9"}
Nov 01 04:15:17.950791 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:17.950678    4930 scope.go:122] "RemoveContainer" containerID="cc5180cdea8ce8fe2bfcab1975f29e6336bcfc89a139032165235b47b41d1bde"
Nov 01 04:15:17.950791 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:17.950774    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891775-hwpxh"
Nov 01 04:15:18.042615 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:18.042536    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cdj2\" (UniqueName: \"kubernetes.io/projected/a4e9e2a3-c46d-492e-a157-2c7097aa1486-kube-api-access-6cdj2\") pod \"maas-api-revoke-sub-0165d7d2-n8jrg\" (UID: \"a4e9e2a3-c46d-492e-a157-2c7097aa1486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-n8jrg"
Nov 01 04:15:18.042814 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:18.042626    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4e9e2a3-c46d-492e-a157-2c7097aa1486-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-n8jrg\" (UID: \"a4e9e2a3-c46d-492e-a157-2c7097aa1486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-n8jrg"
Nov 01 04:15:18.042814 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:18.042672    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4e9e2a3-c46d-492e-a157-2c7097aa1486-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-n8jrg\" (UID: \"a4e9e2a3-c46d-492e-a157-2c7097aa1486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-n8jrg"
Nov 01 04:15:18.042814 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:18.042745    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eee1b089-0f8d-4d02-a454-2d0bda24f02b-maas-api-service-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:15:18.042814 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:18.042754    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjcks\" (UniqueName: \"kubernetes.io/projected/eee1b089-0f8d-4d02-a454-2d0bda24f02b-kube-api-access-bjcks\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:15:18.043934 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:18.043907    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4e9e2a3-c46d-492e-a157-2c7097aa1486-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-n8jrg\" (UID: \"a4e9e2a3-c46d-492e-a157-2c7097aa1486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-n8jrg"
Nov 01 04:15:18.045497 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:18.045470    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4e9e2a3-c46d-492e-a157-2c7097aa1486-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-n8jrg\" (UID: \"a4e9e2a3-c46d-492e-a157-2c7097aa1486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-n8jrg"
Nov 01 04:15:18.108035 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:18.108013    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cdj2\" (UniqueName: \"kubernetes.io/projected/a4e9e2a3-c46d-492e-a157-2c7097aa1486-kube-api-access-6cdj2\") pod \"maas-api-revoke-sub-0165d7d2-n8jrg\" (UID: \"a4e9e2a3-c46d-492e-a157-2c7097aa1486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-n8jrg"
Nov 01 04:15:18.176199 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:18.176136    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891775-hwpxh"]
Nov 01 04:15:18.180269 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:18.180247    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891775-hwpxh"]
Nov 01 04:15:18.184851 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:18.184832    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-n8jrg"
Nov 01 04:15:18.558974 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:18.558947    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-n8jrg"]
Nov 01 04:15:18.560593 ip-10-0-141-122 kubenswrapper[4930]: W1101 04:15:18.560538    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4e9e2a3_c46d_492e_a157_2c7097aa1486.slice/crio-ca0a2f83da5f40c9541aa4f0ce054f5df904f883f9c016d23b1d0c418d36e2ba WatchSource:0}: Error finding container ca0a2f83da5f40c9541aa4f0ce054f5df904f883f9c016d23b1d0c418d36e2ba: Status 404 returned error can't find the container with id ca0a2f83da5f40c9541aa4f0ce054f5df904f883f9c016d23b1d0c418d36e2ba
Nov 01 04:15:18.959128 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:18.959098    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-n8jrg" event={"ID":"a4e9e2a3-c46d-492e-a157-2c7097aa1486","Type":"ContainerStarted","Data":"7ccea5fda0c11c97de46fb9c718b93fd32e9df75ba499e7882fd422677f0e771"}
Nov 01 04:15:18.959264 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:18.959154    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-n8jrg" event={"ID":"a4e9e2a3-c46d-492e-a157-2c7097aa1486","Type":"ContainerStarted","Data":"ca0a2f83da5f40c9541aa4f0ce054f5df904f883f9c016d23b1d0c418d36e2ba"}
Nov 01 04:15:19.075004 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:19.074939    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-n8jrg" podStartSLOduration=2.07492373 podStartE2EDuration="2.07492373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:15:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:15:19.073827198 +0000 UTC m=+4123.596833788" watchObservedRunningTime="2026-11-01 04:15:19.07492373 +0000 UTC m=+4123.597930318"
Nov 01 04:15:19.121321 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:15:19.121244    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 04:15:19.619081 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:19.619045    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eee1b089-0f8d-4d02-a454-2d0bda24f02b" path="/var/lib/kubelet/pods/eee1b089-0f8d-4d02-a454-2d0bda24f02b/volumes"
Nov 01 04:15:19.965778 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:19.965741    4930 generic.go:356] "Generic (PLEG): container finished" podID="a4e9e2a3-c46d-492e-a157-2c7097aa1486" containerID="7ccea5fda0c11c97de46fb9c718b93fd32e9df75ba499e7882fd422677f0e771" exitCode=0
Nov 01 04:15:19.965951 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:19.965824    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-n8jrg" event={"ID":"a4e9e2a3-c46d-492e-a157-2c7097aa1486","Type":"ContainerDied","Data":"7ccea5fda0c11c97de46fb9c718b93fd32e9df75ba499e7882fd422677f0e771"}
Nov 01 04:15:21.115478 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:21.115452    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-n8jrg"
Nov 01 04:15:21.172993 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:21.172966    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4e9e2a3-c46d-492e-a157-2c7097aa1486-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4e9e2a3-c46d-492e-a157-2c7097aa1486-maas-api-xks-ca\") pod \"a4e9e2a3-c46d-492e-a157-2c7097aa1486\" (UID: \"a4e9e2a3-c46d-492e-a157-2c7097aa1486\") "
Nov 01 04:15:21.173141 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:21.173016    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4e9e2a3-c46d-492e-a157-2c7097aa1486-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4e9e2a3-c46d-492e-a157-2c7097aa1486-maas-api-openshift-ca\") pod \"a4e9e2a3-c46d-492e-a157-2c7097aa1486\" (UID: \"a4e9e2a3-c46d-492e-a157-2c7097aa1486\") "
Nov 01 04:15:21.173141 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:21.173066    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4e9e2a3-c46d-492e-a157-2c7097aa1486-kube-api-access-6cdj2\" (UniqueName: \"kubernetes.io/projected/a4e9e2a3-c46d-492e-a157-2c7097aa1486-kube-api-access-6cdj2\") pod \"a4e9e2a3-c46d-492e-a157-2c7097aa1486\" (UID: \"a4e9e2a3-c46d-492e-a157-2c7097aa1486\") "
Nov 01 04:15:21.173383 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:21.173357    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4e9e2a3-c46d-492e-a157-2c7097aa1486-maas-api-openshift-ca" pod "a4e9e2a3-c46d-492e-a157-2c7097aa1486" (UID: "a4e9e2a3-c46d-492e-a157-2c7097aa1486"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:15:21.175226 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:21.175202    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4e9e2a3-c46d-492e-a157-2c7097aa1486-maas-api-xks-ca" pod "a4e9e2a3-c46d-492e-a157-2c7097aa1486" (UID: "a4e9e2a3-c46d-492e-a157-2c7097aa1486"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:15:21.175392 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:21.175367    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4e9e2a3-c46d-492e-a157-2c7097aa1486-kube-api-access-6cdj2" pod "a4e9e2a3-c46d-492e-a157-2c7097aa1486" (UID: "a4e9e2a3-c46d-492e-a157-2c7097aa1486"). InnerVolumeSpecName "kube-api-access-6cdj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:15:21.274463 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:21.274383    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4e9e2a3-c46d-492e-a157-2c7097aa1486-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:15:21.274463 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:21.274426    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cdj2\" (UniqueName: \"kubernetes.io/projected/a4e9e2a3-c46d-492e-a157-2c7097aa1486-kube-api-access-6cdj2\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:15:21.274463 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:21.274444    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4e9e2a3-c46d-492e-a157-2c7097aa1486-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:15:21.978083 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:21.977998    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-n8jrg"
Nov 01 04:15:21.978083 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:21.978029    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-n8jrg" event={"ID":"a4e9e2a3-c46d-492e-a157-2c7097aa1486","Type":"ContainerDied","Data":"ca0a2f83da5f40c9541aa4f0ce054f5df904f883f9c016d23b1d0c418d36e2ba"}
Nov 01 04:15:21.978277 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:21.978085    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca0a2f83da5f40c9541aa4f0ce054f5df904f883f9c016d23b1d0c418d36e2ba"
Nov 01 04:15:22.072980 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:22.072945    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:15:22.274639 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:22.274515    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:15:29.648473 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:29.648426    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xdpvq"]
Nov 01 04:15:29.649098 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:29.649075    4930 cpu_manager.go:414] "containerMap: removing container" podUID="eee1b089-0f8d-4d02-a454-2d0bda24f02b" containerName="cleanup"
Nov 01 04:15:29.649206 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:29.649099    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eee1b089-0f8d-4d02-a454-2d0bda24f02b" containerName="cleanup"
Nov 01 04:15:29.649206 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:29.649141    4930 cpu_manager.go:414] "containerMap: removing container" podUID="a4e9e2a3-c46d-492e-a157-2c7097aa1486" containerName="revoke-keys"
Nov 01 04:15:29.649206 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:29.649149    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4e9e2a3-c46d-492e-a157-2c7097aa1486" containerName="revoke-keys"
Nov 01 04:15:29.649381 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:29.649255    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4e9e2a3-c46d-492e-a157-2c7097aa1486" containerName="revoke-keys"
Nov 01 04:15:29.652640 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:29.652620    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xdpvq"
Nov 01 04:15:29.710315 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:29.710278    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xdpvq"]
Nov 01 04:15:29.752158 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:29.752130    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cb0c346-db7c-4b47-9866-a55f7b77c2a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-xdpvq\" (UID: \"7cb0c346-db7c-4b47-9866-a55f7b77c2a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xdpvq"
Nov 01 04:15:29.752290 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:29.752169    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cb0c346-db7c-4b47-9866-a55f7b77c2a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-xdpvq\" (UID: \"7cb0c346-db7c-4b47-9866-a55f7b77c2a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xdpvq"
Nov 01 04:15:29.752368 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:29.752345    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msjr7\" (UniqueName: \"kubernetes.io/projected/7cb0c346-db7c-4b47-9866-a55f7b77c2a7-kube-api-access-msjr7\") pod \"maas-api-revoke-sub-6b19338f-xdpvq\" (UID: \"7cb0c346-db7c-4b47-9866-a55f7b77c2a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xdpvq"
Nov 01 04:15:29.756829 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:29.756809    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 04:15:29.756946 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:29.756828    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 04:15:29.756946 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:29.756855    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 04:15:29.756946 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:29.756860    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7br8\""
Nov 01 04:15:29.853331 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:29.853306    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cb0c346-db7c-4b47-9866-a55f7b77c2a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-xdpvq\" (UID: \"7cb0c346-db7c-4b47-9866-a55f7b77c2a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xdpvq"
Nov 01 04:15:29.853483 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:29.853344    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cb0c346-db7c-4b47-9866-a55f7b77c2a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-xdpvq\" (UID: \"7cb0c346-db7c-4b47-9866-a55f7b77c2a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xdpvq"
Nov 01 04:15:29.853483 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:29.853441    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msjr7\" (UniqueName: \"kubernetes.io/projected/7cb0c346-db7c-4b47-9866-a55f7b77c2a7-kube-api-access-msjr7\") pod \"maas-api-revoke-sub-6b19338f-xdpvq\" (UID: \"7cb0c346-db7c-4b47-9866-a55f7b77c2a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xdpvq"
Nov 01 04:15:29.853939 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:29.853915    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cb0c346-db7c-4b47-9866-a55f7b77c2a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-xdpvq\" (UID: \"7cb0c346-db7c-4b47-9866-a55f7b77c2a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xdpvq"
Nov 01 04:15:29.855752 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:29.855733    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cb0c346-db7c-4b47-9866-a55f7b77c2a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-xdpvq\" (UID: \"7cb0c346-db7c-4b47-9866-a55f7b77c2a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xdpvq"
Nov 01 04:15:29.914487 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:29.914421    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msjr7\" (UniqueName: \"kubernetes.io/projected/7cb0c346-db7c-4b47-9866-a55f7b77c2a7-kube-api-access-msjr7\") pod \"maas-api-revoke-sub-6b19338f-xdpvq\" (UID: \"7cb0c346-db7c-4b47-9866-a55f7b77c2a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xdpvq"
Nov 01 04:15:29.965722 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:29.965701    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xdpvq"
Nov 01 04:15:30.339220 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:30.339191    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xdpvq"]
Nov 01 04:15:30.341202 ip-10-0-141-122 kubenswrapper[4930]: W1101 04:15:30.341173    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cb0c346_db7c_4b47_9866_a55f7b77c2a7.slice/crio-4eebb46be468e061928bf69a6167449ebbf93a46e46442c4c54def410c934dfd WatchSource:0}: Error finding container 4eebb46be468e061928bf69a6167449ebbf93a46e46442c4c54def410c934dfd: Status 404 returned error can't find the container with id 4eebb46be468e061928bf69a6167449ebbf93a46e46442c4c54def410c934dfd
Nov 01 04:15:31.029522 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:31.029437    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xdpvq" event={"ID":"7cb0c346-db7c-4b47-9866-a55f7b77c2a7","Type":"ContainerStarted","Data":"118514e9eb26f813d86db2ad181fcfeee90676a036343f9019e23d1285177547"}
Nov 01 04:15:31.029522 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:31.029504    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xdpvq" event={"ID":"7cb0c346-db7c-4b47-9866-a55f7b77c2a7","Type":"ContainerStarted","Data":"4eebb46be468e061928bf69a6167449ebbf93a46e46442c4c54def410c934dfd"}
Nov 01 04:15:31.166443 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:31.166234    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xdpvq" podStartSLOduration=2.166212865 podStartE2EDuration="2.166212865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:15:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:15:31.161823671 +0000 UTC m=+4135.684830260" watchObservedRunningTime="2026-11-01 04:15:31.166212865 +0000 UTC m=+4135.689219454"
Nov 01 04:15:32.036540 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:32.036504    4930 generic.go:356] "Generic (PLEG): container finished" podID="7cb0c346-db7c-4b47-9866-a55f7b77c2a7" containerID="118514e9eb26f813d86db2ad181fcfeee90676a036343f9019e23d1285177547" exitCode=0
Nov 01 04:15:32.036540 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:32.036541    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xdpvq" event={"ID":"7cb0c346-db7c-4b47-9866-a55f7b77c2a7","Type":"ContainerDied","Data":"118514e9eb26f813d86db2ad181fcfeee90676a036343f9019e23d1285177547"}
Nov 01 04:15:33.200805 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:33.200774    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xdpvq"
Nov 01 04:15:33.289035 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:33.289001    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cb0c346-db7c-4b47-9866-a55f7b77c2a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cb0c346-db7c-4b47-9866-a55f7b77c2a7-maas-api-openshift-ca\") pod \"7cb0c346-db7c-4b47-9866-a55f7b77c2a7\" (UID: \"7cb0c346-db7c-4b47-9866-a55f7b77c2a7\") "
Nov 01 04:15:33.289193 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:33.289096    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cb0c346-db7c-4b47-9866-a55f7b77c2a7-kube-api-access-msjr7\" (UniqueName: \"kubernetes.io/projected/7cb0c346-db7c-4b47-9866-a55f7b77c2a7-kube-api-access-msjr7\") pod \"7cb0c346-db7c-4b47-9866-a55f7b77c2a7\" (UID: \"7cb0c346-db7c-4b47-9866-a55f7b77c2a7\") "
Nov 01 04:15:33.289193 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:33.289158    4930 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cb0c346-db7c-4b47-9866-a55f7b77c2a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cb0c346-db7c-4b47-9866-a55f7b77c2a7-maas-api-xks-ca\") pod \"7cb0c346-db7c-4b47-9866-a55f7b77c2a7\" (UID: \"7cb0c346-db7c-4b47-9866-a55f7b77c2a7\") "
Nov 01 04:15:33.289398 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:33.289373    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cb0c346-db7c-4b47-9866-a55f7b77c2a7-maas-api-openshift-ca" pod "7cb0c346-db7c-4b47-9866-a55f7b77c2a7" (UID: "7cb0c346-db7c-4b47-9866-a55f7b77c2a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 04:15:33.289510 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:33.289495    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cb0c346-db7c-4b47-9866-a55f7b77c2a7-maas-api-openshift-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:15:33.291263 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:33.291237    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cb0c346-db7c-4b47-9866-a55f7b77c2a7-maas-api-xks-ca" pod "7cb0c346-db7c-4b47-9866-a55f7b77c2a7" (UID: "7cb0c346-db7c-4b47-9866-a55f7b77c2a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 04:15:33.291382 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:33.291273    4930 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cb0c346-db7c-4b47-9866-a55f7b77c2a7-kube-api-access-msjr7" pod "7cb0c346-db7c-4b47-9866-a55f7b77c2a7" (UID: "7cb0c346-db7c-4b47-9866-a55f7b77c2a7"). InnerVolumeSpecName "kube-api-access-msjr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 04:15:33.390972 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:33.390945    4930 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msjr7\" (UniqueName: \"kubernetes.io/projected/7cb0c346-db7c-4b47-9866-a55f7b77c2a7-kube-api-access-msjr7\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:15:33.390972 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:33.390971    4930 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cb0c346-db7c-4b47-9866-a55f7b77c2a7-maas-api-xks-ca\") on node \"ip-10-0-141-122.us-west-1.compute.internal\" DevicePath \"\""
Nov 01 04:15:34.048423 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:34.048338    4930 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xdpvq"
Nov 01 04:15:34.048423 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:34.048367    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xdpvq" event={"ID":"7cb0c346-db7c-4b47-9866-a55f7b77c2a7","Type":"ContainerDied","Data":"4eebb46be468e061928bf69a6167449ebbf93a46e46442c4c54def410c934dfd"}
Nov 01 04:15:34.048423 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:34.048422    4930 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4eebb46be468e061928bf69a6167449ebbf93a46e46442c4c54def410c934dfd"
Nov 01 04:15:41.733842 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:41.733757    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-8549fcc7fb-s5trb_66e26411-cab9-4f6a-8814-53165e790a45/authorino/0.log"
Nov 01 04:15:49.121739 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:15:49.121697    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 04:15:50.575277 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:50.575232    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:15:53.050210 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:53.050174    4930 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea63aa93-5ntmv"]
Nov 01 04:15:53.099643 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:53.099614    4930 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea63aa93-5ntmv"]
Nov 01 04:15:53.618929 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:53.618893    4930 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2ad2d57-84ba-42c2-95cb-0492edd90fd3" path="/var/lib/kubelet/pods/b2ad2d57-84ba-42c2-95cb-0492edd90fd3/volumes"
Nov 01 04:15:54.921982 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:54.921948    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5d96cbcb67-k2pgr_6b7143c4-cf31-4517-9d32-879238fdb77d/manager/0.log"
Nov 01 04:15:57.284294 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:57.284261    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-n8jrg_a4e9e2a3-c46d-492e-a157-2c7097aa1486/revoke-keys/0.log"
Nov 01 04:15:57.506812 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:57.506781    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-xdpvq_7cb0c346-db7c-4b47-9866-a55f7b77c2a7/revoke-keys/0.log"
Nov 01 04:15:57.733091 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:57.733060    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-m7xld_1135cbb4-8c8e-4f6c-b081-4bdbe2617263/revoke-keys/0.log"
Nov 01 04:15:59.443502 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:59.443473    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-8549fcc7fb-s5trb_66e26411-cab9-4f6a-8814-53165e790a45/authorino/0.log"
Nov 01 04:15:59.904046 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:15:59.903997    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-k7f54_b06cb628-601b-44ce-9667-aa39fdbe1c01/manager/0.log"
Nov 01 04:16:00.200615 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:16:00.200509    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:16:00.604597 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:16:00.604547    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-d4pzp_1f32a73e-8d68-4f8a-bae1-a85c34d4fd80/manager/0.log"
Nov 01 04:16:01.168323 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:16:01.168290    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-rgb2s_347b9e3e-b16b-416f-abeb-137af0e6e8b0/manager/0.log"
Nov 01 04:16:01.983095 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:16:01.983052    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9_7785e6a9-3334-4855-9744-23a8295e3116/istio-proxy/0.log"
Nov 01 04:16:03.334660 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:16:03.334629    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7c5b899d5f-cf66s_07489556-954b-45d9-9ebc-53b2920d4004/istio-proxy/0.log"
Nov 01 04:16:03.606382 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:16:03.606294    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7c5b899d5f-d89gf_d4fa40a8-e13b-4c67-954e-d283853b8fe4/istio-proxy/0.log"
Nov 01 04:16:04.691932 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:16:04.691900    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7c5b899d5f-wmcn6_1dd1479c-0836-4b90-a7df-6f5d1a4683c6/istio-proxy/0.log"
Nov 01 04:16:05.186828 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:16:05.186794    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7c5b899d5f-xtgmg_12bbd677-485b-4653-8261-8a3399da12d5/istio-proxy/0.log"
Nov 01 04:16:05.805896 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:16:05.805860    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh_c26292f6-f496-4947-968d-55497ed63981/istio-proxy/0.log"
Nov 01 04:16:06.557163 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:16:06.557133    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-ddbff58bc-5dlt4_be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88/router/0.log"
Nov 01 04:16:07.828185 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:16:07.828150    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-64vp4_16d41e60-8b45-447c-aec0-798c5712ece0/main/0.log"
Nov 01 04:16:08.105021 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:16:08.104992    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-5rwkr_53a7d5be-9872-49c5-9f21-f1e2f55d444a/main/0.log"
Nov 01 04:16:08.950199 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:16:08.950161    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-qhchk_98c81f0f-a540-4758-a3fa-e46f639e89d6/main/0.log"
Nov 01 04:16:09.491929 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:16:09.491904    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp_667240b0-64a7-45cd-84fe-9cd7e0d8a301/main/0.log"
Nov 01 04:16:10.506426 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:16:10.506397    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-8549fcc7fb-s5trb_66e26411-cab9-4f6a-8814-53165e790a45/authorino/0.log"
Nov 01 04:16:12.579100 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:16:12.579064    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:16:12.730881 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:16:12.730843    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:16:19.121168 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:16:19.121072    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 04:16:21.270898 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:16:21.270863    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5d96cbcb67-k2pgr_6b7143c4-cf31-4517-9d32-879238fdb77d/manager/0.log"
Nov 01 04:16:23.204272 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:16:23.204239    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-n8jrg_a4e9e2a3-c46d-492e-a157-2c7097aa1486/revoke-keys/0.log"
Nov 01 04:16:23.438591 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:16:23.438552    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-xdpvq_7cb0c346-db7c-4b47-9866-a55f7b77c2a7/revoke-keys/0.log"
Nov 01 04:16:23.672440 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:16:23.672411    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-m7xld_1135cbb4-8c8e-4f6c-b081-4bdbe2617263/revoke-keys/0.log"
Nov 01 04:16:25.082750 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:16:25.082721    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-8549fcc7fb-s5trb_66e26411-cab9-4f6a-8814-53165e790a45/authorino/0.log"
Nov 01 04:16:25.548898 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:16:25.548871    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-k7f54_b06cb628-601b-44ce-9667-aa39fdbe1c01/manager/0.log"
Nov 01 04:16:26.291218 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:16:26.291184    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-d4pzp_1f32a73e-8d68-4f8a-bae1-a85c34d4fd80/manager/0.log"
Nov 01 04:16:26.825365 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:16:26.825325    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-rgb2s_347b9e3e-b16b-416f-abeb-137af0e6e8b0/manager/0.log"
Nov 01 04:16:27.469967 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:16:27.469936    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9_7785e6a9-3334-4855-9744-23a8295e3116/istio-proxy/0.log"
Nov 01 04:16:28.815715 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:16:28.815684    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7c5b899d5f-cf66s_07489556-954b-45d9-9ebc-53b2920d4004/istio-proxy/0.log"
Nov 01 04:16:29.086645 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:16:29.086604    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7c5b899d5f-d89gf_d4fa40a8-e13b-4c67-954e-d283853b8fe4/istio-proxy/0.log"
Nov 01 04:16:30.137687 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:16:30.137650    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7c5b899d5f-wmcn6_1dd1479c-0836-4b90-a7df-6f5d1a4683c6/istio-proxy/0.log"
Nov 01 04:16:30.737015 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:16:30.736980    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7c5b899d5f-xtgmg_12bbd677-485b-4653-8261-8a3399da12d5/istio-proxy/0.log"
Nov 01 04:16:31.334859 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:16:31.334827    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-776d577b7-69qfh_c26292f6-f496-4947-968d-55497ed63981/istio-proxy/0.log"
Nov 01 04:16:32.073396 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:16:32.073363    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-ddbff58bc-5dlt4_be72b1cb-ee7a-4eca-a0f7-a5d0a39bec88/router/0.log"
Nov 01 04:16:33.035044 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:16:33.035014    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-64vp4_16d41e60-8b45-447c-aec0-798c5712ece0/main/0.log"
Nov 01 04:16:33.322766 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:16:33.322688    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-5rwkr_53a7d5be-9872-49c5-9f21-f1e2f55d444a/main/0.log"
Nov 01 04:16:34.153382 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:16:34.153352    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-qhchk_98c81f0f-a540-4758-a3fa-e46f639e89d6/main/0.log"
Nov 01 04:16:34.723997 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:16:34.723971    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-k2sxp_667240b0-64a7-45cd-84fe-9cd7e0d8a301/main/0.log"
Nov 01 04:16:38.520283 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:16:38.520251    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jh955_f29ea61e-2a78-4c11-9566-f52497e135ff/console-operator/2.log"
Nov 01 04:16:38.553251 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:16:38.553230    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jh955_f29ea61e-2a78-4c11-9566-f52497e135ff/console-operator/2.log"
Nov 01 04:16:49.121709 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:16:49.121674    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 04:16:50.983238 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:16:50.983204    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:17:00.575028 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:17:00.574989    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:17:06.160757 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:17:06.160721    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5d96cbcb67-k2pgr_6b7143c4-cf31-4517-9d32-879238fdb77d/manager/0.log"
Nov 01 04:17:06.738588 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:17:06.738546    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5d96cbcb67-k2pgr_6b7143c4-cf31-4517-9d32-879238fdb77d/manager/0.log"
Nov 01 04:17:19.121321 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:17:19.121282    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 04:17:41.413745 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:17:41.413715    4930 scope.go:122] "RemoveContainer" containerID="bba2e2e4565f4f38251c28750715bc4b36275c76f624a0de6d4eaefdb9720a88"
Nov 01 04:17:49.034134 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:17:49.034048    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5d96cbcb67-k2pgr_6b7143c4-cf31-4517-9d32-879238fdb77d/manager/0.log"
Nov 01 04:17:49.122140 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:17:49.122102    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 04:17:49.624262 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:17:49.624227    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5d96cbcb67-k2pgr_6b7143c4-cf31-4517-9d32-879238fdb77d/manager/0.log"
Nov 01 04:17:50.996338 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:17:50.996301    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-p847s/must-gather-hbr6n"]
Nov 01 04:17:50.996848 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:17:50.996830    4930 cpu_manager.go:414] "containerMap: removing container" podUID="7cb0c346-db7c-4b47-9866-a55f7b77c2a7" containerName="revoke-keys"
Nov 01 04:17:50.996911 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:17:50.996849    4930 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cb0c346-db7c-4b47-9866-a55f7b77c2a7" containerName="revoke-keys"
Nov 01 04:17:50.996959 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:17:50.996927    4930 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cb0c346-db7c-4b47-9866-a55f7b77c2a7" containerName="revoke-keys"
Nov 01 04:17:50.998000 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:17:50.997972    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:17:51.000262 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:17:51.000246    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p847s/must-gather-hbr6n"
Nov 01 04:17:51.057445 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:17:51.057409    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p847s\"/\"openshift-service-ca.crt\""
Nov 01 04:17:51.084117 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:17:51.084081    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4ae3de13-9f60-4ce9-a7af-57f0bd3b43d5-must-gather-output\") pod \"must-gather-hbr6n\" (UID: \"4ae3de13-9f60-4ce9-a7af-57f0bd3b43d5\") " pod="openshift-must-gather-p847s/must-gather-hbr6n"
Nov 01 04:17:51.084260 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:17:51.084244    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnxr5\" (UniqueName: \"kubernetes.io/projected/4ae3de13-9f60-4ce9-a7af-57f0bd3b43d5-kube-api-access-qnxr5\") pod \"must-gather-hbr6n\" (UID: \"4ae3de13-9f60-4ce9-a7af-57f0bd3b43d5\") " pod="openshift-must-gather-p847s/must-gather-hbr6n"
Nov 01 04:17:51.106973 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:17:51.106939    4930 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-p847s\"/\"default-dockercfg-d54w4\""
Nov 01 04:17:51.107135 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:17:51.107043    4930 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p847s\"/\"kube-root-ca.crt\""
Nov 01 04:17:51.109434 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:17:51.109407    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p847s/must-gather-hbr6n"]
Nov 01 04:17:51.166681 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:17:51.166651    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:17:51.184978 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:17:51.184949    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4ae3de13-9f60-4ce9-a7af-57f0bd3b43d5-must-gather-output\") pod \"must-gather-hbr6n\" (UID: \"4ae3de13-9f60-4ce9-a7af-57f0bd3b43d5\") " pod="openshift-must-gather-p847s/must-gather-hbr6n"
Nov 01 04:17:51.185136 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:17:51.185032    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnxr5\" (UniqueName: \"kubernetes.io/projected/4ae3de13-9f60-4ce9-a7af-57f0bd3b43d5-kube-api-access-qnxr5\") pod \"must-gather-hbr6n\" (UID: \"4ae3de13-9f60-4ce9-a7af-57f0bd3b43d5\") " pod="openshift-must-gather-p847s/must-gather-hbr6n"
Nov 01 04:17:51.185308 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:17:51.185288    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4ae3de13-9f60-4ce9-a7af-57f0bd3b43d5-must-gather-output\") pod \"must-gather-hbr6n\" (UID: \"4ae3de13-9f60-4ce9-a7af-57f0bd3b43d5\") " pod="openshift-must-gather-p847s/must-gather-hbr6n"
Nov 01 04:17:51.245524 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:17:51.245490    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnxr5\" (UniqueName: \"kubernetes.io/projected/4ae3de13-9f60-4ce9-a7af-57f0bd3b43d5-kube-api-access-qnxr5\") pod \"must-gather-hbr6n\" (UID: \"4ae3de13-9f60-4ce9-a7af-57f0bd3b43d5\") " pod="openshift-must-gather-p847s/must-gather-hbr6n"
Nov 01 04:17:51.315940 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:17:51.315852    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p847s/must-gather-hbr6n"
Nov 01 04:17:51.322154 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:17:51.322118    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:17:51.692965 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:17:51.692938    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p847s/must-gather-hbr6n"]
Nov 01 04:17:51.694318 ip-10-0-141-122 kubenswrapper[4930]: W1101 04:17:51.694288    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ae3de13_9f60_4ce9_a7af_57f0bd3b43d5.slice/crio-0bf7f75ac0bd02b571be47d52363832681ab9451c486633be55a65b0e8833079 WatchSource:0}: Error finding container 0bf7f75ac0bd02b571be47d52363832681ab9451c486633be55a65b0e8833079: Status 404 returned error can't find the container with id 0bf7f75ac0bd02b571be47d52363832681ab9451c486633be55a65b0e8833079
Nov 01 04:17:51.696296 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:17:51.696280    4930 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 04:17:51.818855 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:17:51.818818    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p847s/must-gather-hbr6n" event={"ID":"4ae3de13-9f60-4ce9-a7af-57f0bd3b43d5","Type":"ContainerStarted","Data":"0bf7f75ac0bd02b571be47d52363832681ab9451c486633be55a65b0e8833079"}
Nov 01 04:17:53.832833 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:17:53.832790    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p847s/must-gather-hbr6n" event={"ID":"4ae3de13-9f60-4ce9-a7af-57f0bd3b43d5","Type":"ContainerStarted","Data":"634b8c2f0c7721719c44693eb0fb92fc6dc148b63a1e3825be7141fb36d942ea"}
Nov 01 04:17:53.832833 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:17:53.832836    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p847s/must-gather-hbr6n" event={"ID":"4ae3de13-9f60-4ce9-a7af-57f0bd3b43d5","Type":"ContainerStarted","Data":"112709ac5884c72a11edf0cec839113f4ae0fcd18fe3afc6ddc37c27cb059bc9"}
Nov 01 04:17:53.949484 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:17:53.949433    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-p847s/must-gather-hbr6n" podStartSLOduration=2.589441602 podStartE2EDuration="3.949419218s" totalImagesPullingTime="1.359977616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 04:17:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:17:53.94788855 +0000 UTC m=+4278.470895142" watchObservedRunningTime="2026-11-01 04:17:53.949419218 +0000 UTC m=+4278.472425830"
Nov 01 04:18:00.194470 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:18:00.193480    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4bvb6_a972ee28-29fa-4755-adcd-b40d476c22f0/global-pull-secret-syncer/0.log"
Nov 01 04:18:00.942512 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:18:00.941473    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:18:02.186304 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:18:02.186180    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-8plk2_581612ca-5a0b-4426-9d07-ee65f85e0412/connection-checker/0.log"
Nov 01 04:18:02.606823 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:18:02.606795    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-fznsl_e60132c0-948e-4f84-883e-d22564f210e1/connection-checker/0.log"
Nov 01 04:18:03.046119 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:18:03.043712    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-j6kdp_ffdeddae-4086-4fb5-b9f8-c8ffcb765925/connection-checker/0.log"
Nov 01 04:18:03.576058 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:18:03.576023    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log"
Nov 01 04:18:05.712627 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:18:05.712046    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-122.us-west-1.compute.internal_5391522643ff9514bed3da110ddd8b71/haproxy/0.log"
Nov 01 04:18:10.436597 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:18:10.435731    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-8549fcc7fb-s5trb_66e26411-cab9-4f6a-8814-53165e790a45/authorino/0.log"
Nov 01 04:18:10.706435 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:18:10.706257    4930 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-p847s/perf-node-gather-daemonset-qcsj5"]
Nov 01 04:18:10.712782 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:18:10.712759    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p847s/perf-node-gather-daemonset-qcsj5"
Nov 01 04:18:10.763613 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:18:10.763581    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-k7f54_b06cb628-601b-44ce-9667-aa39fdbe1c01/manager/0.log"
Nov 01 04:18:10.789313 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:18:10.789267    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7ec17a8e-33ec-443c-8088-af9dddbfb484-proc\") pod \"perf-node-gather-daemonset-qcsj5\" (UID: \"7ec17a8e-33ec-443c-8088-af9dddbfb484\") " pod="openshift-must-gather-p847s/perf-node-gather-daemonset-qcsj5"
Nov 01 04:18:10.789466 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:18:10.789333    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhdkt\" (UniqueName: \"kubernetes.io/projected/7ec17a8e-33ec-443c-8088-af9dddbfb484-kube-api-access-vhdkt\") pod \"perf-node-gather-daemonset-qcsj5\" (UID: \"7ec17a8e-33ec-443c-8088-af9dddbfb484\") " pod="openshift-must-gather-p847s/perf-node-gather-daemonset-qcsj5"
Nov 01 04:18:10.789466 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:18:10.789400    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7ec17a8e-33ec-443c-8088-af9dddbfb484-podres\") pod \"perf-node-gather-daemonset-qcsj5\" (UID: \"7ec17a8e-33ec-443c-8088-af9dddbfb484\") " pod="openshift-must-gather-p847s/perf-node-gather-daemonset-qcsj5"
Nov 01 04:18:10.789466 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:18:10.789434    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7ec17a8e-33ec-443c-8088-af9dddbfb484-lib-modules\") pod \"perf-node-gather-daemonset-qcsj5\" (UID: \"7ec17a8e-33ec-443c-8088-af9dddbfb484\") " pod="openshift-must-gather-p847s/perf-node-gather-daemonset-qcsj5"
Nov 01 04:18:10.789606 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:18:10.789470    4930 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ec17a8e-33ec-443c-8088-af9dddbfb484-sys\") pod \"perf-node-gather-daemonset-qcsj5\" (UID: \"7ec17a8e-33ec-443c-8088-af9dddbfb484\") " pod="openshift-must-gather-p847s/perf-node-gather-daemonset-qcsj5"
Nov 01 04:18:10.802103 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:18:10.802074    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p847s/perf-node-gather-daemonset-qcsj5"]
Nov 01 04:18:10.890348 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:18:10.890315    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhdkt\" (UniqueName: \"kubernetes.io/projected/7ec17a8e-33ec-443c-8088-af9dddbfb484-kube-api-access-vhdkt\") pod \"perf-node-gather-daemonset-qcsj5\" (UID: \"7ec17a8e-33ec-443c-8088-af9dddbfb484\") " pod="openshift-must-gather-p847s/perf-node-gather-daemonset-qcsj5"
Nov 01 04:18:10.890532 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:18:10.890355    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7ec17a8e-33ec-443c-8088-af9dddbfb484-podres\") pod \"perf-node-gather-daemonset-qcsj5\" (UID: \"7ec17a8e-33ec-443c-8088-af9dddbfb484\") " pod="openshift-must-gather-p847s/perf-node-gather-daemonset-qcsj5"
Nov 01 04:18:10.890532 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:18:10.890379    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7ec17a8e-33ec-443c-8088-af9dddbfb484-lib-modules\") pod \"perf-node-gather-daemonset-qcsj5\" (UID: \"7ec17a8e-33ec-443c-8088-af9dddbfb484\") " pod="openshift-must-gather-p847s/perf-node-gather-daemonset-qcsj5"
Nov 01 04:18:10.890532 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:18:10.890407    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ec17a8e-33ec-443c-8088-af9dddbfb484-sys\") pod \"perf-node-gather-daemonset-qcsj5\" (UID: \"7ec17a8e-33ec-443c-8088-af9dddbfb484\") " pod="openshift-must-gather-p847s/perf-node-gather-daemonset-qcsj5"
Nov 01 04:18:10.890532 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:18:10.890465    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7ec17a8e-33ec-443c-8088-af9dddbfb484-podres\") pod \"perf-node-gather-daemonset-qcsj5\" (UID: \"7ec17a8e-33ec-443c-8088-af9dddbfb484\") " pod="openshift-must-gather-p847s/perf-node-gather-daemonset-qcsj5"
Nov 01 04:18:10.890532 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:18:10.890475    4930 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7ec17a8e-33ec-443c-8088-af9dddbfb484-proc\") pod \"perf-node-gather-daemonset-qcsj5\" (UID: \"7ec17a8e-33ec-443c-8088-af9dddbfb484\") " pod="openshift-must-gather-p847s/perf-node-gather-daemonset-qcsj5"
Nov 01 04:18:10.890532 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:18:10.890520    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7ec17a8e-33ec-443c-8088-af9dddbfb484-lib-modules\") pod \"perf-node-gather-daemonset-qcsj5\" (UID: \"7ec17a8e-33ec-443c-8088-af9dddbfb484\") " pod="openshift-must-gather-p847s/perf-node-gather-daemonset-qcsj5"
Nov 01 04:18:10.890807 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:18:10.890522    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7ec17a8e-33ec-443c-8088-af9dddbfb484-proc\") pod \"perf-node-gather-daemonset-qcsj5\" (UID: \"7ec17a8e-33ec-443c-8088-af9dddbfb484\") " pod="openshift-must-gather-p847s/perf-node-gather-daemonset-qcsj5"
Nov 01 04:18:10.890807 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:18:10.890584    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ec17a8e-33ec-443c-8088-af9dddbfb484-sys\") pod \"perf-node-gather-daemonset-qcsj5\" (UID: \"7ec17a8e-33ec-443c-8088-af9dddbfb484\") " pod="openshift-must-gather-p847s/perf-node-gather-daemonset-qcsj5"
Nov 01 04:18:10.951139 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:18:10.951109    4930 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhdkt\" (UniqueName: \"kubernetes.io/projected/7ec17a8e-33ec-443c-8088-af9dddbfb484-kube-api-access-vhdkt\") pod \"perf-node-gather-daemonset-qcsj5\" (UID: \"7ec17a8e-33ec-443c-8088-af9dddbfb484\") " pod="openshift-must-gather-p847s/perf-node-gather-daemonset-qcsj5"
Nov 01 04:18:11.025948 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:18:11.025872    4930 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p847s/perf-node-gather-daemonset-qcsj5"
Nov 01 04:18:11.402984 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:18:11.402956    4930 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p847s/perf-node-gather-daemonset-qcsj5"]
Nov 01 04:18:11.404383 ip-10-0-141-122 kubenswrapper[4930]: W1101 04:18:11.404350    4930 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7ec17a8e_33ec_443c_8088_af9dddbfb484.slice/crio-ea1842a6c75c66e00d4df8d8925b7443ffa1a8340f29508358ca57d0aac0c746 WatchSource:0}: Error finding container ea1842a6c75c66e00d4df8d8925b7443ffa1a8340f29508358ca57d0aac0c746: Status 404 returned error can't find the container with id ea1842a6c75c66e00d4df8d8925b7443ffa1a8340f29508358ca57d0aac0c746
Nov 01 04:18:11.968231 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:18:11.968187    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p847s/perf-node-gather-daemonset-qcsj5" event={"ID":"7ec17a8e-33ec-443c-8088-af9dddbfb484","Type":"ContainerStarted","Data":"03d7f3cce9054f853b857c383146a9ab564a337584c7f85119d736e8fd0d98cc"}
Nov 01 04:18:11.968684 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:18:11.968267    4930 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p847s/perf-node-gather-daemonset-qcsj5" event={"ID":"7ec17a8e-33ec-443c-8088-af9dddbfb484","Type":"ContainerStarted","Data":"ea1842a6c75c66e00d4df8d8925b7443ffa1a8340f29508358ca57d0aac0c746"}
Nov 01 04:18:11.969233 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:18:11.969211    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-p847s/perf-node-gather-daemonset-qcsj5"
Nov 01 04:18:12.062845 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:18:12.062806    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-d4pzp_1f32a73e-8d68-4f8a-bae1-a85c34d4fd80/manager/0.log"
Nov 01 04:18:12.087970 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:18:12.087915    4930 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-p847s/perf-node-gather-daemonset-qcsj5" podStartSLOduration=2.087900355 podStartE2EDuration="2.087900355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 04:18:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 04:18:12.084613126 +0000 UTC m=+4296.607619711" watchObservedRunningTime="2026-11-01 04:18:12.087900355 +0000 UTC m=+4296.610906937"
Nov 01 04:18:12.938238 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:18:12.938201    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-rgb2s_347b9e3e-b16b-416f-abeb-137af0e6e8b0/manager/0.log"
Nov 01 04:18:18.494624 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:18:18.494583    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-xbdmd_fecfe177-e63f-490f-94bc-c307041640d5/cluster-monitoring-operator/0.log"
Nov 01 04:18:18.694820 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:18:18.694786    4930 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-62ccn_c014e8ac-58ca-4161-a778-ba58b0e84147/konnectivity-agent/0.log" limit=1024
Nov 01 04:18:18.993118 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:18:18.993089    4930 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-p847s/perf-node-gather-daemonset-qcsj5"
Nov 01 04:18:19.121456 ip-10-0-141-122 kubenswrapper[4930]: E1101 04:18:19.121417    4930 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2vfgr" containerName="extractor"
Nov 01 04:18:21.136018 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:18:21.135975    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2sb2r_d0d60814-1a5f-41d2-a8d2-6b274e743224/node-exporter/0.log"
Nov 01 04:18:21.550129 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:18:21.550039    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2sb2r_d0d60814-1a5f-41d2-a8d2-6b274e743224/kube-rbac-proxy/0.log"
Nov 01 04:18:21.965866 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:18:21.965838    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2sb2r_d0d60814-1a5f-41d2-a8d2-6b274e743224/init-textfile/0.log"
Nov 01 04:18:33.309423 ip-10-0-141-122 kubenswrapper[4930]: I1101 04:18:33.309393    4930 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-5dfc747dd96ngs9_7785e6a9-3334-4855-9744-23a8295e3116/istio-proxy/0.log"